Warning: Trying to access array offset on value of type null in /home/gregcklo/public_html/SiteCrawler/_variables.php on line 12

Warning: Trying to access array offset on value of type null in /home/gregcklo/public_html/SiteCrawler/_variables.php on line 13

Warning: Trying to access array offset on value of type null in /home/gregcklo/public_html/SiteCrawler/_variables.php on line 14

Warning: Trying to access array offset on value of type null in /home/gregcklo/public_html/SiteCrawler/_variables.php on line 15
Website Links

Website Links

Site Crawl Parameters

URL Parameters
Checking these options will increase the number of links in your website catalogue. Including anchor links will potentially add duplicates to your catalogue. Including query strings will allow for unique pages that are served through query strings; excluding query strings could also remove duplicate pages on some sites (like those that use a query string for user login features).
HTML Features
Track Links
Max Timeout
This is the maximum amount of time allowed to process each link. Setting the timeout to zero will remove the limit entirely, which could allow for more accurate information about site links, but could also mean longer crawl times.
 seconds
Max Connections
This is the maximum number of pages that will be crawled at once. More connections can potentially speed up your site crawl, but can possibly cause the crawler to get hung up as well.
Unique Index
Since URLs are unique (whereas page titles can be the same on different pages), using URLs as the crawling index ensures a complete mapping of your website. However, some sites (like blogs or wikis) may construct URLs dynamically, so the links in the body of one page do not necessarily correspond one-to-one with the page URLs for the site; in this case, indexing by page title will remove duplicate page scans.
URL Exclusions
Enter specific words or characters here, separated by a comma, to block certain URLs from being catalogued. This feature is especially useful for blogs and wikis that archive multiple saved versions of pages, where you'd only want to catalogue one page. Keywords like "page-revisions" or "?rev=" or "?mode=embedded" can prevent duplicates or drafts of pages from being catalogued.
This form creates the parameters for a site crawl and saves the settings to your account. Once created, the site will appear in the My Sites section below, where you can click the "Begin Site Crawl" button to create your site outline and scan for dead links.
Site Title
Domain Root
Start Page
(if different from root)
Continue - The server has received the request headers, and the client should proceed to send the request body.
Switching Protocols - The requester has asked the server to switch protocols.
Checkpoint - Used in the resumable requests proposal to resume aborted PUT or POST requests.
OK - The request is OK (this is the standard response for successful HTTP requests).
Created - The request has been fulfilled, and a new resource is created.
Accepted - The request has been accepted for processing, but the processing has not been completed.
Non-Authoritative Information - The request has been successfully processed, but is returning information that may be from another source.
No Content - The request has been successfully processed, but is not returning any content.
Reset Content - The request has been successfully processed, but is not returning any content, and requires that the requester reset the document view.
Partial Content - The server is delivering only part of the resource due to a range header sent by the client.
Multiple Choices - A link list. The user can select a link and go to that location. Maximum five addresses.
Moved Permanently - The requested page has moved to a new URL.
Found - The requested page has moved temporarily to a new URL.
See Other - The requested page can be found under a different URL.
Not Modified - Indicates the requested page has not been modified since last requested.
Switch Proxy - No longer used.
Temporary Redirect - The requested page has moved temporarily to a new URL.
Resume Incomplete - Used in the resumable requests proposal to resume aborted PUT or POST requests.
Bad Request - The request cannot be fulfilled due to bad syntax.
Unauthorized - The request was a legal request, but the server is refusing to respond to it. For use when authentication is possible but has failed or not yet been provided.
Payment Required - Reserved for future use.
Forbidden - The request was a legal request, but the server is refusing to respond to it.
Not Found - The requested page could not be found but may be available again in the future.
Method Not Allowed - A request was made of a page using a request method not supported by that page.
Not Acceptable - The server can only generate a response that is not accepted by the client.
Proxy Authentication Required - The client must first authenticate itself with the proxy.
Request Timeout - The server timed out waiting for the request.
Conflict - The request could not be completed because of a conflict in the request.
Gone - The requested page is no longer available.
Length Required - The "Content-Length" is not defined. The server will not accept the request without it.
Precondition Failed - The precondition given in the request evaluated to false by the server.
Request Entity Too Large - The server will not accept the request, because the request entity is too large.
Request-URI Too Long - The server will not accept the request, because the URL is too long. Occurs when you convert a POST request to a GET request with a long query information.
Unsupported Media Type - The server will not accept the request, because the media type is not supported.
Requested Range Not Satisfiable - The client has asked for a portion of the file, but the server cannot supply that portion.
Expectation Failed - The server cannot meet the requirements of the Expect request-header field.
Internal Server Error - A generic error message, given when no more specific message is suitable.
Not Implemented - The server either does not recognize the request method, or it lacks the ability to fulfill the request.
Bad Gateway - The server was acting as a gateway or proxy and received an invalid response from the upstream server.
Service Unavailable - The server is currently unavailable (overloaded or down).
Gateway Timeout - The server was acting as a gateway or proxy and did not receive a timely response from the upstream server.
HTTP Version Not Supported - The server does not support the HTTP protocol version used in the request.
Network Authentication Required - The client needs to authenticate to gain network access.

Copyright © 2024   Home Instructions HTTP Codes Terms and Policies