With the help of our customers we discovered that GET and HEAD requests are handled in an unpredictable fashion by some servers.
Some background: GET and HEAD are HTTP 1.1 request methods described in RFC2616 (section 9) , by RFC definition the difference between these methods is: "The HEAD method is identical to GET except that the server MUST NOT return a message-body in the response. The metainformation contained in the HTTP headers in response to a HEAD request SHOULD be identical to the information sent in response to a GET request. This method can be used for obtaining metainformation about the entity implied by the request without transferring the entity-body itself. This method is often used for testing hypertext links for validity, accessibility, and recent modification. "
According to this definition, we use the HEAD method to check external links on a web page (full page check). Unfortunately, some web servers return "Not Found" or other errors for HEAD requests. This may be the result of improper server configuration or unusual behavior required from the web server.
We are not going to change all HEAD requests to GET, because this would necessitate the download of a great number of large images and files just to check their availability. Fortunately, this issue is detected immediately and settings may be adjusted, (see "Solution" above), when the monitoring target is being set up.
Labels: get, head, http request, not found