Dealing with unexpected results using HTTPGet

[dhdurgee]

I am getting some unexpected behavior attempting to retrieve data with HTTPGet at times. In both cases I am getting a 0 for the first item of the result, but sometimes I get the data in the third item and sometimes I don't. When I get the data item two contains (EG:)

http/1.1 200 ok
date Tue, 22 Mar 2011 15:22:34 GMT
server Apache/2.2.3 (Red Hat)
x-powered-by PHP/5.2.17
vary Accept-Encoding,Cookie
cache-control max-age=300, must-revalidate
wp-super-cache Served supercache file from PHP
connection close
transfer-encoding chunked
content-type text/html; charset=UTF-8

When I don't get the data item two contains (EG:)

http/1.1 301 moved permanently
date Tue, 22 Mar 2011 15:19:33 GMT
server Apache/2.2.3 (Red Hat)
x-powered-by PHP/5.2.17
vary Cookie
x-pingback http://mdlottery.com/xmlrpc.php
location http://mdlottery.com/mega-millions-deta ... lts-31811/
connection close
transfer-encoding chunked
content-type text/html; charset=UTF-8

Obviously I can detect the problem, but what is the appropriate way to deal with it? I don't see this happening in a browser, so there must be an appropriate action taken there. It also appears that HTTPGet does not, at least by default, take this action. I guess I could simply keep trying until I get the proper results, but that does not seem particularly elegant or efficient.

Dave

[Morten|Dyalog]

Dave,

http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html says:

10.3.2 301 Moved Permanently

The requested resource has been assigned a new permanent URI and any future references to this resource SHOULD use one of the returned URIs. Clients with link editing capabilities ought to automatically re-link references to the Request-URI to one or more of the new references returned by the server, where possible. This response is cacheable unless indicated otherwise.

The new permanent URI SHOULD be given by the Location field in the response. Unless the request method was HEAD, the entity of the response SHOULD contain a short hypertext note with a hyperlink to the new URI(s).

If the 301 status code is received in response to a request other than GET or HEAD, the user agent MUST NOT automatically redirect the request unless it can be confirmed by the user, since this might change the conditions under which the request was issued.

The HTTPGet utility is not intended to be able to deal with all the different HTTP return codes. It may well be a good idea for someone to write a more sophisticated client tool. On the other hand, your application is supposed to "take note" that a link has moved, so it is probably inappropriate for a general utility to just issue a 2nd request.

[dhdurgee]

Thanks for the pointer to the rfc, I should have though to check that. In this particular case I believe that the server is not returning the 301 code appropriately, as the location URI is identical to the requested URI and the entity contains only a "0" with cr,lf pairs.
I have implemented a delay and resend to deal with the situation.

Dave