c# - HttpWebRequest with caching enabled throws exceptions -

i'm working on small c#/wpf application interfaces web service implemented in ruby on rails, using handcrafted httpwebrequest calls , json serialization. without caching, works it's supposed to, , i've got http authentication , compression working well.

once enable caching, setting request.cachepolicy = new httprequestcachepolicy(httprequestcachelevel.cacheifavailable);, things go awry - in production environment. when connecting simple webrick instance, things work fine, http/1.1 304 not modified expected , httpwebrequest delivers cached content.

when try same against production server, running nginx/0.8.53 + phusion passenger 3.0.0, application breaks. first request (uncached) served properly, on second request results in 304 response, webexception stating "the request aborted: request canceled." invoke request.getresponse().

i've run connections through fiddler, hasn't helped whole lot; both webrick , nginx return empty entity body, albeit different response headers. intercepting request , changing response headers nginx match of webrick didn't change anything, leading me think keep-alive issue; setting request.keepalive = false; changes nothing, though - doesn't break stuff when connecting webrick, , doesn't fix stuff when connecting nginx.

for it's worth, webexception.innerexception nullreferenceexception following stacktrace:

at system.net.httpwebrequest.checkcacheupdateonresponse() @ system.net.httpwebrequest.checkresubmitforcache(exception& e) @ system.net.httpwebrequest.dosubmitrequestprocessing(exception& exception) @ system.net.httpwebrequest.processresponse() @ system.net.httpwebrequest.setresponse(coreresponsedata coreresponsedata) 

headers (working) webrick connection:

########## request /users/current.json http/1.1 authorization: basic *redacted* content-type: application/json accept: application/json accept-charset: utf-8 host: testbox.local:3030 if-none-match: "84a49062768e4ca619b1c081736da20f" accept-encoding: gzip, deflate connection: keep-alive ########## response http/1.1 304 not modified x-ua-compatible: ie=edge etag: "84a49062768e4ca619b1c081736da20f" date: wed, 01 dec 2010 18:18:59 gmt server: webrick/1.3.1 (ruby/1.8.7/2010-08-16) x-runtime: 0.177545 cache-control: max-age=0, private, must-revalidate set-cookie: *redacted* 

headers (exception-throwing) nginx connection:

########## request /users/current.json http/1.1 authorization: basic *redacted* content-type: application/json accept: application/json accept-charset: utf-8 host: testsystem.local:8080 if-none-match: "a64560553465e0270cc0a23cc4c33f9f" accept-encoding: gzip, deflate connection: keep-alive ########## response http/1.1 304 not modified connection: keep-alive status: 304 x-powered-by: phusion passenger (mod_rails/mod_rack) 3.0.0 etag: "a64560553465e0270cc0a23cc4c33f9f" x-ua-compatible: ie=edge,chrome=1 x-runtime: 0.240160 set-cookie: *redacted* cache-control: max-age=0, private, must-revalidate server: nginx/0.8.53 + phusion passenger 3.0.0 (mod_rails/mod_rack) 

update:

i tried doing quick-and-dirty manual etag cache, turns out that's no-go: webexception when invoking request.getresponce(), telling me "the remote server returned error: (304) not modified." - yeah, .net, kinda knew that, , i'd (attempt to) handle myself, grr.

update 2:

getting closer root of problem. showstopper seems difference in response headers initial request. webrick includes date: wed, 01 dec 2010 21:30:01 gmt header, isn't present in nginx reply. there's other differences well, intercepting initial nginx reply fiddler , adding date header, subsequent httpwebrequests able process (unmodified) nginx 304 replies.

going try workaround, getting nginx add date header.

update 3:

it seems serverside issue phusion passenger, have open issue lack of date header. i'd still httpwebrequest's behavior is... suboptimal.

update 4:

added microsoft connect ticket bug.

so, turns out phusion passenger (or nginx, depending on how @ - , thin well) doesn't add date http response header, combined see bug in .net httpwebrequest (in situation there's no if-modified-since, date shouldn't necessary) leading problem.

the workaround this particular case edit our rails applicationcontroller:

class applicationcontroller < actioncontroller::base     # ...other stuff here      before_filter :add_date_header     # bugfix .net httpwebrequst 304-handling bug , various     # webservers' lazyness in not adding date: response header.     def add_date_header         response.headers['date'] = time.now.to_s     end end 

update:

turns out it's bit more complex "just" setting httprequestcachepolicy - repro, need have manually constructed http basic auth. involved components following:

1. http server doesn't include http "date:" response header.
2. manual construction of http authorization request header.
3. use of httprequestcachepolicy.

smallest repro i've been able come with:

namespace repro {     using system;     using system.io;     using system.net;     using system.net.cache;     using system.text;      class reproprog     {         const string requesturl = "http://drivelog.miracle.local:3030/users/current.json";          // manual construction of http basic auth don't unnecessary server         // roundtrip telling auth, if use         // httpwebrequest.credentials.         private static void setauthorization(httpwebrequest request, string _username, string _password)         {             string userandpass = string.format("{0}:{1}", _username, _password);             byte[] authbytes = encoding.utf8.getbytes(userandpass.tochararray());             request.headers["authorization"] = "basic " + convert.tobase64string(authbytes);         }          static public void dorequest()         {             var request = (httpwebrequest) webrequest.create(requesturl);              request.method = "get";             request.cachepolicy = new httprequestcachepolicy(httprequestcachelevel.cacheifavailable);             setauthorization(request, "user@domain.com", "12345678");              using(var response = request.getresponse())             using(var stream = response.getresponsestream())             using(var reader = new streamreader(stream))             {                 string reply = reader.readtoend();                 console.writeline("########## server reply: {0}", reply);             }         }          static public void main(string[] args)         {             dorequest();    // works             dorequest();    // explodes         }     } }