jedidja.ca

Before reading this, refresh your memory of Joel Spolsky's wonderful post on the Law of Leaky Abstractions.

After determining it was possible to write a replacement HttpClient for WP8.1 Silverlight, I wanted to go a bit deeper and propose an actual fix for the devs at Mashape. Although submitting the error message (The server committed a protocol violation. Section=ResponseHeader Detail=CR must be followed by LF) seemed sufficient, given that the problem persisted a month later I wondered if something was wrong with the .Net framework (hah!)

Strings, Bytes, and CRLF

The relevant code in my "custom" HTTP client that read data from the StreamSocket.InputStream (via a DataReader) looked like this (simplified):

await reader.LoadAsync(10240);
data += 
  reader.ReadString(reader.UnconsumedBufferLength);

Viewing the value in the debugger or outputting it to the debug window didn't show any obvious missing line-feeds, but perhaps the reader was doing something weird.

Swapping the code to

await reader.LoadAsync(bytesToRead);
var buffer = new byte[reader.UnconsumedBufferLength];
reader.ReadBytes(buffer);

also showed that everything was as it should be. Every single line in the response header ended in 0D0A.

Just in case the reader was doing something weird, I double-checked in Fiddler. Sure enough, there were only 0D0A pairs.

Huh. I figured at this point it was time to go step through the Framework code.

ParseHeadersStrict

I knew from reading Mehdi's extremely in-depth article on the same problem that problem probably stemmed from somewhere in ParseHeadersStrict, although it took me a while to get there. Given that I had verified from two different sources that every header line did end in CRLF, I thought perhaps something else in the Framework was acting up. As it turns out, it was exactly where I should have looked in the first place.

Here's the relevant code from WebHeaderCollection.cs. At this point we are looping through a byte[] buffer containing the response.

// Find the header name; only regular characters allowed.
int iBeginName = i;
for (; i < effectiveSize && 
      (ch = 
         byteBuffer[i] > 127 ? 
           RfcChar.High :
           RfcCharMap[byteBuffer[i]])
       == RfcChar.Reg; 
       i++);    

Gotta love the ternary operator and assignment used for the comparison inside the condition section of a for loop :) Basically, this is looping through each line of the response header looking for lines like

Content-Type: application/json; charset=UTF-8

and ensuring they're valid. The above loop is ensuring that the header name (before the colon) doesn't have any weird characters in it.

The next portion of code ensures that we read the colon and header value and Most Importantly that the line ends with CRLF. Given how many times I had seen the error message CR must be followed by LF, I was quite curious to see how this was going to fail.

// Read to a colon.
int iEndName = i - 1;
int crlf = 0;  // 1 = cr, 2 = crlf
for (; i < effectiveSize && 
      (ch = 
         byteBuffer[i] > 127 ? 
           RfcChar.High : 
           RfcCharMap[byteBuffer[i]]) 
      != RfcChar.Colon; 
      i++)
{
  switch (ch)
  {
    case RfcChar.WS:
    if (crlf == 1)
    {
      break;
    }
    crlf = 0;
    continue;

    case RfcChar.CR:
    if (crlf == 0)
    {
      crlf = 1;
      continue;
    }
    break;

    case RfcChar.LF:
    if (crlf == 1)
    {
      crlf = 2;
      continue;
    }
    break;
  }
  parseStatus = DataParseStatus.Invalid;
  parseErrorCode = WebParseErrorCode.CrLfError;
  goto quit;
}

And yet it blew up. Again. Even though I could see in the VS Memory window that every single 0D was followed by OA.

After a round of tracing, I noticed that the header was mostly being parsed. And then it jumped out at me. The line it was failing on looked like this:

X-RateLimit-API Requests-Limit: 7500

and there was definitely something different about that header name compared to all the others.

The Devil, the details, and RFCs

Those of you who have worked with HTTP at this level will immediately know the issue here, but I'm brand new to it so off to read the spec I went. It didn't take long to discover these very pertinent sections:

Message Headers

message-header = field-name ":" [ field-value ]
field-name     = token

Augmented BNF

SP             = <US-ASCII SP, space (32)>

token          = 1*<any CHAR except CTLs or separators>
separators     = "(" | ")" | "<" | ">" | "@"
                  | "," | ";" | ":" | "\" | <">
                  | "/" | "[" | "]" | "?" | "="
                  | "{" | "}" | SP | HT

I can't highlight code, but the point is that you can't have a space in the header name. Very simple. And although Microsoft was correct to disallow this in their strict interpretation of RFC2616, it would have been nicer to have a different error message for this scenario instead of:

The server committed a protocol violation. Section=ResponseHeader Detail=CR must be followed by LF`

I sent all this info off to Mashape this evening and am hoping they can fix it soon.

Update: Mashape fixed this issue on August 21.

In my last post, I discussed how Windows Phone and Store apps will refuse to parse HTTP responses (namely headers) that do not strictly follow RFC2616. After a few weeks of waiting for Mashape to fix their responses, I thought it might be worth it to explore other options.

The first idea was to implement a custom handler, but that quickly ended when I found out that the system's handler gets called first for a response.

The above image used to be hosted at http://www.asp.net/web-api/overview/web-api-clients/httpclient-message-handlers but the page disappeared in the last day or two.

After stumbling through the usual mix of Windows Phone 8.1 Silverlight vs Store (RT) docs, I came across an RT class that is also available on Silverlight: StreamSocket. Knowing very little about the protocol, this page and the RFC page on Messages the were a great resources.

It turns out (ignoring error handling, edge cases, etc. etc.) that you can implement your own HTTP client fairly simply!

var hostname = new HostName("www.w3.org");
var socket = new StreamSocket();
await socket.ConnectAsync(hostname, "80");

var request = "GET /Protocols/rfc2616/rfc2616-sec4.html HTTP/1.1\r\n" +
              "Host: www.w3.org\r\n" +
              "\r\n";

var writer = new DataWriter(socket.OutputStream);
writer.WriteString(request);
await writer.StoreAsync();

var reader = new DataReader(socket.InputStream);
reader.InputStreamOptions = InputStreamOptions.Partial;

string data = string.Empty;    
var cts = new CancellationTokenSource();

bool doneReading = false;
uint bytesToRead = 4096;
uint totalBytesRead = 0;

while (!doneReading)
{
  try
  {
    cts.CancelAfter(10 * 1000);
    var bytesRead = 
      await reader.LoadAsync(bytesToRead).AsTask(cts.Token);
    data += reader.ReadString(reader.UnconsumedBufferLength);

    totalBytesRead += bytesRead;
  }
  catch (TaskCanceledException)
  {
    doneReading = true;
  }
}

socket.Dispose();

Since LoadAsync returns the # of bytes loaded into the stream, you might think (as I did) that if it was less than the bytesToRead parameter the response was over and you could close the socket. Unfortunately this isn't necessarily true.

This (lazy) sample solution lets the socket timeout after 10s and assumes that since the server has stopped sending information, all is well.

The correct way to do it, as Jon Skeet has already pointed out, is to parse the header, read the value of Content-Length and then request the appropriate amount of data aftewards. This is slightly non-trivial, as you are dealing with a stream and there is no built-in way of magically parsing out the header nor its contents. It's far from difficult, though, and I will post a Github link once I get some decent looking code prepared.

The only other tweak left was to switch to https to work with Mashape. This is as simple as replacing

await socket.ConnectAsync(hostname, "80");

with

await socket.ConnectAsync(hostname, "https", SocketProtectionLevel.Tls12);

Now we can handle "protocol violations" on Windows Phone/Store apps with no problems!

(Actually, if you are on a WinRT app, MessageWebSocket seems like an even easier solution.)

Since I lost several hours on this today, I thought I'd quickly share my experience. There were very few results for the inital terms I searched for so perhaps this may help someone in the future.

Laziness

I've been working on what will hopefully be my first released Windows Phone app (8.1 Silverlight) for the last two months. When I say two months, I worked on it for two weeks before the World Cup started and finally picked it up again today.

In the interim, it appears that Mashape has slightly changed the way you use their API. Or, more specifically, they've removed developer keys and only app-specific keys remain. (This was the first 30 minutes lost; I still don't see a tweet or blog post about this).

So it's my own fault I'm only now discovering there have been some changes:

• the removal of account keys
• the websuite UI (which is very difficult to view in Chrome on Windows)
• the data returned from an API call (?!)

Complete Confusion

After noticing the sample API page required you to now use X-Mashape-Key instead of X-Mashape-Authorization, I went ahead and made a bunch of other changes to my app without running it (Poor, poor Avatar). As soon as I made an API request, the code crashes with

Catastrophic failure
(Exception from HRESULT: 0x8000FFFF (E_UNEXPECTED))
at System.Runtime.CompilerServices
                 .TaskAwaiter
                 .ThrowForNonSuccess(Task task)
at System.Runtime
         .CompilerServices
         .TaskAwaiter
         .HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices
                 .TaskAwaiter`1
                 .GetResult()

Here's the code that had worked so well a few weeks ago:

var httpClient = new HttpClient();
httpClient.DefaultRequestHeaders
          .Add("X-Mashape-Authorization", "...");

var uri = new Uri("...");
var response = await httpClient.GetStringAsync(uri);

var stuff = JsonConvert.DeserializeObject<...>(response);

Huh. What had I missed?

• using an await incorrectly
• some kind of incorrect binding
• a change to the WP SDK
• ???

So I undid all my changes, changed the one header, and still got the same crash.

Which usually means the Emulator is broken. Or Visual Studio. Or perhaps even the OS. After a complete relaunch and reboot, it turned out it was still my problem.

Fiddler to the rescue! Or not.

My next thought was that perhaps along with the header name change, I needed to include some more information in the GET request. I could tell from my REST client that I shouldn't need anything else, but you never know. After hacking around with Fiddler for the better part of an hour (see here and here), I gave up.

Windows Store

I know Windows Phone Silverlight has its own quirks, but since there is still an overlap with WinRT apps, I thought perhaps creating a "normal" Windows Store app might help. There was also a better possibility of it working with Fiddler.

And yet the same crash persisted.

Even trying HttpRequestMessage and GetStringAsync yielded the same odd Catastrophic Failure. "PC Load Letter???" Seriously!

The one odd, ODD thing was now Fiddler was showing me a proper result. No HTTP error codes or gobbeldygook; the response seemed like exactly what I'd expect.

Good ol' WPF to the rescue

Throwing together a sample WPF app finally rewarded me with some useful information:

The server committed a protocol violation. Section=ResponseHeader Detail=CR must be followed by LF

And a few quick searches later, http://stackoverflow.com/questions/18580016/httpclient-server-committed-a-protocol-violation-section-responseheader-detail

(wonderful, for WPF apps - it took all of two seconds to verify the app.config change would let the SendAsync pass)

followed by http://social.msdn.microsoft.com/Forums/windowsapps/en-US/5a8f4b30-a91a-407f-aa6e-7be6ff68a152/how-to-set-useunsafeheaderparsing-of-httpclient

(not so good for Windows Store or Windows Phone apps. Sigh.)

Browsing through the code for unirest-net I couldn't see how it would possibly work even with Mashape's own recommended library. But, just in case, I downloaded it, built it, and was "rewarded" with the same message a few minutes later.

Conclusion

Windows Phone and WinRT error messages can be very annoying to debug. And depending on a third-party for your app can be very trying. But the sense of satisfaction at being able to submit a "good" bug report is ... almost worth it :)

I'm off to submit a bug report and cross my fingers. Let's hope I didn't miss something really simple in my investigation...

Update (July 25): Although I did get a dialog started with Marco and Morgan, the only advice I've received so far is to use the app.config fix (which doesn't work on Windows Phone). I'm really glad we hadn't launched a product yet; we'd have some very unhappy customers at this point.