Main menu

Star InactiveStar InactiveStar InactiveStar InactiveStar Inactive
Ok - So I've noticed twitter was not on my IM anymore. I regularly post via my gmail account on my rim. I didn't think much of this.

Marcel sent me an invite to plurk, and some posts on his lug-nuts list showed signs of twitter problems, so I decided to go over to the site to see what's what.

Seems they've been having issues for a while - I started off my journey to a "we're down" page. But! The blog is up, so I mossy on over there to see what's what.

I found the entry:  It's Not Rocket Science, But It's Our Work  - Here we have a link to the infamous status page. We also have some Q/A which yielded:

Q: Is it true that you only have a single master MySQL server running replication to two slaves, and the architecture doesn’t auto-switch to a hot backup when the master goes down?
A: We currently use one database for writes with multiple slaves for read queries. As many know, replication of MySQL is no easy task, so we've brought in MySQL experts to help us with that immediately. We've also ordered new machines and failover infrastructure to handle emergencies.

Ok, Their having problems, according to the site. Maybe I can help them out, over to the jobs page and their looking for a Systems Eng:


  • - M.S. Computer Science or related field preferred
  • - Extensive experience building large-scale server applications
  • - Expert knowledge developing and debugging in Java and C/C++ on Unix
  • - Knowledge of python and ruby
  • - Experience with distributed systems, operating system internals, filesystems, compilers, threading models, and server architectures
  • - Disciplined approach to testing and quality assurance
  • - Great written communication and documentation abilities
The bolded jumps right out at me. Sheesh! No wonder their having problems. Maybe I should dig into their infrastructure a little more some day.

Quick scan via netcraft shows us their using Linux/Apache - good call. Still need more details.

wget can let me see the host responses with -S:

jason@homer:/# wget -S
           => `index.html'
Connecting to||:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 301 Moved Permanently
  Date: Sat, 07 Jun 2008 01:48:20 GMT
  Server: Apache
  Cache-Control: max-age=300
  Expires: Sat, 07 Jun 2008 01:53:20 GMT
  Vary: Accept-Encoding
  Content-Length: 227
  Connection: close
  Content-Type: text/html; charset=iso-8859-1
Location: [following]
           => `index.html'
Connecting to||:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Date: Sat, 07 Jun 2008 01:48:22 GMT
  Server: hi
  Status: 200 OK
  X-Runtime: 0.01355
  ETag: "0b7227441bc85f068424c5ba65b9eb54"
  Cache-Control: private, max-age=0, must-revalidate, max-age=300
  Content-Type: text/html; charset=UTF-8
  Content-Length: 7302
  Via: 1.0
  Expires: Sat, 07 Jun 2008 01:53:22 GMT
  Vary: Accept-Encoding
  Connection: close
Length: 7,302 (7.1K) [text/html]

Going to yeilded a 301 Moved to /index.html (good, did this right). But whats interesting is, they have clobbered apache (baddly) to report a different server type. The 301 shows "apache", but the following 200 for the index.html shows "hi".

Also, jugging from the "Via" line, we have a load ballancer before the web servers - I assumed as much, but lets dig [Pun Entended] into the dns.

;; ANSWER SECTION:        60      IN      CNAME            60      IN      A
Odd - only one IP address. Must have a VIP on front end routers. No big deal there.

Nothing else shows us about the backend that I can see - so far.


** Updated **