Velocity Session In the Belly of the Whale Operations at Twitter

In the Belly of the Whale: Operations at Twitter

John Adams (Twitter)
8:50am Thursday, 06/24/2010
Keynotes, Operations Ballroom ABCD
This is presentation will be streamed live along with the other keynotes.
John will cover many aspects of Twitter’s scaling efforts, including:

Finding the weak points in Ruby on Rails, and repairing them.
In house Peer-to-Peer: High speed deploys across thousands of machines in no time at all
Managing thousands of machines: Why you need a central machine database, now.
User management: How do you onboard many new developers and still remain fault tolerant?
Caching methodologies and Twitter’s open source efforts (Scarling, CacheMoney, Libmemcached)
Asynchronous versus Synchronous processing during request lifetime
Life after Syslog: What do do when it won’t work anymore.

John Adams (Twitter Operations) has worked in computer security, operations, and systems engineering for over 16 years. Prior to Twitter, he has worked at Apple, Inktomi, c|net, and a major video-on-demand site, improving security, performance, and reliability at large scale.

Notes on the Session

  • What’s changed at Twitter since last year
    • Now at 210 employees
    • Increased knowledge of Unicorn, Apache, Memcached
    • 700 million searches a day
    • 65 million tweets per day

Call Tweets Per Second (TPS)

Willingly admit that

MTTD to MTTR (Detect quickly and recover fast)…

  • Find weakest point –> Take Corrective Action –> Move on to the Next Weakest Point
    • and
    • Introspect and Google Perftools
    • Syslog doesn’t work at high loads
    • Track HTTP 500’s using Whale Watchers

Change Management

  • Reviewboard
  • Puppet
  • Deploy Watcher

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s