In the Belly of the Whale: Operations at Twitter
John Adams (Twitter)
8:50am Thursday, 06/24/2010
Keynotes, Operations Ballroom ABCD
This is presentation will be streamed live along with the other keynotes.
John will cover many aspects of Twitter’s scaling efforts, including:
Finding the weak points in Ruby on Rails, and repairing them.
In house Peer-to-Peer: High speed deploys across thousands of machines in no time at all
Managing thousands of machines: Why you need a central machine database, now.
User management: How do you onboard many new developers and still remain fault tolerant?
Caching methodologies and Twitter’s open source efforts (Scarling, CacheMoney, Libmemcached)
Asynchronous versus Synchronous processing during request lifetime
Life after Syslog: What do do when it won’t work anymore.
John Adams (Twitter Operations) has worked in computer security, operations, and systems engineering for over 16 years. Prior to Twitter, he has worked at Apple, Inktomi, c|net, and a major video-on-demand site, improving security, performance, and reliability at large scale.
- What’s changed at Twitter since last year
- Now at 210 employees
- Increased knowledge of Unicorn, Apache, Memcached
- 700 million searches a day
- 65 million tweets per day
Call Tweets Per Second (TPS)
Willingly admit that
MTTD to MTTR (Detect quickly and recover fast)…
- Find weakest point –> Take Corrective Action –> Move on to the Next Weakest Point
- Dev.twitter.com and status.twitter.com
- Introspect and Google Perftools
- Syslog doesn’t work at high loads
- Track HTTP 500’s using Whale Watchers
- Deploy Watcher