Twitter

tunisianbromidrosisInternet and Web Development

Feb 5, 2013 (4 years and 7 months ago)

148 views

Twitter Copes
with Web2.0rhea

The Twits are catching up…

Alex Carian

ISM 158

May 6, 2010

From TheRegister.co.uk:

“Twitter: It's the end of the
sysadmin as we know it”


“"This is a whole new world…
For the longest time, people
ran large data systems on a
kind of ad hoc basis. We're in
a world now where so many
people are depending on the
real
-
time web. A system
administrator is not just a
system administrator
anymore.”
1

John Adams, Ops Engineer

-

Ad hoc solution:
designed for a
specific problem or
task, cannot be
adapted to other
purposes
3

The Real
-
Time Web?

-
Initially, the core of Twitter’s code ran:


-

Ruby on Rails (application framework)

-

Apache/Mongrel (web servers)


“You have to use analytics. You have
to grab data. You have to look at
where a site is trending and where
things are going so you can scale...
If you don't start doing this work
early, if you don't start collecting this
early, you will fail.”
-

John Adams,
Twitter Ops Engineer

Twitter’s Aches and Pains

-

Despite the benefits of
the RoR environment,
Twitter experiences
outages, most notably
during Steve Jobs’
Keynote address at
Macworld ‘08
3

-
A developer, in his spare time, ports a section of the
website code into the Scala programming language

-

Allows for object
-
oriented and functional
programming paradigms

-

Task that originally took 2.5 hrs now takes…


20 seconds!

Functional Bits

-
Twitter maintains Ruby on Rails for building “user facing
features” and MySQL for database management
1

-

Now entrusts back end infrastructure to Unicorn, a RoR
application server

-

Grocery Store model

-

Still open source

-

Still based on Mongrel (the former app server)


Solipsism 2.0

-
Twitter saw a 1358% increase in traffic in 2009 (comScore)

-

Utilizes a monitoring system known as ‘Ganglia’

-

open source program grown out of UCB and NSF grants

-

Tracks 15,000 points of site performance, including grid
load, cluster load, # of hosts up/down

TWITPOCALYPSE?!

The “Fail Whale”

-

Many suspected Twitter’s
service would come
unwound after reaching the
Unique Tweet ID limit

(2^31 = 4,294,967,295)

-
Twitter’s ops team mined site data to predict when the
limit would be reached

-

Allowed the team to implement necessary changes to
avoid complications

-

Current number of Tweets posted: ~13,475,000,000
5

Another Word on Sysadmin 2.0…

“"Another discovery that we made when trying to increase the scale of
Twitter was that disk is the new tape," he said. "With any sort of social
networking operation
-

juggling followers, sending mail, etc.
-

disk is
extremely slow.”
1

-
John Adams

-

Hard drive? That’s so 2009…. Or 1959…

-
Twitter now uses memcached

-

Data and objects cached in RAM to reduce the
number of times an external data source (such as a
database or API) must be read.
4

-

Don’t overdo it.

Twitter’s Open Back End

-

Company uses open sourced technology in most operations

-

RoR, Unicorn, Ganglia,

-

Developed several tools at home

-

Kestrel, message queue server

-

FlockDB, follower database

-

Both now released as open source


Will Twitter maintain its open source track record?

Sources Cited

1.
Metz, Cade. “Twitter: It's the end of the sysadmin as we know it”. The
Register, May 4, 2010.
<http://www.theregister.co.uk/2010/05/04/twitter_and_the_end_of_the_sysad
min_as_we_know_it/>

2.
Metz, Cade. “Twitter jilts Ruby for Scala” The Register, April 1, 2009.
http://www.theregister.co.uk/2009/04/01/twitter_on_scala/

3.
Riley, Duncan. “Twitter Fails Macworld Keynote Test” TechCrunch, Jan 2008.
<http://techcrunch.com/2008/01/15/twitter
-
fails
-
macworld
-
keynote
-
test/>

3.
“Ad hoc”. Wikipedia

4.
“memcached”. Wikipedia

5.
“Twitter Twitpocalypse Status”. Accessed May 6, 2010.
<http://www.twitpocalypse.com>

Sources Cited

1.
Metz, Cade. “Twitter: It's the end of the sysadmin as we know it”. The
Register, May 4, 2010.
<http://www.theregister.co.uk/2010/05/04/twitter_and_the_end_of_the_sysad
min_as_we_know_it/>

2.
Metz, Cade. “Twitter jilts Ruby for Scala” The Register, April 1, 2009.
http://www.theregister.co.uk/2009/04/01/twitter_on_scala/

3.
Riley, Duncan. “Twitter Fails Macworld Keynote Test” TechCrunch, Jan 2008.
<http://techcrunch.com/2008/01/15/twitter
-
fails
-
macworld
-
keynote
-
test/>

3.
“Ad hoc”. Wikipedia

4.
“memcached”. Wikipedia

5.
“Twitter Twitpocalypse Status”. Accessed May 6, 2010.
<http://www.twitpocalypse.com>