140 Proof Serves Ads on Social Streams at Lightning Speed Using New Relic

moneygascityInternet and Web Development

Dec 8, 2013 (3 years and 6 months ago)


140 Proof Serves Ads

on Social Streams at
Lightning Speed Using
New Relic
Founded in 2009 by Jon Elvekrog and John Manoogian III, 140 Proof

is the first premium advertising platform for social media connecting millions

of digital shoppers and sharers via real time interest graph targeting.

140 Proof Interest Graph targeting fuses social relevance with cutting-edge
patented technology. The 140 Proof Targeting API reads the public social
stream data and creates an idealized “persona” for each user. Targeted

ads are then served to users based on their persona. By analyzing hundreds
of thousands of public data points, 140 Proof uses their Relevance Engine
combined with a REST API to select the most appropriate 140 character

ads for social software users.
Brands create ads via 140 Proof that appear above or beside the social stream of relevant users fitting specific

social “personas”. Ads are natively integrated into social and mobile apps and can be replied to and/or shared,
potentially multiplying the reach of the ad and lowering the advertiser’s effective cost. To deliver awesome
performance for all their customers, 140 Proof’s API
engineering team maintains a geo-distributed cluster

of servers to deliver ads at lightning speed to any location.
140 Proof’s customers include Chevrolet, Microsoft,
Disney, Infiniti, Levi’s, Coke, AT&T, Pantene, Verizon,

Best Buy, and Victoria’s Secret.

Ruby, Rails, Redis, Resque, Memcache, MySQL, Nginx,
Unicorn, Git, and Pivotal Tracker.
case study: 140 Proof
At a Glance
Social advertising API
San Francisco, New York,
Los Angeles, Chicago, and Detroit
Customer Since:
Use Case:
Ad Serving API
Why New Relic:
Knew it was the
leading Rails APM and considered best
of breed.
“If we had to launch a new business without New
Relic, I think the first thing we would do is invent New
Relic, because it’s become such a critical part of our
deployment and monitoring workflow. It makes our
process consistently successful and helps us drive
product performance. It’s essential to our success.”
—John Manoogian III, Co-Founder

and CTO of 140 Proof.
140 Proof’s core challenge is delivering the most relevant ad as fast as possible. Running a successful social
advertising business requires processing huge amounts of data from the Interest Graph to determine which ads

to serve social users – a true big data challenge if ever there was one. Complicating this further, ads need to appear
at lightning speed or social users will simply move on to the next activity without ever seeing the ad. To both crunch
big data and deliver highly targeted ads to social sites instantaneously, 140 Proof depends on consistent high
performance from three custom web applications they’ve developed:
The Automatic Optimization application, affectionately known internally as the ‘Gator’ (short for Alligator, itself a
mishearing of ‘Allocator’,) a continuous optimizer for advertising campaigns. At any given moment, hundreds of
campaigns are competing to be the most relevant possible ad for a user.
Only one ad, however, can be shown at a time. It’s the Gator’s job to keep
track of dozens of targeting criteria for each active customer campaign
data and its intended audiences, and to continuously recalculate these
parameters based on campaign effectiveness and various feedback loops.
The Effective Delivery Ad Serving API handles rapid delivery of ads to
applications. It uses public social data (the Interest Graph) to match the
most relevant ad campaigns to the right active users. Ads are currently
served in over 50 different social apps across Facebook, Twitter and other
social sites. The Ad Serving API receives the most traffic of any 140 Proof
app; every user who signs into a social app generates a new API request,
every second of the day. Social software is always active. This 24/7
operation requires processing massive amounts of data to determine what
kind of ad to present to each user. Performance monitoring is critical to the
API since it needs to serve every ad instantaneously when the user signs in
or the opportunity is lost.
The Proof of Performance application delivers the internal metrics that are essential to brand advertising customers
who demand deep metrics to measure engagement and proof of campaign performance. 140 Proof relies on New
Relic to monitor this critical application that the advertising operations team and account management teams use
to query massive datasets to produce the proof of performance metrics. 140 Proof’s traffic and load is driven by
patterns of social engagement. In addition to traditional times — like early morning, lunch, after work and late evening
— there can be huge traffic spikes driven by events like the Super Bowl (10,000 Tweets per second in the last three
minutes), the Olympics (116,000 Tweets per minute during the closing ceremonies), political elections, weather
events, entertainment events, etc. Elasticity, scalability and top performance with minimal load are all critical to
achieving success in this environment.
From the company’s start, 140 Proof knew they needed excellent real time performance monitoring and management
if they were going to succeed in the social advertising business. Early on in the company’s development, they chose
New Relic for application performance monitoring and management to help them achieve their goals.

Since switching to New Relic in 2010, 140 Proof has evolved their use of the service and has integrated it into their
unique continuous deployment process. The company appreciates New Relic’s rich feature set and relies heavily on
its real time metrics.
As John Manoogian III, 140 Proof Co-Founder and CTO, says, “Transaction Traces have been invaluable to us.
Modern web software development, in the age of ubiquitous frameworks like Rails and plugin-rich environments like
Ruby, is all about connecting pieces in just the right configuration. When writing software, especially social software
on the web, you’re not just testing one system but the cross product of many open source tools developed by many
people for many different use cases. If done right, it’s faster development than ever. If done wrong, look out. Like it
or not, no matter who you are, if you’re writing apps for the web, your fate is tied to code developed by people other
than you. New Relic has helped us smooth out the complexity in this environment by letting us isolate and remove
any external dependencies that could potentially slow making the most relevant API match.”
“With New Relic, we’re able to quickly identify and remediate a slow query that is degrading API performance,”
continues John. “Inefficient queries can be optimized and
decoupled from the main API loop by moving them into
delayed jobs, and that ‘Kaizen’ (continuous improvement)
feedback loop, powered in part by New Relic, has yielded
massive performance improvements over time.”
John and his engineering team place a high value on New
Relic’s real time performance management strengths, as
well as how New Relic presents information. “New Relic
lets us slice and dice our API cluster reporting to get real
time views of our performance as seen from the rest of the
world, which is critical for us,” observes John. “The quality
and readability of the graphs are generally better than
anything else we’ve seen.”
140 Proof relies on Average Response Time monitoring
to ensure that relevant ads are served within the targeted
30-millisecond response time – which they achieve.
Equally important is their Apdex score. In fact, the team developed a Google Chrome extension that turns red if their
score falls below the acceptable threshold, an ever-present early warning system.
The 140 Proof engineering team is proud of their continuous deployment workflow, which blasts development
builds with real production web traffic in a controlled environment, to ensure that no code can ever degrade API
performance in production. And New Relic is key to monitoring the heath of these builds. The team calls this cluster
of machine instances its ‘Canary’ server, an allusion to canaries sent into coal mines to alert miners of potential
danger. “We developed the Canary process about a year after the business really started growing quickly,” John
explains. “The team was growing, API traffic was through the roof, and with the amount of daily deploys we push,
we needed a better way to test and validate code continuously, rather than post-launch. We also knew that systems
under heavy production load behave much differently than in any idealized testing context. So we came up with the
“Our company is built around engineering culture,”
comments John. “We believe in building, doing,
measuring, and iterating. Engineers like to own their
destiny, including the performance of their code.
New Relic’s web UI makes it easy for us to link
responsibility for performance to the team through
every step of the lifecycle. It’s definitely saved us
headcount and salary dollars…”
—John Manoogian III, Co-Founder

and CTO of 140 Proof.
John continues, “When an engineer checks into Master (using Git), that commit gets automatically deployed to
a Canary server. A proportional slice of live traffic from the production servers gets replayed against the Canary
codebase in real time. We run New Relic on both the Canary and the Production clusters to collect and compare
performance information from both environments in real time. We know the exact impact of any code change well
before it ever makes it to production.” The team loves having a “view into the future” via the Canary, and it’s a key
element in their ability to consistently achieve the company’s application performance quality and performance goals

Using New Relic, 140 Proof has maintained a high-velocity, rapid deployment model, even while pushing code
changes to their API cluster several times an hour. And since adopting New Relic in 2010, the company has achieved
a four-fold improvement in response time.
Using Interest Graphs to develop persona based online ad marketing programs is a classic big data analysis
challenge. 140 Proof relies on New Relic APM to monitor performance on this application. Additionally as an
engineering-driven culture, New Relic helps the team move quickly to proactively identify and correct problems before
they reach the production environment. “Our company is built around engineering culture,” comments John. “We
believe in building, doing, measuring, and iterating. Engineers like to own their destiny, including the performance
of their code. New Relic’s web UI makes it easy for us to link responsibility for performance to the team through
every step of the lifecycle. It’s definitely saved us headcount and salary dollars, but most of all, the ownership and
accountability that every engineer has just makes us better. New Relic is critical in that process since it provides every
engineer the information they need about their code and how it performs.”
Manoogian concludes, “Engineers who gravitate to startups value the ability to learn quickly and not be boxed in. We
want to preserve as much of that kind of atmosphere as we can while we grow and expand our business. New Relic
is pretty critical to that. When you remove the dichotomy of ops and engineering, everyone is on the same team and
there’s no way to point fingers at someone else so it really drives ownership. You don’t have an excuse if your code
doesn’t work. If you give people the ability to monitor and manage themselves, they have to be 100% accountable.”
issue is on the database side or web service side. That simple piece of information gives you a head start and saves
you time.”
About New Relic
New Relic, Inc. is the all-in-one web application performance management provider for the cloud and the data center.
Its SaaS solution, which combines real user monitoring, application monitoring, and availability monitoring in a single
solution built from the ground up, changes the way developers and operations teams manage web application
performance in real time. More than 25,000 organizations use New Relic to optimize over 53 billion daily metrics
captured. newrelic.com
© Copyright 2012, New Relic, Inc. All rights reserved. All trademarks, trade names, service marks and logos referenced herein belong to their respective companies. 08.06.12