pgmemcache - 2ndQuadrant

pridefulauburnData Management

Dec 16, 2012 (4 years and 8 months ago)

201 views

Protecting the irreplaceable | f-secure.com
pgmemcache

Hannu Valtonen 2010-06-30

Agenda



Slideware



Me


memcached



pgmemcache



When not to use
pgmemcache



When to use it


Future


Demo


Questions
© F-Secure

July 3, 2010

2
Me



Senior Software Engineer @ F-Secure Labs


Backend architect dealing with exponential malware growth


Large distributed systems /
PostgreSQL
databases


(
http://www.f-secure.com/weblog
)


pgmemcache
Maintainer since 2009


Occasional Open Source contributor (
PostgreSQL
,
libmemcached
,
pylibmc
, Wine etc)
© F-Secure

July 3, 2010

3
memcached




memcached
is a high-performance, distributed memory object
caching system, generic in nature, but originally intended for use in
speeding up dynamic web applications by alleviating database load.“

memcached.org



Used by many large sites


Client bindings for most languages


Originally developed for
LiveJournal
in 2003 by Brad Fitzpatrick
© F-Secure

July 3, 2010

4
memcached



Key / Value store


Memcache
is a
CACHE



data is volatile – DON’T rely on it being there


Security aspects


No encryption



No authentication (SASL introduced in 1.4.3)


Usually IP firewalling
© F-Secure

July 3, 2010

5
pgmemcache
history



Created in 2004 by Sean
Chittenden
(1.0)


Based on Sean’s
libmemcache
(without the
d
)


Neil Conway took over in 2007 (1.1)


I took over in 2009 (2.x)


Rewritten on top of
libmemcached
(the one with the
d
)


Frequent releases
© F-Secure

July 3, 2010

6
So why should you use
memcache
?


You keep on getting more and more customers


Your
DBs
are just not scaling for the read load


No matter how many boxes or how big the boxes you buy they’re
not enough


Most sites aren’t like this


Your boss would like spend less on scaling and more towards his
bonus / golden parachute

© F-Secure

July 3, 2010

7
When shouldn’t you use
pgmemcache
?


You just plain don’t need caching


If this is the case, cheers, move on


Ok, you do need caching but you don’t want to use it for these


Purely for reading data within DB functions and transactions


Rather use client side
memcached
calls - your DB boxes cost a lot of money,
don’t use them for this


You don’t have
PostgreSQL
around


Otherwise you obviously want to use it


© F-Secure

July 3, 2010

8
Light remark about previous slide
FEEL FREE TO IGNORE THE
ADVICE ON THE PREVIOUS
SLIDE


© F-Secure

July 3, 2010

9
Ok, but why
pgmemcache



pgmemcache
“sees” the persistent data change


Proximity to
persistent
data and its
transformations


Creation (add/set) & Change (set/replace) & Deletion (delete)


Less sync issues


Multiple writers often problematic


DB changes
memcached
at commit time


DB invalidates the cache


Clients don’t have to worry about this
© F-Secure

July 3, 2010

10
Ok, but why
pgmemcache
(continued)


Often far less code to maintain on client side


Especially if more than one client app using the same data


Other clients often only do reads when using
pgmemcache

© F-Secure

July 3, 2010

11
pgmemcache
requirements


Nothing funny


PostgreSQL
8.4.x


Broke compatibility with 8.3.x in 2.x series


Libmemcached
0.39 and 0.40


Libmemcached
keeps on breaking API compatibility and we have to cope with it


See
README.pgmemcache
for a comprehensive list
© F-Secure

July 3, 2010

12
pgmemcache
basics



libmemcached
bindings for
PostgreSQL



Very similar to
MySQL’s

memcached

UDFs



Supports


memcached
commands


set / get (single/multi) / delete / replace /
incr
/ stats


libmemcached

behaviors



NOREPLY / NO_BLOCK etc (20+ different ones)


Returning
PostgreSQL
return values out of these
© F-Secure

July 3, 2010

13
libmemcached



APIs in constant flux



Backward incompatible changes introduced often


Only maintained C client API for
memcached



Release early – Release often


Brian Aker is the maintainer
© F-Secure

July 3, 2010

14
libmemcached



Supports different
behaviors



TCP
recv
timeouts


Different hashing mechanisms


Poor man’s replication


USE_UDP


NO_BLOCK / NO_REPLY


See man
memcached_behavior_get
for a complete listing
© F-Secure

July 3, 2010

15
pgmemcache

config



postgresql.conf



custom_variable_classes
= '
pgmemcache



pgmemcache.default_servers
='localhost:11211’ #default port 11211


pgmemcache.default_behavior
='MEMCACHED_BEHAVIOR_NOR
EPLY:1,MEMCACHED_BEHAVIOR_NO_BLOCK:1’
© F-Secure

July 3, 2010

16
pgmemcache
future stuff


Support for new
libmemcached
features


SASL authentication in CVS


Packaging for
pgmemcache
(.
deb
support already there)


Better documentation


Run time get/set
behavior



Automatic scripts for trigger generation


Possibly with the JSON type coming in
PostgreSQL
9.1


Far future


SQL / MED
© F-Secure

July 3, 2010

17
Demo time


Demo time
© F-Secure

July 3, 2010

18
Questions?


Questions?
© F-Secure

July 3, 2010

19