Mappying Open Source

chantingrompΚινητά – Ασύρματες Τεχνολογίες

10 Δεκ 2013 (πριν από 3 χρόνια και 4 μήνες)

61 εμφανίσεις

Mappy
-
ing

Open Source

09
/
21
/
2013
,
FOSS4G2013










@
AudreyMalherbe

Audrey Malherbe


Head of
geo

platform


at

Mappy

audrey.malherbe
@
mappy.com

367 millions

route
requests

in 2012

4 millions

referenced

business

Mappy
,
a brand of
SoLocal

Group

About 10 millions

monthly

users


27 millions

monthly

visits


Provide route planner and local services on the web and
mobiles devices





Feedback on the overhaul of
the mapping engine of
Mappy

around
Mapnik

and
PostGIS

Pla
n



The
path


Our
journey


Feedback



Les enjeux du projet

The
path

The path


starting

point



Background


Map

processing

for 16
years

with

TomTom
/AND
databases


Maps

are
used

by
Mappy

& French
YellowPages

media


«

home
-
made

»
processing

mastered

but
ageing

The future


Repositionnning

map

to fit new
requirements

: the
purpose

of the
map

is

now

to support content


Open
-
source
-
oriented

strategy

(
moving

towards
)


Anticipate

emerging

challenges (
vector

tiles
, HD
maps
, relief, ...)



Geographic

data

SDK Mobile

Mappy

API AJAX

Mappy

Stack

Mappy

Engine

Templates

Stack

Mappy

Cache

Data
processing

Mostly

pre
-
computation

Duration

: 10
days

Non standard

proprietary

format

12 zoom
levels

Software system

Fully

mastered

(
behaviour
)

Some

dead

code

Costly

evolutions

Non
negociable

performances

6 Billions
req
/
month

BACK OFFICE

PRODUCTION

The path
-

goals


Create

a
map

design able to support content


Be
ready

to release
with

the new version of
our

website

fr.mappy.com



Reduce

the
computational

time


address

weaknesses

of the
platform

and
keep in mind its strengths


Streamlining

map

tile

production


Standardizing

(REST API, etc.)


Build

for the future (Able to
scale

and support new services)




Take

a
Deep

Breath
.

Start
Again
.

Geographic


data

API AJAX
NewMappy

SDK Mobile

New
Mappy

KyotoCabinet

KyotoCabinet

New
platform


in production

Reduced

Preparation

Time

2days (
vector
) + 2days (relief)

Standard

Live code

Mapnik

contribution

Reduced

operating
cost

BACK OFFICE

PRODUCTION


Le chemin

Our
journey

O
ur

journey

-

Starting point


Heterogeneous knowledge of the team


on open source technologies


Training &
getting

help
from

expert


Bring

a culture of open source



Split
project


Migrate

brick by brick in
order

to
learn

and master the
platform



«

Eat

our

own

food

»


Test
it

fast

Innophi

Geographic


data

SDK Mobile

Mappy

API AJAX

Mappy

Stack

Mappy

Engine

Templates

Stack

Mappy

Cache

API AJAX
NewMappy

SDK Mobile

New
Mappy

KyotoCabinet

KyotoCabinet

BACK OFFICE

PRODUCTION

BACK OFFICE

PRODUCTION

Templates


Step

0

Map

design

By marketing and
Map

designer

Geographic


data

SDK Mobile

Mappy

API AJAX

Mappy

Stack

Mappy

Engine

Templates

Stack

Mappy

Cache

API AJAX
NewMappy

SDK Mobile

New
Mappy

KyotoCabinet

KyotoCabinet

BACK OFFICE

PRODUCTION

BACK OFFICE

PRODUCTION

Step

1

Migration of
database

and
map

engine

Keep

Front interface

Work

on
database

performance

Geographic


data

SDK Mobile

Mappy

API AJAX

Mappy

Stack

Mappy

Engine

Templates

Stack

Mappy

Cache

API AJAX
NewMappy

SDK Mobile

New
Mappy

KyotoCabinet

KyotoCabinet

BACK OFFICE

PRODUCTION

BACK OFFICE

PRODUCTION

Step

2

Cache migration

New APIs

Add

relief /HD
tiles

Work

on performance

and configuration (Cache/Service/APIs)

Geographic

data

SDK Mobile

Mappy

API AJAX

Mappy

Stack

Mappy

Engine

Templates

Stack

Mappy

Cache

API AJAX
NewMappy

SDK Mobile

New
Mappy

KyotoCabinet

KyotoCabinet

BACK OFFICE

PRODUCTION

BACK OFFICE

PRODUCTION

Step

3

Streamlining

databases

Work

on
scaling


Optimizing

process

Our
journey



key

points


Be able to challenge
historical

choices


Reach a better consensus with our customers
(Web/Mobile)


Less

time on
tools

implementation

than

configuration


We

built

prototype in
two

weeks


We

worked

on configuration
during

4
months


Real issue : association of
metadata


with

map

tiles


(
dynamic

display on
map

icons
)



Our
journey



technical

choices


PostGIS
:
transactional

database
/
scalable
/standard
compliant


Mapnik

:
approved

by marketing and
technical

teams!


Tornado

:


High performance
reputation
,
already

used

by mobile team


Varnish

:


Highyield



Already

used


by front teams


Nothing

to

prove



Our
journey




Contribution to
Mapnik



Mappy

was legitimate to participate in an open source project


Develop

an open source
policy

at

Mappy


Arrange
technical

organization

(Open Source Manager,
Github

website

: https://github.com/Mappy)


Contribute

to
Mapnik

:
implementation

of
parallelism

with

PostGIS

(
available

on master). Performance gain: 25%


Some

tools

availables

on
Github

like

pycnik





Feedback

Feedback


Key figures


Production
platform

handles

85% of the
load

(All media
except

Mobile YP)


14 servers:


Varnish

: 2 /
Tornik

: 8 /
PostGIS

: 4


Output
without

cache:


Peak

240
req
/s


Average

throughput

150
req
/s


Output
with

cache: 2500
req
/s (
peak
)


Cache Hit Ratios: 94%


Cache
preload
: 2 millions
objects


Postgis

database
: 75 Go


Relief
database
: 300 Go


Feedback


team


Positive


Team motivation,
proud

and in control of
our

new
platform


PostGIS

and
Mapnik

tools


Contribution/open source world


Negative


Difficult

to
identify

correct
projects

splitting


Contribute

lately

to
Mapnik


Contributing

requires

available

time (discussion
with

community
/ pull
requests
)


Deployment

by
our

customer

not as
fast

as
expected



Thank

you


@
AudreyMalherbe

https
://
github.com
/
Mappy