D4.3 (WP4): Heterogeneous super-cluster ... - RELEASE Project

bolivialodgeInternet and Web Development

Dec 14, 2013 (4 years and 7 months ago)


A High-Level Paradigm for Reliable Large-Scale Server Software
A Specic Targeted Research Project (STReP)
D4.3 (WP4):Heterogeneous super-cluster Infrastructure
Due date of deliverable:31st December 2012
Actual submission date:9th January 2013
Start date of project:1st October 2011 Duration:36 months
Lead contractor:Erlang Solutions Ltd.Revision:0.1
Purpose:To build and describe a framework that enables automatic deployment and monitoring
of an Erlang/OTP application into a given number of Erlang nodes running in a heterogeneous cloud
 We revisited the CCL architecture presented in deliverable D4.2,moving from a command-line
tool to a fully-blended virtualization infrastructure.
 We moved from a homogenous deployment environment to a heterogeneous one.
 We presented the RELEASE project at two major conferences (HiPEAC and Codemotion),col-
laborated with a team from a parallel FP7 project (Prowess) and started a thesis project about
the implementation of a CCL plugin.
Conclusion:In this deliverable we have introduced a revised architecture for the developed pro-
totype such that we are now capable of deploying Distributed Erlang applications on dierent Cloud
infrastructures.The introduced architectural changes pave the way for adding control and monitoring
functionalities which will enable users to deploy Distributed Erlang applications in a heterogeneous
infrastructure,based on capability prole matching.
Project funded under the European Community Framework 7 Programme (2011-14)
Dissemination Level
Restricted to other programme participants (including the Commission Services)
Restricted to a group specied by the consortium (including the Commission Services)
Condential only for members of the consortium (including the Commission Services)
ICT-287510 (RELEASE) 9th January 2013 1
Heterogeneous super-cluster Infrastructure
Roberto Aloi <roberto.aloi@erlang-solutions.com>
Enrique Fernandez <enrique.fernandez@erlang-solutions.com>
Torben Homann <torben.hoffmann@erlang-solutions.com>
1 Introduction 3
2 From a Command Line Tool to a Fully-blended Virtualization Infrastracture 3
2.1 Evolving the CCL Architecture................................3
2.2 The CCL Components.....................................4
2.3 Cross-component Communication...............................5
2.4 The Heterogeneity Challenge.................................7
3 Implementation 7
3.1 A solution to the heterogeneity problem:Deltacloud....................7
3.2 A RESTful Interface for CCL.................................8
3.3 Code Injection in Erlang....................................9
4 Interacting with CCL 9
4.1 Prerequisites..........................................9
4.2 Overview............................................10
4.3 Deploying and monitoring Erlang clusters with CCL....................10
5 Dissemination and Related Work 12
6 Conclusions and Future Work 14
A The oam.cong le 15
Glossary 17
ICT-287510 (RELEASE) 9th January 2013 2
Executive Summary
The main objective of this deliverable (D4.3) is to move from an homogeneous architecture which only
allows the user to deploy Erlang applications on a single Cloud infrastructure,to a heterogeneous one
which,given a Distributed Erlang Application,allows users to automatically deploy the application
into heterogeneous clusters running in dierent cloud environments.To succeed in our journey from
an homogeneous infrastructure to a heterogeneous one,we have heavily revised the architecture of the
developed prototype.In the previous deliverable (D4.2) we presented a command-line interface that
has now been transformed into a full- edged operations and maintenance application.At the heart of
the prototype lies Deltacloud,an open-source Apache project enabling the communication to dierent
Cloud providers via a common interface.The AMQP protocol is used to ensure a scalable and ecient
cross-component communication.Last but not least,the developed prototype exposes a RESTful
interface;atop of which sits a web dashboard intended for easing the deployment and management
of Erlang applications on heterogeneous super-clusters.The revised architecture along with all the
other newly implemented features presented in this deliverable represent a clear step forward towards
a broker layer (CCL) capable of creating,managing and dynamically scaling super-clusters of smaller
heterogeneous clusters,based on capability prole matching.
ICT-287510 (RELEASE) 9th January 2013 3
1 Introduction
Deliverable D4.3 is part of the Work Package WP4 of the Release project.The Work Package,titled
Scalable Virtualization Infrastructure,is aimed at providing a broker layer (CCL) capable of creating,
managing and dynamically scaling heterogeneous super-clusters of Erlang nodes,based on capability
prole matching.More specically,deliverable D4.3 focuses on heterogeneous environments,as op-
posed to the homogeneous environments analyzed as part of deliverable D4.2.It also assumes a static
conguration for the cluster (e.g.in terms of nodes composing the cluster),rather than relying on a
dynamic,capability-driven approach which will be considered during deliverable D4.4.
The document is structured as follows:
Section 2 describes the evolution of CCL from a command-line tool to a fully-blended virtualization
infrastructure and introduces the new CCL architecture,analyzing the CCL components and their
Section 3 gives an overview of the technologies used to implement CCL,explaining the reasons
behind their adoption.
Section 4 provides an introduction to using CCL through the CCL Dashboard.It walks the reader
through a simple scenario in which a user wants to build,deploy and monitor a minimalistic Erlang
application into a homogeneous cluster.
Section 5 lists the conferences where the work conducted as part of this deliverable has been pre-
sented.It also introduces some work performed in relation to CCL.
Section 6 contains our conclusions and some notes about the expected future work.
2 From a Command Line Tool to a Fully-blended Virtualization In-
In the past few months,CCL has evolved from a simple command-line tool to a fully-blended virtual-
ization infrastructure,consisting of several inter-connected Erlang nodes.In this section we describe
this evolution process and we present the components making up the CCL architecture,with focus on
the concept of heterogeneity,driving force of this deliverable.
2.1 Evolving the CCL Architecture
In the RELEASE deliverable D4.2[22] we described CCL as a command-line tool used by the end user
to build and deploy Erlang applications into the Amazon EC2 cloud environment.The communication
between the CCL node and the other Erlang nodes composing a cluster was mainly happening over SSH
channels,not taking direct advantage of the Erlang distribution protocol.In the same deliverable we
introduced for the rst time the concept of a esl-core (hidden) node,an Operations and Maintenance
(or,more simply,O&M) node responsible of monitoring and operating on the other Erlang nodes in
the cluster.
We have recently decided to split the esl-core node into two separate entities:ccl-oam and ccl-
cluster,as depicted in gure 1.Ccl-oam runs within the same environment as the other Erlang nodes
and can therefore take direct advantage of the Erlang Distribution protocol while interacting with the
rest of the cluster.On the contrary,we place the ccl-cluster nodes outside that environment,laying
ICT-287510 (RELEASE) 9th January 2013 4
the foundation for the concept of a super-cluster,which will be exploited as part of the D4.4 (i.e.
Capability-driven Cluster On Demand) deliverable.
Figure 1:Overview of CCL Architecture
Figure 1 depicts an overview of the current CCL architecture.Let's analyze the CCL components
in detail and the way they communicate each other.Please note that a dashed arrow in the gure
represents that the connected components communicate via Remote Procedure Call (RPC),normally
implemented over the AMQP[21];while a straight line means the communication happens over the
Erlang Distribution Protocol.
2.2 The CCL Components
CCL-API The CCL-API node represents the entry point to the CCL ecosystem.This component
can be seen as an orchestration layer,used to interact with the rest of the system.Interaction with users
and other CCL components is performed via the exposed RESTful and RPC interfaces,respectively.
CCL-API is also responsible for maintaining a global database up-to-date.This database contains
information such as user accounts,cloud service providers credentials linked to each user account,in-
formation about the available clusters in the system,etc.The CCL-API node is in charge of forwarding
any incoming request to the CCL component responsible for handling it (e.g.it forwards a deploy a
new Erlang node request to the CCL-Cluster node responsible for managing the cluster where the
new Erlang node is going to be deployed on).A normal deployment of CCL could consist of several
CCL-API nodes and by a HTTP proxy balancing the load across them.
CCL-CLUSTER This component is responsible for managing a logical cluster.In CCL terms,a
cluster is a set of homogeneous Erlang nodes deployed within the same Cloud infrastructure,all of them
running the same Erlang release (e.g.Riak).A CCL-Cluster node handles all the requests related to
the creation and deletion of Erlang nodes on the cluster it is responsible for.When it comes to deploying
an Erlang application,this component can also be seen as a cache that will serve the source code of
ICT-287510 (RELEASE) 9th January 2013 5
the application to the Erlang nodes that are going to be running the application,avoiding then having
to access the project repository where the application is hosted as many times as nodes are deployed.
CCL-O&M The CCL-O&M component consists of a hidden Erlang node[2].In Distributed Erlang,
hidden nodes (started with the -hidden command line ag) are commonly used to inspect the status
of a system in an unobtrusive way.In fact,connections between hidden nodes and other nodes are
not transitive.Using the Distributed Erlang protocol,the CCL-O&M node is responsible of injecting
a lightweight Erlang process (an agent) into each of the other Erlang nodes belonging to the cluster.
Injected processes represent a backdoor,used by CCL-O&M to control and monitor a cluster.
2.3 Cross-component Communication
In the previous development phase of CCL we did not require any complex messaging since all
we had was a command-line interface that could be used for deploying a given distributed Erlang
application into a set of machines running on the Amazon EC2 cloud environment.Nevertheless,as
already described in Section 2.1,CCL has evolved into a multi-component application which does now
require a more complex messaging protocol.To this end,we have decided upon using RabbitMQ [24],
an open-source message broker that implements the AMQP protocol [21],for all the cross-component
communication.One thing that makes RabbitMQ interesting for us is its simplicity for clustering
several RabbitMQ servers on the same local network as a single logical broker,making the whole
messaging infrastructure more scalable and resilient to failures.
Figure 2:AMQP messaging model
In order to better understand how CCL benets from using RabbitMQ,it is important to grasp
how RabbitMQ and AMQP work.Figure 2 illustrates in brief the AMQP model where:
1.messages are published by producers to exchanges;
2.exchanges then distribute these messages to queues according to some routing rules;
3.either messages are delivered to consumers subscribed to queues,or consumers directly pull mes-
sages from queues
This does not end here,though.The AMQP protocol features dierent built-in exchange types
in uencing how messages are routed to queues.Four exchange types are dened in AMQP:
ICT-287510 (RELEASE) 9th January 2013 6
Direct exchanges They deliver messages to queues based on the specied routing key,which makes
it ideal for unicast communication.A queue is bound to an exchange with a routing key,which is then
matched when a message arrives to the exchange.If both routing keys are the same,the message is
delivered to the queue.
Fanout exchanges They deliver messages to all queues that are bound to it regardless of the routing
Topic exchanges They treat the routing key as a list of zero or more words,delimited by'.',and
supports special wild characters,such as'*'and'#',matching a single word or zero or more words,
Header exchanges They route messages to queues not accordingly to a routing key,but accordingly
to a routing algorithm specied using a special argument,which can either be set to all or any.Queues
are bound to this exchange with a table of arguments and and will only receive messages if all or any of
their binding arguments - depending on the policy set on the exchange - matches the routing argument
of the message.
Now that we know how the AMQP protocol works we can move on and see how we have integrated
RabbitMQ into CCL.
Figure 3:AMQP messaging model in CCL
 The ccl.api queue is used to remotely invoke functions in the CCL-API node.Analogously,
ccl.fsuper-clusterg.fclusterg and ccl.fsuper-clusterg.fclusterg.foamg queues are used to remotely
invoke functions in the corresponding cluster and OAM node,respectively.
ICT-287510 (RELEASE) 9th January 2013 7
 All ccl.fsuper-clusterg.fclusterg queues are bound to the ccl.fsuper-clusterg fanout exchange,
such that all messages sent to the latter will be forwarded to all the slaves deployed within the
encompassing cluster.
 The ccl.fsuper-clusterg queue is shared among all cluster nodes within the same cluster.It is a
task queue used to balance the load among all available clusters.
 The ccl.fsuper-clusterg.status is a queue where all the clusters periodically send their status
updates.This information is used by CCL-API to check the availability of the dierent clusters.
2.4 The Heterogeneity Challenge
By looking at the current Cloud Infrastructure oering,one thing is clear - cloud interoperability has
not been one the main drivers in building these infrastructures -.Due to the lack of cooperation
between the main cloud service providers where the leading vendors in the Cloud market have only
focused on promoting their own,incompatible Cloud technologies,customers of current Cloud infras-
tructure oerings are locked into a single Cloud infrastructure.Since the big players seem not to be
interested in providing a common framework for accessing Cloud services from dierent sources,several
standardisation initiatives have popped up in order to help out with this.It is clear that having such
common framework would allow Cloud customers to easily migrate their Cloud resources between cloud
service providers,compare the characteristics of dierent Cloud oerings in order to choose the one
that better suits their requirements,etc.In the context of the RELEASE project,the main benet for
CCL would be the possibility to deploy Erlang nodes on dierent Clouds without the burden of having
to deal with several,non-consistent interfaces.
The implementation of CCL presented in D4.2 [22] was strictly tightened to the Amazon EC2 Cloud
infrastructure oering.The major challenge we agreed on facing in the next phase of the project was
to move from a homogeneous environment to a heterogeneous one where clusters were not necessarily
Amazon-based.After all,the ultimate goal of the Release WP4 is to provide a broker layer (CCL)
capable of creating,managing and dynamically scaling super-clusters of smaller heterogeneous clusters,
based on capability prole matching.
In this deliverable we explore the possibility of creating heterogeneous super-clusters of on-demand
clusters provisioned from competing cloud service providers,making it possible to combine groups
of instance types,possibly from dierent providers,into a larger super-cluster.As a result of this
exploratory work,we present a revised CCL architecture,such that it now allows the deployment of
Erlang releases not only on Amazon EC2,but also on other competing Cloud infrastructures.In the
next section we describe in detail how the problem has been solved.
3 Implementation
The purpose of this section is to give an overview of the technologies used to implement the CCL
components presented in the previous sections and to illustrate some of the reasons for using them.
3.1 A solution to the heterogeneity problem:Deltacloud
In the previous release of CCL we made use of erlcloud [15],a popular open-source Erlang library,for
deploying Erlang applications on Amazon EC2.Sadly,due to its lack of support for other cloud service
providers and our aim at supporting an heterogeneous infrastructure involving multiple cloud service
providers,we have been forced to dismiss erlcloud and to consider other alternatives enabling us to
interact not only with Amazon EC2,but also with other big players such as OpenStack,OpenNebula
and Eucalyptus,to cite some.After having evaluated a set of dierent options including OCCI [12]
ICT-287510 (RELEASE) 9th January 2013 8
and CIMI [10] and based on the maturity of the projects and the simplicity in terms their ease of
integration into CCL,we have nally decided upon using Deltacloud[13],an open-source Apache project
that implements a simple REST-based API for accessing a broad list of cloud service providers or,in
words of its authors,an API that abstracts the dierences between cloud service providers.
Figure 4:Deltacloud Integration
In order to programmatically interact with Deltacloud,and due to the fact that there was no Erlang
client available at the time of this writing,we have implemented our own simple Erlang wrapper
for Deltacloud enabling us to programmatically manage Virtual Machines on any of the supported
backends.Figure 4 illustrates how CCL makes use of Deltacloud for deploying Erlang nodes on several
Cloud infrastructures.
Assuming we want to deploy a Virtual Machine on Amazon EC2,all we need to do is start a Delta-
cloud service running the EC2 driver (this can be done by issuing the following command:deltacloudd
-i ec2) and interact with the implemented wrapper as illustrated in the following lines:
% Starts the Deltacloud Erlang client.
{ok,Client} = erldcloud:start(Username,Password,Endpoint),
% Creates an instance using the specified image and hardware profile.
{ok,Instance} =
erldcloud:create_instance(#erldcloud_instance{image ="ami-fac0cd8e",
% Deletes the previously created VM.
ok = erldcloud:delete_instance(Instance#erldcloud_instance.id,Client).
3.2 A RESTful Interface for CCL
All of the CCL functionalities are exposed to the outer world through a RESTful (REpresentational
State Transfer) interface.Real-time data are pushed to clients using the Websocket[18] protocol.Cow-
boy,a small,fast and modular HTTP server written in Erlang[20],has been used to dispatch HTTP
ICT-287510 (RELEASE) 9th January 2013 9
requests to Erlang handlers and to handle websocket connections.On the top of the REST API of-
fered by Cowboy,we built a sample web application (the CCL dashboard - see section 4).To build
the CCL Dashboard we made use of the Twitter Bootstrap[23] framework and the jQuery[19] library.
A special note goes to the MVC library backbone.js[11],which allowed us to represent our data as
Models,revealing a perfect t for the RESTful API exposed by Cowboy.Finally,the Javascript library
Highcharts.js[5] has been used for creating the charts visible in the CCL dashboard.
3.3 Code Injection in Erlang
We have mentioned that the ccl-oam node injects some code into the Erlang nodes belonging to the
cluster.This is achieved by means of the following two lines of code:
The Erlang Remote Procedure Call services allow us to evaluate a function call on a remote Erlang
node.In the above case we are remotely invoking two functions:
The rst function call makes use of the code module.The module is a mere interface to the Erlang
code server,a process which deals with the loading of compiled code into a running Erlang runtime
system.This function is used to load object code on the remote Erlang node.The argument Bin
contains the object code for the module Mod.Filename is only used by the code server to keep a record
of from which le the object code for Mod comes.
The second function call starts a gen
server (the agent) on the remote node.The agent will be a
backdoor for the ccl-oam node.Please note how the ccl-oam node passes a reference to itself to the
remote node,using the built-in function node().This reference will be used by the agent to communicate
back with the ccl-oam node.It also plays a role in case of hostile take-over (i.e.when the ccl-oam node
is replaced by a dierent node,for example after an outage).
The problematic part of using this approach was to understand when a remote node was ready
to accept some injected code.Using a standard net administration routine such as net
was not sucient and we had to explicitly verify that system processes such as the code
loader and
sup were running on the remote node,before the code injection could be considered suc-
4 Interacting with CCL
This section gives you an introduction to using CCL through the CCL Dashboard.We walk through a
simple scenario in which the user wants to build,deploy and monitor a minimalistic Erlang application
into a homogeneous cluster.
4.1 Prerequisites
We assume that the Erlang application written by the end user follows the principles and directory
structure specied by the Erlang/OTP middleware[3].We also assume that the repository containg the
application is hosted on GitHub in the form of a private project and it uses rebar[6] as its build-tool.
Using rebar standard commands it is possible to generate an Erlang release[4] out of the application
source code.
ICT-287510 (RELEASE) 9th January 2013 10
Figure 5:The CCL dashboard
4.2 Overview
The Erlang application we are going to deploy is a sample Erlang distributed application,named m8ball
and written by Fred Hebert,author of the\Learn You some Erlang"book[16].The application consists
of a fortune teller server which answers all of its users'questions.A copy of the m8ball application is
available at the following URL:
The tasks required by the end user to achieve his objectives are summarized as follows:
1.Include a CCL conguration le (i.e.a oam.config le) in the m8ball repository
2.Link the CCL user account to a GitHub account
3.Create a logical cluster
4.Create a node type
5.Deploy one or several Erlang nodes
6.Launch one or several Erlang nodes
7.Gather information from each of the deployed nodes
8.Eventually stop one or several Erlang nodes
Let's analyze the above steps in detail.
4.3 Deploying and monitoring Erlang clusters with CCL
Include a CCL conguration le in the repository To customize the behaviour of CCL for a
specic Erlang application,it is possible for the user to include a conguration le,also known as the
oam.config le,in the root directory of the application repository.Even if conguration les currently
require to be written in Erlang terms,we plan to change the format of the conguration le into a more
generic syntax,less bound to the Erlang language.
The content of the oam.config le for the m8ball application is reported here.
ICT-287510 (RELEASE) 9th January 2013 11
%% -*- mode:erlang -*-
{compile_cmds,["make","make rel"]}.
{start_cmds,["make start"]}.
The metrics parameter contains a list of the application-specic metrics (and the respective polling
frequencies) that the user intends to monitor.In our case,we want to know how many questions are
asked to our fortune teller server.We also communicate to CCL the list of commands required to build
and start the m8ball Erlang release and the cookie[1] required by CCL to communicate with the m8ball
nodes.For more information about the format of a CCL conguration le and the available options
please refer to the A appendix.
Link the CCL user account to a GitHub account If the Erlang application is hosted on GitHub
in the form of a private project,the user needs to authenticate with GitHub.This operation,from an
end-user perspective,is as simple as going to a user prole page in the CCL dashboard and clicking on
a Link to GitHub button.This causes the user to be redirected to the github.com pages,where he can
authorize the CCL application to access its GitHub account (an application-specic token is generated
and passed to CCL).Using this approach,based on the OAuth[9] open protocol,we avoid storing any
sensitive information - such as the GitHub user credentials - on the CCL servers.
Create a node type Using the navigation bar,the user can create a new node type.When creating a
new node type,the user can select a name
,a description and a source type.We currently support two
source types:github and local.The modular internal architecture of CCL allows us to easily extend the
supported source types,whenever this is required.Picking the github option allows the user to insert
the actual location,in terms of GitHub account and repository name,where the m8ball application
Deploy one or several Erlang nodes It is now possible for the user to use the navigation bar to
\create a new node".He picks m8ball as the node type and a quantity of nodes which he wants to
deploy for that node type.A progress bar shows the status of the deployment operation.
Launch one or several Erlang nodes The newly deployed node appears in the list of available
nodes.The user can start the given node by clicking on the start button.Once started,the node is
marked as up.
Gather information from each of the deployed nodes By clicking on a given node name,the
bottom part of the screen shows detailed information about the selected node.Information span from
standard info,such as the node uptime and the list of application - or releases - running on the node,
to the custom metrics specied in the oam.config le.
Eventually stop one or several Erlang nodes Other actions are available on the deployed nodes,
such as stopping a node or deleting it.
Name will be used as part of the node name for the deployed nodes
ICT-287510 (RELEASE) 9th January 2013 12
Figure 6:CCL:creating a new node type
Figure 7:CCL:Node is up and running
5 Dissemination and Related Work
In October 2012,the RELEASE project has been presented at the HiPEAC Computing Systems Week
[17] in Ghent,Belgium.Particular attention has been reserved to CCL and the Scalable Virtualization
Infrastructure described in this document.The slides presented at the HiPEAC conference are available
online at:
In November 2012,CCL and RELEASE have been included in a Erlang and the Cloud talk at the
Codemotion 2012[8] conference,held in Venice,Italy.The slides from the Codemotion conference are
available online at:
During the development of CCL,we have been collaborating with the Prowess team at Erlang So-
lutions.Prowess[14] is a project funded by the Seventh Framework Programme (FP7) of the European
ICT-287510 (RELEASE) 9th January 2013 13
Figure 8:CCL:Gathering metrics
Community.As part of the project,aimed at\developing property-based testing for web services and
Internet applications in order to achieve a real improvement of testing eciency",the Prowess team is
working on a distributed load testing tool for massive systems.The tool,written in Erlang,will take
copious advantage of the features provided by CCL,both in terms of deployment and monitoring of
Erlang clusters of nodes.Using CCL,it will be possible to incrementally add loader nodes to an Erlang
cluster,until a desired load has been generated.The work which will be conducted on CCL as part of
the RELEASE deliverable D4.4 (capability-driven cluster on demand) will allow users of the Prowess
team to automate the cloud backend selection based on a cost-capability model and to collect real-time
statistics about system performance during load tests.
During the past few months,we have also collaborated with many commercial Online Service
Companies.This allowed us to gather a series of requirements which,if implemented,will allow CCL
to play a game-changer role in the context of deployment and monitoring for Erlang applications.
Among the requirements:
 Erlang VM ne-tuning via the CCL dashboard
 Assisted hot-code upgrades
 Aided migration from a staging environment to a production environment
 Execution bulk commands on a cluster of Erlang nodes
 Support for sending patches (revisited Erlang modules) to Erlang nodes
 SNMP support
 Setting thresholds and alarms for given metrics
 Log aggregation (e.g.SASL logs) facilities
 Historical data browsing
 Live tracing
 Auto-scaling
ICT-287510 (RELEASE) 9th January 2013 14
 Crash dumps storage
These requirements are currently undergoing a revision and prioritization process.Some of them
will be added to CCL as part of the next RELEASE deliverables.Please refer to the 6 section for more
In addition to the above,a thesis project is currently being conducted as part of a collaboration
between the University of Catania - Italy - and Erlang Solutions.The thesis project is aimed at the
inclusion of the Bencherl scalability benchmarking suite - one of the former RELEASE deliverables -
into CCL,in the form of a CCL plugin.The plugin will allow the allocation of Bencherl resources (i.e.
additional Erlang nodes) and the launch of benchmarks from the CCL dashboard,allowing the user to
assess the scalability of an Erlang application without the need of installing the Bencherl suite.It also
allows to congure the Bencherl suite through the CCL dashboard,in a graphical manner.
6 Conclusions and Future Work
In this deliverable we have presented how we have tackled the heterogeneity challenge.To this end,
we have introduced a revised architecture for CCL which has allowed us to move from an Amazon-
dependant command-line interface to a fully- edged operations and maintenance application capable
of deploying Distributed Erlang applications on dierent Cloud infrastructures.From a prototype
development stand-point,we have also made good progress,introducing several important changes
such as the development of an Erlang wrapper for interacting with Deltacloud,the adoption of the
AMQP protocol for the cross-component communication and the implementation of a RESTful interface
enabling end-users to easily interact with the developed prototype.All the good progress made in this
deliverable allows us to focus now on the upcoming challenges including adding control and monitoring
functionalities which will enable user to deploy Distributed Erlang applications based on capability
prole matching.
ICT-287510 (RELEASE) 9th January 2013 15
A The oam.cong le
When using CCL to deploy,control and monitor a distributed Erlang application,the developer can
specify some conguration options which are used by the framework to ne-tune its behaviour for the
specic application.The conguration options,expressed in Erlang Terms format,need to be specied
in an optional oam.config le,located in the root directory of the application repository.Appropriate
default values for the options are assumed if they have not been included in the conguration le.A
sample oam.config le follows.
%% -*- mode:erlang -*-
{compile_cmds,["make","make rel"]}.
{start_cmds,["make start"]}.
The following table contains a list of all the available conguration options,accompanied by their
respective default values.
List of custom metrics,
espressed in a slightly mod-
ied version of the Erlang
MFA format (see below)
List of commands which will
be executed to build an Erlang
[\./rebar get-deps",\./rebar
compile",\./rebar generate"]
List of commands which will
be executed to start an Erlang
Erlang cookie used for commu-
nication between the ccl-oam
node and the other nodes be-
longing to the cluster
Flag used for auto-generating
names for Erlang nodes
Table 1:CCL Conguration options,meanings and default values
A special note goes to the metrics parameter,used to express custom metrics.Every entry of the
list looks like a tuple of four elements:
Currently we only support counter as the metrics type,but we are adding support for gauges,
histograms,histories and meters.The Name parameter is used to identify the metric and to label it in
ICT-287510 (RELEASE) 9th January 2013 16
the CCL dashboard.PollingFrequency indicates how often - in seconds - the metric has to be polled.
The last argument (MFA
) is a reference to the Erlang function that has to be called on the node to
extract the given metric.In the next release of CCL we plan to introduce an auto-discovery mechanism
for metrics,whenever this is applicable.For example,the user could use folsom[7] to expose Erlang
events and metrics.In this case CCL will be able to automatically discover all of those metrics,without
any explicit conguration required.
MFA stands for\Module,Function and Arguments"
ICT-287510 (RELEASE) 9th January 2013 17
cloud service provider A company that oers infrastructure as a ser-
vice to other businesses or individuals.
cluster A logical abstraction representing a set of Er-
lang nodes running the same Erlang applica-
tion,deployed on the same administrative do-
heterogeneous super-cluster A super-cluster where clusters are deployed
on dierent administrative domains from
competing cloud service providers.
homogeneous super-cluster A super-cluster where all clusters are de-
ployed on the same administrative domain.
super-cluster A logical abstraction representing a set of
ICT-287510 (RELEASE) 9th January 2013 18
[1] Ericsson AB.Distributed Erlang - Security.
[2] Ericsson AB.Erlang - Hidden Nodes.
[3] Ericsson AB.Erlang/OTP Applications.
[4] Ericsson AB.Erlang/OTP Releases.
[5] Highsoft Solutions AS.Highcharts JS.
[6] Inc.Basho Technologies.Rebar.
[7] Boundary.Folsom.
[8] Codemotion.Codemotion Venice 2012.
[9] OAuth Community.OAuth.
[10] Inc.Distributed Management Task Forum.CIMI.
[11] DocumentCloud.backbone.js.
[12] Open Grid Forum.Open Cloud Computing Interface (OCCI).
[13] Apache Software Foundation.Deltacloud.
[14] FP7.Prowess Project.
[15] Gleber.Cloud Computing APIs For Erlang.
[16] Fred Hebert.Learn You Some Erlang - Distributed Applications.
[17] HiPEAC.European Network of Excellence on High Performance and Embedded Architecture and
[18] IETF.The Websocket protocol.
ICT-287510 (RELEASE) 9th January 2013 19
[19] The jQuery Foundation.jQuery.
[20] Nine Nines.Cowboy.
[22] RELEASE Project.D4.2 (WP4):Homogeneous Cluster Infrastructure.
[23] Inc.Twitter.Twitter Bootstrap.
[24] VMware.RabbitMQ.