Integrating AWS CloudFormation with Opscode Chef

Arya MirInternet and Web Development

Feb 13, 2012 (5 years and 8 months ago)

2,163 views

AWS CloudFormation gives you an easy way to create the set of resources such as Amazon EC2 instance, Amazon RDS database instances and Elastic Load Balancers needed to run your application. The template describes what resources you need and AWS CloudFormation takes care of how: provisioning the resources in an orderly and predictable fashion, handling and recovering from any failures or issues. For more details of using AWS CloudFormation


Integrating AWS CloudFormation with Opscode Chef

1

Integrating

AWS CloudFormation

with Opscode Chef


AWS CloudFormation gives you an easy way to create the set of resources such as Amazon EC2
instance, Amazon RDS database instances and Elastic Load Balancers needed to run
y
our
application.
The template describes
what

resources you need and AWS CloudFormat
ion takes care
of
how:
provisioning the resources in an orderly and predictable fashion, handling and recovering
from any
failures or issues. For more details
of using AWS CloudFormation
see the
AWS
Clo
udFormation product detail page
.

While AWS CloudFormation takes care of provisioning all the resources, it raises the obvious
question of how
your

application
software
is deployed, configured and executed on the Amazon
EC2 instances. There are many options
, each of which has implications on how quickly your
application is
ready

and how flexible you need to be in terms of deploying new versions of the
software.
The remainder of this document shows how to use the AWS CloudFormation bootstrap
helper scripts to

deploy application
s

using Opscode Chef. It builds on the features of AWS
CloudFormation introduced in the whitepaper
Bootstrapping Applicati
ons With AWS
CloudFormation
.

Overview

Chef is
an open source infrastructure automation solution from Opscode, written in Ruby, that
allows you
to automate the configuration of your systems and the applications that sit on top of it.
It is a client/server
application where clients pull the configuration
s

from the chef server, and all
the work to transform the configuration into an instance that serves a function takes place on the
instance itself.

By building on the ability to define packages, files and co
nfiguration in a CloudFormation template,
you can use AWS CloudFormation in 2 ways:

1)

To deploy and configure a Chef Server
.

2)

To bootstrap clients with the Chef client software

In addition to the basic bootstrapping, AWS CloudFormation provides
an Ohai
plug
-
i
n to the Chef
client software that allows configuration keys needed to bootstrap applications (such as the host
name of the database to connect to) to be defined in the template metadata. This provides an easy,
one
-
stop
-
shop to define the role for a server

and any runtime properties that the role may require.

Chef can be used in 3 different ways to automate your application provisioning:

Chef Solo:

This is a standalone version of Chef that runs locally on your node and does not need
access to a Chef server.

In this mode, you need to download all the configuration information such
as cookbooks and role definitions to the local disk

or provide a link to a tarball with all the
cookbooks, roles etc. to the chef
-
solo command line tool
.

Chef Server and Client
: Che
f clients run locally on your instances and connect to a Chef server that
contains all the package information, package dependencies and configuration information
required to install and provision your applications.


Integrating AWS CloudFormation with Opscode Chef

2

Hosted Chef
: OpsCode provides a highly a
vailable, highly scalable Chef Server in the Cloud. As with
Chef Server, your Chef clients connect to the Hosted Chef service to retrieve application package
information and configuration.

For more details of the different Chef configurations see the
Opscode Chef wiki
.

In this document, we cover how to use Chef Solo and Chef Server to deploy applications using AWS
CloudFormation to bootstrap Chef. Whether you are using Chef Solo or the Chef client o
n an EC2
instance to deploy your applications, you need to first install Chef Solo and then, if you are using
Chef Client, use Chef Solo to bootstrap the client software. Using the template metadata and the
AWS CloudFormation helper scripts, the remainder
of this section shows how you can bootstrap
Chef Solo, the Chef Client and even the Chef Server from a base AMI allowing you to automatically
build out a Chef
-
based deployment from scratch.

Note: This document assumes some familiarity with Opscode Chef
;

it

is not intended to be a
tutorial on using Chef.

Using Chef Solo

Chef Solo can be used to deploy Chef cookbooks and roles without a dependency on a Chef Server.
Chef Solo can be installed via a Ruby Gem package
;

however, it requires a number of other
dependent packages to be installed. By using resource metadata and the AWS CloudFormation
helpers, you can deploy Chef Solo on a base AMI via Cloud
-
init. The following Snippet shows the
metadata and Cloud
-
init script

you might use to bootstrap Chef Solo:

"MyInstance": {


"Type": "AWS::EC2::Instance",


"Metadata" : {


"AWS::CloudFormation::Init" : {


"config" : {


"packages" : {


"rubygems" : {


"chef" : [ "0.10.2" ]


},



"yum" : {


"gcc
-
c++" : [],


"ruby
-
devel" : [],


"make" : [],


"autoconf" : [],


"automake" : [],


"rubygems" : []


}


},


"files" : {


"/etc/chef/solo
.rb" : {


"content" : { "Fn::Join" : ["", [


"log_level :info
\
n",


"log_location STDOUT
\
n",


"file_cache_path
\
"/var/chef
-
solo
\
"
\
n",


"cookbook_path
\
"/var/chef
-
solo/cookbooks
\
"
\
n",



"json_attribs
\
"/etc/chef/node.json
\
"
\
n",


"recipe_url
\
"", { "Ref" : "RecipeURL" }, "
\
"
\
n"


]] },


"mode" : "000644",


"owner" : "root",


"group" : "wheel"


}
,


"/
etc
/chef/
node.json
" : {


"content" : { "Fn::Join" : ["", [


:


]] },


"mode" : "000644",


"owner" : "root",


"group" : "wheel"


}
,



}


}


}


},


Integrating AWS CloudFormation with Opscode Chef

3


"Properties": {


"Secu
rityGroups": [ { "Ref": "EC2SecurityGroup" } ],


"ImageId": ami
-
xxxxxxxx,


"UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [


"#!/bin/
ba
sh
\
n",


"
/opt/aws/bin/
cfn
-
init
-
s ", { "Ref" : "AWS::StackName" }, "
-
r
WebServer

",


"
--
access
-
key ", { "Ref" : "HostKeys" },


"
--
secret
-
key ", {"Fn::GetAtt": ["HostKeys", "SecretAccessKey"]},


"
--
region

", { "Ref" : "AWS::Region" },

"

&&
",


"
chef
-
solo
\
n
"
,


"
/opt/aws/bin/
cfn
-
signal

e
$? '", { "Ref" : "WaitHandle" }, "'
\
n"


]]}},


"KeyName": { "Ref": "KeyName" },


"InstanceType": { "Ref": "InstanceType" }


}

}


Things to note about the template snippet:



The various development tools needed to install Chef Solo are installed

via Yum. The Yum
packages are processed and installed prior to the Ruby Gems packages being installed by
cfn
-
init.



Chef Solo is installed via
RubyGem
s
. This template installs a specific version of the Chef
code,
namely 0.10.2.



Chef Solo is configured usin
g 3 pieces of data: The
/etc/chef/solo.rb

file contains the basic
configuration. It points to
/etc/chef/node.json

that contains the configuration to apply to
the node and the
recipe_url

which is a URL reference to a Chef recipe to download.



The Cloud
-
init
script downloads and installs the AWS CloudFormation helper scripts, calls
cfn
-
init

to deploy the packages and files needed for Chef Solo and then runs Chef Solo to
deploy the Chef recipe defined by the recipe_url. If any errors occur in the configuration,

the script calls
cfn
-
signal

to fail the stack creation.

For more details on configuring Chef Solo see the
OpcCode Chef wiki
.

The following template shows the full template needed to deploy

a h
ighly available (multi
-
AZ)
instance of

WordPress
on a base Amazon Linux AMI
using
an Amazon RDS database as the
backend store via Chef Solo
:

{


"AWSTemplateFormatVersion" : "2010
-
09
-
09",




"Description" : "Install a WordPress deployment using an Amazon

RDS database instance for storage. This template demonstrates using the AWS
CloudFormation bootstrap scripts to install Chef Solo and then Chef Solo is used to install a simple WordPress recipe. **WARN
ING** This template creates
an Amazon EC2 instance and

an RDS database. You will be billed for the AWS resources used if you create a stack from this template.",




"Parameters" : {




"KeyName" : {


"Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instances",


"Type" : "String"


},




"FrontendType" : {


"Description" : "Type of Frontend instance",


"Type" : "String",


"Default" : "m1.small",


"AllowedValues" : [ "t1.micro", "m1.small", "m1.large", "m1.xlarge", "m2.xlarge", "m2.2xlarge", "m2.4xlarge", "c1.medium", "c
1.xlarge",
"cc1.4xlarge" ],


"ConstraintDescription" : "must be a valid EC2 instance type."



},




"GroupSize":
{


"Default": "1",


"Description" : "The default number of EC2 instances for the frontend cluster",


"Type": "Number"


},




"MaxSize": {


"Default": "1",


"Description" : "The maximum number of EC2 instances for the front
end",


"Type": "Number"


Integrating AWS CloudFormation with Opscode Chef

4


},




"DBClass" : {


"Default" : "db.m1.small",


"Description" : "Database instance class",


"Type" : "String",


"AllowedValues" : [ "db.m1.small", "db.m1.large", "db.m1.xlarge", "db.m2.xlarge", "d
b.m2.2xlarge", "db.m2.4xlarge" ],


"ConstraintDescription" : "must select a valid database instance type."


},




"DBName": {


"Default": "wordpress",


"Description" : "The WordPress database name",


"Type": "String",


"MinLength": "1",


"MaxLength": "64",


"AllowedPattern" : "[a
-
zA
-
Z][a
-
zA
-
Z0
-
9]*",


"ConstraintDescription" : "must begin with a letter and contain only alphanumeric characters."


},




"DBUser": {


"Default": "admin",



"NoEcho": "true",


"Description" : "The WordPress database admin account username",


"Type": "String",


"MinLength": "1",


"MaxLength": "16",


"AllowedPattern" : "[a
-
zA
-
Z][a
-
zA
-
Z0
-
9]*",


"ConstraintDescription" : "must begi
n with a letter and contain only alphanumeric characters."


},




"DBPassword": {


"Default": "admin",


"NoEcho": "true",


"Description" : "The WordPress database admin account password",


"Type": "String",


"MinLength": "
1",


"MaxLength": "41",


"AllowedPattern" : "[a
-
zA
-
Z0
-
9]*",


"ConstraintDescription" : "must contain only alphanumeric characters."


},




"MultiAZDatabase" : {


"Default" : "false",


"Description" : "If true, creates a Multi
-
AZ deployment of the RDS database",


"Type" : "String",


"AllowedValues" : [ "true", "false" ],


"ConstraintDescription" : "must be either true or false."


}


},




"Mappings" : {


"
AWSInstanceType2Arch" : {


"t1.micro" : { "Arch" : "32" },


"m1.small" : { "Arch" : "32" },


"m1.large" : { "Arch" : "64" },


"m1.xlarge" : { "Arch" : "64" },


"m2.xlarge" : { "Arch" : "64" },


"m2.2xlarge" : { "
Arch" : "64" },


"m2.4xlarge" : { "Arch" : "64" },


"c1.medium" : { "Arch" : "32" },


"c1.xlarge" : { "Arch" : "64" },


"cc1.4xlarge" : { "Arch" : "64" }


},


"AWSRegionArch2AMI" : {


"us
-
east
-
1" : { "32" : "ami
-
7f4
18316", "64" : "ami
-
7341831a" },


"us
-
west
-
1" : { "32" : "ami
-
951945d0", "64" : "ami
-
971945d2" },


"eu
-
west
-
1" : { "32" : "ami
-
24506250", "64" : "ami
-
20506254" },


"ap
-
southeast
-
1" : { "32" : "ami
-
74dda626", "64" : "ami
-
7edda62c" }
,


"ap
-
northeast
-
1" : { "32" : "ami
-
dcfa4edd", "64" : "ami
-
e8fa4ee9" }


}


},




"Resources" : {



"CfnUser" : {


"Type" : "AWS::IAM::User",


"Properties" : {


"Path": "/",


"Policies": [{


"PolicyName": "root",


"PolicyDocument": { "Statement":[{


"Effect":"Allow",


"Action":"cloudformation:DescribeStackResource",


"Resource":"*"


}]}


}]


}


},



"HostKeys" : {



"Type" : "AWS::IAM::AccessKey",


Integrating AWS CloudFormation with Opscode Chef

5


"Properties" : {


"UserName" : {"Ref": "CfnUser"}


}


},




"ElasticLoadBalancer": {


"Type": "AWS::ElasticLoadBalancing::LoadBalancer",


"Properties": {


"Listeners": [


{ "InstancePort": 80,


"Protocol": "HTTP",


"LoadBalancerPort": "80"


}


],


"HealthCheck": {


"HealthyThreshold": "2",


"Timeout": "5",


"Interval": "
10",


"UnhealthyThreshold": "5",


"Target": "HTTP:80/wp
-
admin/install.php"


},


"AvailabilityZones": {


"Fn::GetAZs": { "Ref": "AWS::Region"}


}


}


},




"WebServerGroup": {


"Type":

"AWS::AutoScaling::AutoScalingGroup",


"Properties": {


"LoadBalancerNames": [{ "Ref": "ElasticLoadBalancer" }],


"LaunchConfigurationName": {"Ref": "LaunchConfig"},


"AvailabilityZones": {


"Fn::GetAZs": { "Ref": "AWS::
Region" }


},


"MinSize": "0",


"MaxSize": { "Ref" : "MaxSize" },


"DesiredCapacity" : { "Ref" : "GroupSize" }


}


},




"LaunchConfig": {


"Type": "AWS::AutoScaling::LaunchConfiguration",


"Metadata" : {


"AWS::CloudFormation::Init" : {


"config" : {


"packages" : {


"rubygems" : {


"chef" : [ "0.10.2" ]


},


"yum" : {


"gcc
-
c++" : [],


"ruby
-
devel" : [],


"make" : [],


"autoconf" : [],


"automake" : [],


"rubygems" : []


}


},


"files" : {


"/etc/ch
ef/solo.rb" : {


"content" : { "Fn::Join" : ["
\
n", [


"log_level :info",


"log_location STDOUT",


"file_cache_path
\
"/var/chef
-
solo
\
"",


"cookbook_path
\
"/var/chef
-
solo/cookbooks
\
"",


"json_attribs
\
"/etc/chef/node.json
\
"",


"recipe_url
\
"https://s3.amazonaws.com/cloudformation
-
examples/wordpress.tar.gz
\
""


]] },



"mode" : "000644",


"owner" : "root",


"group" : "wheel"


},


"/etc/chef/node.json" : {


"content" : {


"wordpress" : {


"db"
: {


"database" : {"Ref" : "DBName"},


"user" : {"Ref" : "DBUser"},


"host" : {"Fn::GetAtt" : ["DBInstance", "Endpoint.Address"]},


"password" : {"Ref" : "DBPasswor
d" }


}


},


"run_list": [ "recipe[wordpress]" ]


},


"mode" : "000644",


"owner" : "root",


"group" : "wheel"


}


}



}


}


Integrating AWS CloudFormation with Opscode Chef

6


},


"Properties": {


"InstanceType" : { "Ref" : "FrontendType" },


"SecurityGroups" : [ { "Ref" : "SSHGroup" }, {"Ref" : "FrontendGroup"} ],


"ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" },


{ "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "FrontendType" }, "Arch" ] } ] },


"KeyName" : { "Ref" : "KeyName" },


"User
Data" : { "Fn::Base64" : { "Fn::Join" : ["", [


"#!/bin/bash
\
n",


"/opt/aws/bin/cfn
-
init
-
s ", { "Ref" : "AWS::StackName" }, "
-
r LaunchConfig ",


"
--
access
-
key ", { "Ref" : "HostKeys" },


"
--
secret
-
key

", {"Fn::GetAtt": ["HostKeys", "SecretAccessKey"]},


"
--
region ", { "Ref" : "AWS::Region" }, " && ",


"chef
-
solo
\
n",


"/opt/aws/bin/cfn
-
signal
-
e $? '", { "Ref" : "WaitHandle" }, "'
\
n"


]]}}


}


}
,



"WaitHandle" : {


"Type" : "AWS::CloudFormation::WaitConditionHandle"


},



"WaitCondition" : {


"Type" : "AWS::CloudFormation::WaitCondition",


"DependsOn" : "WebServerGroup",


"Properties" : {


"Handle" : {"Ref" :
"WaitHandle"},


"Timeout" : "600"


}


},




"DBInstance" : {


"Type": "AWS::RDS::DBInstance",


"Properties": {


"Engine" : "MySQL",


"DBName" : { "Ref": "DBName" },


"Port" : "3306",


"MultiAZ" : { "Ref" : "MultiAZDatabase" },


"MasterUsername" : { "Ref": "DBUser" },


"DBInstanceClass" : { "Ref" : "DBClass" },


"DBSecurityGroups" : [{ "Ref": "DBSecurityGr
oup" }],


"AllocatedStorage" : "5",


"MasterUserPassword": { "Ref": "DBPassword" }


}


},




"DBSecurityGroup": {


"Type": "AWS::RDS::DBSecurityGroup",


"Properties": {


"DBSecurityGroupIngress": { "EC2Security
GroupName": { "Ref": "FrontendGroup"} },


"GroupDescription": "Frontend Access"


}


},




"SSHGroup" : {


"Type" : "AWS::EC2::SecurityGroup",


"Properties" : {


"GroupDescription" : "Enable SSH access via port
22",


"SecurityGroupIngress" : [ { "IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22", "CidrIp" : "0.0.0.0/0" } ]


}


},




"FrontendGroup" : {


"Type" : "AWS::EC2::SecurityGroup",


"Properties" : {


"GroupDescription" : "Enable HTTP access via port 80",


"SecurityGroupIngress" : [ { "IpProtocol" : "tcp", "FromPort" : "80", "ToPort" : "80",


"SourceSecurityGroupOwnerId" : {"Fn::GetAtt" : ["ElasticLoadBalancer", "SourceSecurityGr
oup.OwnerAlias"]},


"SourceSecurityGroupName" : {"Fn::GetAtt" : ["ElasticLoadBalancer", "SourceSecurityGroup.GroupName"]}


} ]


}


}


},




"Outputs" : {


"WebsiteURL" : {


"Value" : { "Fn::Join" : ["",

["http://", { "Fn::GetAtt" : [ "ElasticLoadBalancer", "DNSName" ]}, "/"]]},


"Description" : "URL to install WordPress"


},


"InstallURL" : {


"Value" : { "Fn::Join" : ["", ["http://", { "Fn::GetAtt" : [ "ElasticLoadBalancer", "DNSName" ]}
, "/wp
-
admin/install.php"]]},


"Description" : "URL to install WordPress"


}


}

}

Things to note about the template:


Integrating AWS CloudFormation with Opscode Chef

7



This template is capable of creating a highly available, multi
-
AZ deployment of WordPress
.
The web servers are wrapped in an Auto Scaling group that spans all EC2 availability zones
in the region and the RDS database instance can optionally be marked as
multi
-
AZ
. The
instances are fronted by an Elastic Load Balancer and are locked down to acce
pt only HTTP
traffic from the load balancer or SSH connections.



The template can be run in any region, with any instance type for the web server. The
Mappings defined in the template (
AW
S
InstanceType2Arch

and
AWSRegionArch2AMI
)
select

the correct architect
ure and the correct EC2 base Amazon Linux AMI for the given
region.



We create an IAM user with AWS security credentials that is given access call only the
CloudFormation DescribeStackResource API. This user is passed to the
cfn
-
init

helper script
via the E
C2 user data.



An Elastic Load Balancer is created in front of the Auto Scaling group to spread inbound
traffic across the instances. It is configured with a simple health check that pings the
WordPress installation page.



The Auto Scaling group launch confi
guration contains the package and file metadata
needed to deploy Chef solo and configure the cookbook to deploy WordPress.



The
solo.rb file is defined in the template and contains a
recipe_url
that
points to a
WordPress Chef recipe
. The chef recipe being u
sed

has been modified slightly from the
Opscode provided version, allowing the hostname of the RDS database instance to be
passed to the WordPress configuration via the
/etc/chef/node.json

configuration file.



The node.json file defines the attribute to con
figure WordPress along with a run_list for
Chef to deploy.



The stack creation waits for at least 1 instance in the Auto Scaling group to signal the
WaitCondition. If you want to wait for all of the instances in the Auto Scaling group to
become available an
d have WordPress installed, you can use the
Count

property of the
WaitCondition to wait for more than 1 instance.

Abstracting out the “muck” to create a re
-
usable Chef Solo template

With AWS CloudFormation embedded templates, it is possible to create a re
-
usable template that
can bootstrap Chef Solo, making it simple to use Chef Solo to deploy you applications. To do this,
we will split up the previous example into common pieces needed to bootstrap Chef Solo and
pieces needed specifically to deploy WordPres
s.

The following is a re
-
usable template that creates an auto
-
scaled, load balanced application to be
configured via Chef Solo:

{


"AWSTemplateFormatVersion": "2010
-
09
-
09",



"Description": "Sample template to bring up an Auto Scaling group running an ap
plication deployed via Opscode Chef solo. This template is a building
block template, designed to be called from a parent template. A WaitCondition is used to hold up the stack creation until the

application is deployed.
**WARNING** This template creates o
ne or more Amazon EC2 instances and CloudWatch alarms. You will be billed for the AWS resources used if you create a
stack from this template.",



"Parameters": {


"KeyName": {


"Type": "String",


"Description" : "Name of an existing EC2 KeyP
air to enable SSH access to the web server"


},


"RecipeURL" : {


"Description" : "The location of the recipe tarball",


"Type": "String"


},


"EC2SecurityGroup": {


Integrating AWS CloudFormation with Opscode Chef

8


"Default": "default",


"Description" : "The EC2 security group that contains instances that need access to the database",


"Type": "String"


},


"StackNameOrId" : {


"Description" : "The StackName or StackId containing the resource with the Chef configurati
on metadata",


"Type": "String",


"MinLength": "1",


"MaxLength": "128"


},


"ResourceName" : {


"Description" : "The Logical Resource Name in the stack defined by StackName containing the resource with the Chef configurat
ion meta
data",


"Type": "String",


"MinLength": "1",


"MaxLength": "128",


"AllowedPattern" : "[a
-
zA
-
Z][a
-
zA
-
Z0
-
9]*"


},


"InstanceType": {


"Default": "m1.small",


"Description" : "Type of EC2 instance for web server",


"T
ype": "String",


"AllowedValues" : [ "t1.micro", "m1.small", "m1.large", "m1.xlarge", "m2.xlarge", "m2.2xlarge", "m2.4xlarge", "c1.medium", "c
1.xlarge",
"cc1.4xlarge" ],


"ConstraintDescription" : "must contain only alphanumeric characters."


},


"WebServerPort": {


"Default" : "8888",


"Type": "Number",


"Description" : "Port for web servers to listen on"


},


"AlarmTopic": {


"Description": "SNS topic to notify if there are operational issues",


"Type": "String"


},


"DesiredCapacity": {


"Default" : "1",


"Type": "Number",


"MinValue": "1",


"MaxValue": "6",


"Description" : "Port for web servers to listen on"


},


"HealthCheckPath" : {


"Default" :

"/",


"Type" : "String",


"Description" : "Elastic Load Balancing HealthCheck path"


}


},



"Mappings" : {


"AWSInstanceType2Arch" : {


"t1.micro" : { "Arch" : "32" },


"m1.small" : { "Arch" : "32" },


"m1.large"

: { "Arch" : "64" },


"m1.xlarge" : { "Arch" : "64" },


"m2.xlarge" : { "Arch" : "64" },


"m2.2xlarge" : { "Arch" : "64" },


"m2.4xlarge" : { "Arch" : "64" },


"c1.medium" : { "Arch" : "32" },


"c1.xlarge" : { "Arch
" : "64" },


"cc1.4xlarge" : { "Arch" : "64" }


},


"AWSRegionArch2AMI" : {


"us
-
east
-
1" : { "32" : "ami
-
7f418316", "64" : "ami
-
7341831a" },


"us
-
west
-
1" : { "32" : "ami
-
951945d0", "64" : "ami
-
971945d2" },


"eu
-
west
-
1" : { "32" : "ami
-
24506250", "64" : "ami
-
20506254" },


"ap
-
southeast
-
1" : { "32" : "ami
-
74dda626", "64" : "ami
-
7edda62c" },


"ap
-
northeast
-
1" : { "32" : "ami
-
dcfa4edd", "64" : "ami
-
e8fa4ee9" }


}


},



"Resources" : {




"CfnUser" : {


"Type" : "AWS::IAM::User",


"Properties" : {


"Path": "/",


"Policies": [{


"PolicyName": "root",


"PolicyDocument": { "Statement":[{


"Effect":"Allow",


"Action":"cloudform
ation:DescribeStackResource",


"Resource":"*"


}]}


}]


}


},



"HostKeys" : {


"Type" : "AWS::IAM::AccessKey",


"Properties" : {


"UserName" : {"Ref": "CfnUser"}


}


Integrating AWS CloudFormation with Opscode Chef

9


},



"ElasticLoadBalancer": {


"Type": "AWS::ElasticLoadBalancing::LoadBalancer",


"Properties": {


"Listeners": [ {


"InstancePort": { "Ref": "WebServerPort" },


"PolicyNames": [ "p1" ],


"Protocol": "HTTP"
,


"LoadBalancerPort": "80"


} ],


"HealthCheck": {


"HealthyThreshold": "2",


"Timeout": "5",


"Interval": "10",


"UnhealthyThreshold": "5",


"Target": { "Fn::Join": [ "", [ "HTTP:", {
"Ref": "WebServerPort" }, { "Ref" : "HealthCheckPath" } ] ] }


},


"AvailabilityZones": { "Fn::GetAZs" : { "Ref" : "AWS::Region" } },


"LBCookieStickinessPolicy": [ {


"CookieExpirationPeriod": "30",


"PolicyName"
: "p1"


} ]


}


},



"WebServerGroup": {


"Type": "AWS::AutoScaling::AutoScalingGroup",


"Properties": {


"LoadBalancerNames": [ { "Ref": "ElasticLoadBalancer" } ],


"LaunchConfigurationName": { "Ref": "LaunchConfig" },


"AvailabilityZones": { "Fn::GetAZs" : { "Ref" : "AWS::Region" } },


"MinSize": "1",


"MaxSize": "6",


"DesiredCapacity" : { "Ref" : "DesiredCapacity" }


}


}
,



"LaunchConfig": {


"Type": "AWS::AutoScaling::LaunchConfiguration",


"Metadata" : {


"AWS::CloudFormation::Init" : {


"config" : {


"packages" : {


"rubygems" : {


"chef" : [ "0.10.2"
]


},


"yum" : {


"gcc
-
c++" : [],


"ruby
-
devel" : [],


"make" : [],


"autoconf" : [],


"automake" : [],


"rubygems" : []


}


},


"files" : {


"/etc/chef/solo.rb" : {


"content" : { "Fn::Join" : ["", [


"log_level :info
\
n",


"log_location STDOUT
\
n",


"file_cache_p
ath
\
"/var/chef
-
solo
\
"
\
n",


"cookbook_path
\
"/var/chef
-
solo/cookbooks
\
"
\
n",


"json_attribs
\
"/etc/chef/node.json
\
"
\
n",


"recipe_url
\
"", { "Ref" : "RecipeURL" }, "
\
"
\
n"


]] },



"mode" : "000644",


"owner" : "root",


"group" : "wheel"


}


}


}


}


},


"Properties": {


"SecurityGroups": [ { "Ref": "EC2SecurityGroup" } ],


"ImageId": { "
Fn::FindInMap": [ "AWSRegionArch2AMI", { "Ref": "AWS::Region" },


{ "Fn::FindInMap": [ "AWSInstanceType2Arch", { "Ref": "InstanceType" }, "Arch" ] } ]},


"UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [


"#!/bin/bash
\
n
",


"function error_exit
\
n",


"{
\
n",


" /opt/aws/bin/cfn
-
signal
-
e 1
-
r
\
"$1
\
" '", { "Ref" : "WaitHandle" }, "'
\
n",


" exit 1
\
n",


"}
\
n",



"/opt/aws/bin/cfn
-
init
-
s ", { "Ref" : "AWS::StackName" }, "
-
r LaunchConfig ",


"
--
access
-
key ", { "Ref" : "HostKeys" },


"
--
secret
-
key ", {"Fn::GetAtt": ["HostKeys", "SecretAccessKey"]},


"
-
-
region ", { "Ref" : "AWS::Region" }, " || error_exit 'Failed to initialize Chef Solo'
\
n",


"/opt/aws/bin/cfn
-
init
-
s ", { "Ref" : "StackNameOrId" }, "
-
r ", { "Ref" : "ResourceName" },


"
--
access
-
key ", { "Ref" : "HostKeys" },


Integrating AWS CloudFormation with Opscode Chef

10



"
--
secret
-
key ", {"Fn::GetAtt": ["HostKeys", "SecretAccessKey"]},


"
--
region ", { "Ref" : "AWS::Region" }, " || error_exit 'Failed to configure the application'
\
n",



"chef
-
solo
\
n",


"/opt/aws/bin/cfn
-
s
ignal
-
e $? '", { "Ref" : "WaitHandle" }, "'
\
n"


]]}},


"KeyName": { "Ref": "KeyName" },


"InstanceType": { "Ref": "InstanceType" }


}


},



"WaitHandle" : {


"Type" : "AWS::CloudFormation::WaitConditionHandle"


},




"WaitCondition" : {


"Type" : "AWS::CloudFormation::WaitCondition",


"DependsOn" : "WebServerGroup",


"Properties" : {


"Handle" : { "Ref" : "WaitHandle" },


"Count" : { "Ref" : "DesiredCapacity" },


"Timeout" : "600"


}


},



"LockInstancesDown" : {


"Type" : "AWS::EC2::SecurityGroupIngress",


"Properties" : {


"GroupName" : { "Ref": "EC2SecurityGroup" },


"IpProtocol" : "tcp",


"FromPort" : { "Ref" :
"WebServerPort" },


"ToPort" : { "Ref" : "WebServerPort" },


"SourceSecurityGroupOwnerId" : {"Fn::GetAtt" : ["ElasticLoadBalancer", "SourceSecurityGroup.OwnerAlias"]},


"SourceSecurityGroupName" : {"Fn::GetAtt" : ["ElasticLoadBalancer"
, "SourceSecurityGroup.GroupName"]}


}


},



"CPUAlarmHigh": {


"Type" : "AWS::CloudWatch::Alarm",


"Properties": {


"AlarmDescription": "Alarm if aggregate CPU too high ie. > 90% for 5 minutes",


"Namespace": "AWS/EC2",


"MetricName": "CPUUtilization",


"Statistic": "Average",


"Dimensions": [ {


"Name": "AutoScalingGroupName",


"Value": { "Ref": "WebServerGroup" }


} ],


"Period": "60",


"Threshold": "90",


"ComparisonOperator": "GreaterThanThreshold",


"EvaluationPeriods": "1",


"AlarmActions": [ { "Ref": "AlarmTopic" } ]


}


},



"TooManyUnhealthyHostsAlarm": {


"Type": "AWS::CloudWatch::Alarm",


"Properties": {



"AlarmDescription": "Alarm if there are any unhealthy hosts.",


"Namespace": "AWS/ELB",


"MetricName": "UnHealthyHostCount",


"Statistic": "Average",


"Dimensions": [ {


"Name": "LoadBalancerName",


"
Value": { "Ref": "ElasticLoadBalancer" }


} ],


"Period": "300",


"EvaluationPeriods": "1",


"Threshold": "0",


"ComparisonOperator": "GreaterThanThreshold",


"AlarmActions": [ { "Ref": "AlarmTopic" } ]


}



},



"RequestLatencyAlarmHigh": {


"Type": "AWS::CloudWatch::Alarm",


"Properties": {


"AlarmDescription": "Alarm if request latency > ",


"Namespace": "AWS/ELB",


"MetricName": "Latency",


"Dimensions": [ {


"Name": "LoadBalancerName",


"Value": { "Ref": "ElasticLoadBalancer" }


} ],


"Statistic": "Average",


"Period": "300",


"EvaluationPeriods": "1",


"Threshold": "1",


"ComparisonOperator": "G
reaterThanThreshold",


"AlarmActions": [ { "Ref": "AlarmTopic" } ]


}


Integrating AWS CloudFormation with Opscode Chef

11


}


},



"Outputs": {


"URL": {


"Value": { "Fn::Join": [ "", [ "http://", { "Fn::GetAtt": [ "ElasticLoadBalancer", "DNSName" ] }, "/" ] ] },


"Description"

: "Website URL"


}


}

}


Things to note about the template:



This template is capable of creating a highly available, multi
-
AZ deployment of WordPress.
The web servers are wrapped in an Auto Scaling group that spans all EC2 availability zones
in the re
gion and the RDS database instance can optionally be marked as
multi
-
AZ
. The
instances are fronted by an Elastic Load Balancer and are locked down to accept only HTTP
traffic from the load balancer or SSH connections.



The Cloud
-
init script calls
cfn
-
init

twice. The first call references the metadata defined on
the WebServer launch configuration. This metadata describes all of the packages and files
needed to bootstrap Chef Solo. The second call to Chef Solo references a stack and logical
resource name pas
sed in as a template parameter. As you will see later in this section, this
run of
cfn
-
init

is used to define the roles for the host.



To complete a “production
-
style” re
-
usable template, we also define a number of AWS
CloudWatch alarms to monitor the appli
cation.



This template is agnostic to the recipe and the role and can be re
-
used by any number of
application deployments to deploy a highly available multi
-
AZ solution via Chef Solo.

In order to deploy an application via AWS CloudFormation, using the Chef
Solo template above, we
need to define a template that describes the application we want to launch. The following template
uses the embedded template defined above, as well as a re
-
usable RDS database template to
launch a WordPress installation:

{


"AWSTe
mplateFormatVersion": "2010
-
09
-
09",



"Description": "This template demonstrates using Opscode Chef Solo to deploy the WordPress application. It uses embedded temp
lates to build an Auto
Scaling group and a Amazon Relational Database Service database insta
nce. **WARNING** This template creates one or more Amazon EC2 instances, an Amazon
RDS database instance and CloudWatch alarms. You will be billed for the AWS resources used if you create a stack from this te
mplate.",



"Parameters": {


"KeyName": {



"Type": "String",


"Description" : "Name of an existing EC2 KeyPair to enable SSH access to the web server"


},


"InstanceType": {


"Default": "m1.small",


"Description" : "Type of EC2 instance for web server",


"Type": "String
",


"AllowedValues" : [ "t1.micro", "m1.small", "m1.large", "m1.xlarge", "m2.xlarge", "m2.2xlarge", "m2.4xlarge", "c1.medium", "c
1.xlarge",
"cc1.4xlarge" ],


"ConstraintDescription" : "must contain only alphanumeric characters."


},


"Datab
aseType": {


"Default": "db.m1.small",


"Description" : "The database instance type",


"Type": "String",


"AllowedValues" : [ "db.m1.small", "db.m1.large", "db.m1.xlarge", "db.m2.xlarge", "db.m2.2xlarge", "db.m2.4xlarge" ],


"Const
raintDescription" : "must contain only alphanumeric characters."


},


"DatabaseUser": {


"NoEcho": "true",


"Type": "String",


"Default" : "admin",


"Description" : "Test database admin account name",


"MinLength": "1",


"MaxLength": "16",


"AllowedPattern" : "[a
-
zA
-
Z][a
-
zA
-
Z0
-
9]*",


"ConstraintDescription" : "must begin with a letter and contain only alphanumeric characters."


},


Integrating AWS CloudFormation with Opscode Chef

12


"DatabasePassword": {


"NoEcho": "true",


"Type": "String",


"Default" : "admin",


"Description" : "Test database admin account password",


"MinLength": "1",


"MaxLength": "41",


"AllowedPattern" : "[a
-
zA
-
Z0
-
9]*",


"ConstraintDescription" : "must contain only alphanumeric characters."


},


"OperatorEmail": {


"Description": "EMail address to notify if there are operational issues",


"Type": "String"


},


"HighlyAvailable": {


"Default": "true",


"Description" : "Create a solution that spans multiple Avai
lability Zones for high availability",


"Type": "String",


"AllowedValues" : [ "true", "false" ],


"ConstraintDescription" : "must be true or false."


}


},



"Mappings" : {


"RegionMap" : {


"us
-
east
-
1" : { "TemplateLocation" : "https://s3.amazonaws.com/cloudformation
-
templates
-
us
-
east
-
1" },


"us
-
west
-
1" : { "TemplateLocation" : "https://s3.amazonaws.com/cloudformation
-
templates
-
us
-
west
-
1" },


"eu
-
west
-
1" : { "Te
mplateLocation" : "https://s3.amazonaws.com/cloudformation
-
templates
-
eu
-
west
-
1" },


"ap
-
northeast
-
1" : { "TemplateLocation" : "https://s3.amazonaws.com/cloudformation
-
templates
-
ap
-
northeast
-
1" },


"ap
-
southeast
-
1" : { "TemplateLocation" : "https:
//s3.amazonaws.com/cloudformation
-
templates
-
ap
-
southeast
-
1" }


},


"AvailabilityMap" : {


"true" : { "InstanceCount" : "2", "MultiAZDB" : "true" },


"false" : { "InstanceCount" : "1", "MultiAZDB" : "false" }


}


},



"Resources" : {



"AlarmTopic" : {


"Type" : "AWS::SNS::Topic",


"Properties" : {


"Subscription" : [ {


"Endpoint" : { "Ref": "OperatorEmail" },


"Protocol" : "email"


} ]


}


},




"EC2SecurityGroup" : {


"Type" : "AWS::EC2::SecurityGroup",


"Properties" : {


"GroupDescription" : "Open up SSH access",


"SecurityGroupIngress" : [ {


"IpProtocol": "tcp",


"FromPort": "22",


"ToPort": "22",


"CidrIp"
: "0.0.0.0/0"


} ]


}


},



"WordPressFrontEnd" : {


"Type" : "AWS::CloudFormation::Stack",


"Metadata" : {


"Comment" : "Create Wordpress web server farm attached to database.",


"AWS::CloudFormation::Init" : {



"config" : {


"files" : {


"/etc/chef/node.json" : {


"content" : {


"wordpress" : {


"db" : {


"host" : {"Fn::G
etAtt" : ["WordpressDatabase", "Outputs.DBAddress"]},


"database" : "WordPressDB",


"user" : {"Ref" : "DatabaseUser"},


"password" : {"Ref" : "DatabasePassword" }


}


},


"run_list": [ "recipe[wordpress]" ]


},


"mode" : "000644",


"owner" : "root",


"group" : "wheel"


}


}


}


}


},


"Properties" : {


"TemplateURL" : { "Fn::Join" : ["/", [{ "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "TemplateLocation" ]},


"chef
-
solo
-
configuration.template" ]]},


Integrating AWS CloudFormation with Opscode Chef

13


"Parameters" : {



"RecipeURL" : "https://s3.amazonaws.com/cloudformation
-
examples/wordpress.tar.gz",


"HealthCheckPath" : "/wp
-
admin/install.php",


"KeyName" : { "Ref" : "KeyName" },


"InstanceType" : { "Ref" : "Inst
anceType"},


"EC2SecurityGroup" : { "Ref" : "EC2SecurityGroup" },


"StackNameOrId" : { "Ref" : "AWS::StackName" },


"ResourceName" : "WordPressFrontEnd",


"AlarmTopic" : { "Ref" : "AlarmTopic" },



"WebServerPort" : "80",


"DesiredCapacity" : { "Fn::FindInMap" : [ "AvailabilityMap", { "Ref" : "HighlyAvailable" }, "InstanceCount" ]}


}


}


},



"WordpressDatabase" : {


"Type" : "AWS::CloudFormation::Stack",


"Metadata" : {


"Comment" : "Database configuration for Wordpress."


},


"Properties" : {


"TemplateURL" : { "Fn::Join" : ["/", [{ "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS:
:Region" }, "TemplateLocation" ]},


"RDS_MySQL_55.template" ]]},


"Parameters" : {


"DBName" : "WordPressDB",


"DBUser" : { "Ref" : "DatabaseUser" },


"DBPassword" : { "Re
f" : "DatabasePassword" },


"DBInstanceClass" : { "Ref" : "DatabaseType" },


"AlarmTopic" : { "Ref" : "AlarmTopic" },


"EC2SecurityGroup" : { "Ref" : "EC2SecurityGroup" },


"MultiAZ" : { "Fn::FindInMap" :

[ "AvailabilityMap", { "Ref" : "HighlyAvailable" }, "MultiAZDB" ]}


}


}


}


},



"Outputs": {


"URL": {


"Value": { "Fn::GetAtt": [ "WordPressFrontEnd", "Outputs.URL" ] },


"Description" : "URL of the website"


}


}

}


Things to note about the template:



AWS CloudFormation requires that templates that are stored in Amazon S3 be stored in the
region in which the stack is created. This template uses a Mapping to define the location of
the embedded templates based on the reg
ion.

(This restriction will be removed in a future
version of AW
S

CloudFormation).



The template uses an abstract input parameter
HighlyAvailable

to define whether to create
a single availability zone or a multi
-
availability zone solution. A Mapping is used

to define
the number of
Amazon
EC2 webserver instances to create as well as the multi
-
AZ property
of the database.



All of the details of bootstrapping Chef Solo are abstracted out of this template. Only the
/etc/chef/node.json

file is defined in this temp
late to create the WordPress role.

Using Chef Server

on Ubuntu

The Opscode Chef Server is a central location for managing cookbooks, recipes, server roles and
authentication of nodes in your environment. It can be bootstrapped using Chef Solo. The
following template shows how to bootstrap the Chef Server
on a Ubuntu
A
MI
:
{

{


"AWSTemplateFormatVersion": "2010
-
09
-
09",



"Description": "Sample template to bring up an Opscode Chef Server using the BootStrap Chef RubyGems installation. This confi
guration creates and
starts the Chef Server with the WebUI enabled, initializ
es knife and uploads specified cookbooks and roles to the chef server. A WaitCondition is used
to hold up the stack creation until the application is deployed. **WARNING** This template creates one or more Amazon EC2 ins
tances. You will be billed
for the A
WS resources used if you create a stack from this template.",



Integrating AWS CloudFormation with Opscode Chef

14


"Parameters": {


"KeyName": {


"Type": "String",


"Description" : "Name of an existing EC2 KeyPair to enable SSH access to the web server"


},


"CookbookLocation": {


"Type": "String",


"Default" : "https://github.com/opscode/cookbooks/tarball/master",


"Description" : "Location of chef cookbooks to upload to server"


},


"RoleLocation": {


"Type": "String",


"Default" : "https://s3.amazonaws.c
om/cloudformation
-
examples/example_chef_roles.tar.gz",


"Description" : "Location of client roles to upload to server"


},


"InstanceType": {


"Default": "m1.small",


"Description" : "Type of EC2 instance for web server",


"Type": "String",


"AllowedValues" : [ "t1.micro", "m1.small", "m1.large", "m1.xlarge", "m2.xlarge", "m2.2xlarge", "m2.4xlarge", "c1.medium", "c
1.xlarge",
"cc1.4xlarge" ],


"ConstraintDescription" : "must contain only alphanumeric character
s."


}


},



"Mappings" : {


"AWSInstanceType2Arch" : {


"t1.micro" : { "Arch" : "32" },


"m1.small" : { "Arch" : "32" },


"m1.large" : { "Arch" : "64" },


"m1.xlarge" : { "Arch" : "64" },


"m2.xlarge" : { "Arc
h" : "64" },


"m2.2xlarge" : { "Arch" : "64" },


"m2.4xlarge" : { "Arch" : "64" },


"c1.medium" : { "Arch" : "32" },


"c1.xlarge" : { "Arch" : "64" },


"cc1.4xlarge" : { "Arch" : "64" }


},


"AWSRegionArch2AMI" : {



"us
-
east
-
1" : { "32" : "ami
-
06ad526f", "64" : "ami
-
1aad5273" },


"us
-
west
-
1" : { "32" : "ami
-
116f3c54", "64" : "ami
-
136f3c56" },


"eu
-
west
-
1" : { "32" : "ami
-
359ea941", "64" : "ami
-
379ea943" },


"ap
-
southeast
-
1" : { "32" : "
ami
-
62582130", "64" : "ami
-
60582132" },


"ap
-
northeast
-
1" : { "32" : "ami
-
d8b812d9", "64" : "ami
-
dab812db" }


}


},



"Resources" : {



"ChefServerUser" : {


"Type" : "AWS::IAM::User",


"Properties" : {


"Path": "/",


"Policies": [{


"PolicyName": "root",


"PolicyDocument": { "Statement":[{


"Effect":"Allow",


"Action": [


"cloudformation:DescribeStackResource",


"s3:Put"


],



"Resource":"*"


}]}


}]


}


},



"HostKeys" : {


"Type" : "AWS::IAM::AccessKey",


"Properties" : {


"UserName" : {"Ref": "ChefServerUser"}


}


},



"ChefServer": {


"Type": "AWS::EC2::Instance",


"Metadata" : {


"AWS::CloudFormation::Init" : {


"config" : {


"packages" : {


"rubygems" : {


"chef" : [],


"ohai" : []


},



"apt" : {


"ruby" : [],


"ruby
-
dev" : [],


"libopenssl
-
ruby" : [],


"rdoc" : [],


"ri" : [],


"irb" : []
,


"build
-
essential" : [],


Integrating AWS CloudFormation with Opscode Chef

15


"wget" : [],


"ssl
-
cert" : [],


"rubygems" : [],


"git" : [],


"s3cmd" : []


}


},


"sources" : {


"/home/ubuntu/chef
-
repo" : "https://github.com/opscode/chef
-
repo/tarball/master",


"/home/ubuntu/chef
-
repo/cookbooks" : { "Ref" : "CookbookLocation" },


"/home/ubuntu/chef
-
repo/roles" : { "Ref" : "RoleLocation" }


},


"files" : {


"/home/ubuntu/setup_environment" : {


"source" : "https://s3.amazonaws.com/cloudformation
-
examples/setup
-
che
f
-
server
-
with
-
knife",


"mode" : "000755",


"owner" : "ubuntu",


"group" : "ubuntu"


},


"/home/ubuntu/.s3cfg" : {


"content" : { "Fn::Join" : ["", [



"[default]
\
n",


"access_key = ", { "Ref" : "HostKeys" }, "
\
n",


"secret_key = ", {"Fn::GetAtt": ["HostKeys", "SecretAccessKey"]}, "
\
n",


"use_https = True
\
n"


]]},


"mode"

: "000644",


"owner" : "ubuntu",


"group" : "ubuntu"


},


"/etc/chef/solo.rb" : {


"content" : { "Fn::Join" : ["
\
n", [


"file_cache_path
\
"/tmp/chef
-
solo
\
"",



"cookbook_path
\
"/tmp/chef
-
solo/cookbooks
\
""


]]},


"mode" : "000644",


"owner" : "root",


"group" : "root"


},


"/etc/chef/chef.json" : {


"content" : {


"chef_server": {


"server_url": "http://localhost:4000",


"webui_enabled": true


},


"run_list": [ "recipe[chef
-
server::rubygems
-
install
]" ]


},


"mode" : "000644",


"owner" : "root",


"group" : "root"


}


}


}


}


},


"Properties": {


"SecurityGroups": [ { "Ref": "ChefServ
erSecurityGroup" } ],


"ImageId": { "Fn::FindInMap": [ "AWSRegionArch2AMI", { "Ref": "AWS::Region" }, { "Fn::FindInMap": [ "AWSInstanceType2Arch", {

"Ref":
"InstanceType" }, "Arch" ] } ]


},


"UserData" : { "Fn::Base64" : { "Fn::Join"
: ["", [


"#!/bin/bash
\
n",



"function error_exit
\
n",


"{
\
n",


" cfn
-
signal
-
e 1
-
r
\
"$1
\
" '", { "Ref" : "ChefServerWaitHandle" }, "'
\
n",


" exit 1
\
n",


"}
\
n",



"apt
-
get
-
y install python
-
setuptools
\
n",


"easy_install https://s3.amazonaws.com/cloudformation
-
examples/aws
-
cfn
-
bootstrap
-
1.0.tar.gz
\
n",


"cfn
-
init
--
region ", { "Ref" : "AWS::Region" },


"
-
s ", { "Ref" :
"AWS::StackName" }, "
-
r ChefServer ",


"
--
access
-
key ", { "Ref" : "HostKeys" },


"
--
secret
-
key ", {"Fn::GetAtt": ["HostKeys", "SecretAccessKey"]}, " || error_exit 'Failed to run cfn
-
init'
\
n",



"# Bootstr
ap chef
\
n",


"export PATH=$PATH:/var/lib/gems/1.8/bin
\
n",


"ln
-
s /var/lib/gems/1.8/bin/chef
-
solo /usr/bin/chef
-
solo
\
n",


"ln
-
s /var/lib/gems/1.8/bin/chef
-
server /usr/bin/chef
-
server
\
n",


"ln
-
s /var/lib/gems/1.
8/bin/chef
-
server
-
webui /usr/bin/chef
-
server
-
webui
\
n",


"ln
-
s /var/lib/gems/1.8/bin/chef
-
solr /usr/bin/chef
-
solr
\
n",


"ln
-
s /var/lib/gems/1.8/bin/chef
-
expander /usr/bin/chef
-
expander
\
n",


"ln
-
s /var/lib/gems/1.8/bin/knif
e /usr/bin/knife
\
n",


"ln
-
s /var/lib/gems/1.8/bin/rake /usr/bin/rake
\
n",


"chef
-
solo
-
c /etc/chef/solo.rb
-
j /etc/chef/chef.json
-
r http://s3.amazonaws.com/chef
-
solo/bootstrap
-
latest.tar.gz > /tmp/chef_solo.log
2>&1 || error_exit 'F
ailed to bootstrap chef server'
\
n",



"# Setup development environment in ubuntu user
\
n",


"sudo
-
u ubuntu /home/ubuntu/setup_environment > /tmp/setup_environment.log 2>&1 || error_exit 'Failed to bootstrap chef server'
\
n",




"# copy validation key to S3 bucket
\
n",


Integrating AWS CloudFormation with Opscode Chef

16


"s3cmd
-
c /home/ubuntu/.s3cfg put /etc/chef/validation.pem s3://", {"Ref" : "PrivateKeyBucket" } ,"/validation.pem >
/tmp/put_validation_key.log 2>&1 || error_exit 'Failed to put Chef Server validation

key'
\
n",



"# If all went well, signal success
\
n",


"cfn
-
signal
-
e $?
-
r 'Chef Server configuration' '", { "Ref" : "ChefServerWaitHandle" }, "'
\
n"


]]}},


"KeyName": { "Ref": "KeyName" },


"InstanceType": { "Ref": "InstanceType" }


}


},



"ChefServerSecurityGroup" : {


"Type" : "AWS::EC2::SecurityGroup",


"Properties" : {


"GroupDescription" : "Open up SSH access plus Chef Server required ports",


"SecurityGroupIngress" : [


{ "IpProtocol": "tcp", "FromPort": "22", "ToPort": "22", "CidrIp": "0.0.0.0/0" },


{ "IpProtocol": "tcp", "FromPort": "4000", "ToPort": "4000", "SourceSecurityGroupName": { "Ref" :"ChefClientSecurityGroup"

}},


{ "IpProtocol": "tcp", "FromPort": "4040", "ToPort": "4040", "CidrIp": "0.0.0.0/0" }


]


}


},



"ChefClientSecurityGroup" : {


"Type" : "AWS::EC2::SecurityGroup",


"Properties" : {


"GroupDescription" : "G
roup with access to Chef Server"


}


},



"PrivateKeyBucket" : {


"Type" : "AWS::S3::Bucket",


"Properties" : {


"AccessControl" : "Private"


},


"DeletionPolicy" : "Delete"


},



"BucketPolicy" : {


"Type" : "AWS::S3::BucketPolicy",


"Properties" : {


"PolicyDocument": {


"Version" : "2008
-
10
-
17",


"Id" : "WritePolicy",


"Statement" : [{


"Sid" : "WriteAccess",



"Action" : ["s3:PutObject"],


"Effect" : "Allow",


"Resource" : { "Fn::Join" : ["", ["arn:aws:s3:::", {"Ref" : "PrivateKeyBucket"} , "/*"]]},


"Principal" : { "AWS": {"Fn::GetAtt" : ["ChefServerUser", "Arn"]} }


}]


},


"Bucket" : {"Ref" : "PrivateKeyBucket"}


}


},



"ChefServerWaitHandle" : {


"Type" : "AWS::CloudFormation::WaitConditionHandle"


},



"ChefServerWaitCondition" : {


"Type" : "AWS::CloudFormation:
:WaitCondition",


"DependsOn" : "ChefServer",


"Properties" : {


"Handle" : { "Ref" : "ChefServerWaitHandle" },


"Timeout" : "1200"


}


}


},



"Outputs" : {


"WebUI" : {


"Description" : "URL of Opscode chef server WebUI",


"Value" : { "Fn::Join" : ["", ["http://", {"Fn::GetAtt" : [ "ChefServer", "PublicDnsName" ]}, ":4040"]]}


},


"ServerURL" : {


"Description" : "URL of newly created Opscode che
f server",


"Value" : { "Fn::Join" : ["", ["http://", {"Fn::GetAtt" : [ "ChefServer", "PublicDnsName" ]}, ":4000"]]}


},


"ChefSecurityGroup" : {


"Description" : "EC2 Security Group with access to Opscode chef server",


"Value" :

{ "Ref" :"ChefClientSecurityGroup" }


},


"ValidationKeyBucket" : {


"Description" : "Location of validation key",


"Value" : {"Ref" : "PrivateKeyBucket" }


}


}

}



Integrating AWS CloudFormation with Opscode Chef

17

Things to note about the template:



The template has 2 parameters
CookbookLocation

and
RoleLocation
. These parameters are
used to populate the Chef server with a set of cookbooks and roles. The default value of the
CookbookLocation

is set to pull down the set of default cookbooks from the Opscode
GITHub repository. The d
efault value for
RoleLocation
is a sample

role for WordPress.

Here’s the wordpress.rb defining the role:


name "wordpress"

description "Example of installing wordpress from the chef repository"

run_list("recipe[wordpress]")




While the Chef Server is
installed via the Cloud
-
init script running as
root
, once installed,
the Chef Server is configured using a script downloaded as part of the
cfn
-
init
run as the
Ubuntu

user via sudo.



When a Chef Server is created, it generates a validation private key that
must be used by all
Chef Clients that are to connect to the server to generate an initial host
-
specific client. To
allow Chef Clients to be automatically provisioned, without manual intervention, the Chef
Server validation key is stored in a private Amazon

S3 bucket, created in the template.
Clients should fetch the private key from the S3 bucket using valid AWS credentials,
generate their own host
-
specific client key and then delete the server key from the host. To
that end, the AWS user created in the tem
plate not only has rights to call CloudFormation
DescribeStackResources, but can also call S3 Put to upload the validation key.



The validation key is uploaded to S3 using the s3cmd package available from the Ubuntu
repository.



The Chef Server is protected
in its own EC2 security group. Clients must be in the client
security group created by this template in order to communicate with the Chef Server.

Configuring the Chef Server using Knife

As described above, the Chef Server is not only provisioned using th
e previous template, but it is
bootstrapped with a set of cookbooks and roles. Chef Server is managed using the utility
knife

which must be configured to connect to the Chef Server. The following script is run as the
ubuntu

user to configure knife and popu
late the Chef Server.

#!/bin/bash


# Error exit function

function error_exit

{


echo $1


exit 1

}


# Grab the keys

cd /home/ubuntu

mkdir .chef

sudo cp /etc/chef/*.pem .chef

sudo chown
-
R ubuntu:ubuntu .chef


# Configure knife

cd /home/ubuntu

knife
configure
-
i > /tmp/knife_configuration.log 2>&1 <<EOF

/home/ubuntu/.chef/knife.rb

http://localhost:4000

serveruser

chef
-
webui

/home/ubuntu/.chef/webui.pem

chef
-
validator

/home/ubuntu/.chef/validation.pem


Integrating AWS CloudFormation with Opscode Chef

18


EOF


# Set ownership of the repo

sudo chown
-
R ubu
ntu:ubuntu /home/ubuntu/chef
-
repo

rm
-
Rf /home/ubuntu/chef
-
repo/cookbooks/gnu_parallel


# Upload cookbooks and roles

if [ $? ]; then


#Upload cookbooks


knife cookbook upload
-
a
-
o /home/ubuntu/chef
-
repo/cookbooks || error_exit 'Failed to upload cookbook
s'



#Upload roles


cd /home/ubuntu/chef
-
repo


rake roles || error_exit 'Failed to upload roles'

else


error_exit 'Failed to initialize knife'

fi


exit 0


In this sample template and installation script, the Chef Server is populated by pulling down the
cookbooks from a URL on to the server instance and then using knife to load them into the Chef
Server.

Installing the Chef Client on Ubuntu

Once you have a w
orking Chef Server, you can use them to configure your clients. As with the Chef
Server, the Chef Client can be bootstrapped using Chef Solo. The following template creates a
single instance WordPress installation using the WordPress recipe from the standa
rd Opscode
cookbook repository.


{


"AWSTemplateFormatVersion": "2010
-
09
-
09",



"Description": "Sample template to bring up an Opscode Chef Client using the BootStrap Chef RubyGems installation. A WaitCond
ition is used to hold up
the stack creation until

the application is deployed. **WARNING** This template creates one or more Amazon EC2 instances. You will be billed for the A
WS
resources used if you create a stack from this template.",



"Parameters": {


"KeyName": {


"Type": "String",


"D
escription" : "Name of an existing EC2 KeyPair to enable SSH access to the web server"


},


"InstanceType": {


"Default": "m1.small",


"Description" : "Type of EC2 instance for web server",


"Type": "String",


"AllowedValues" : [
"t1.micro", "m1.small", "m1.large", "m1.xlarge", "m2.xlarge", "m2.2xlarge", "m2.4xlarge", "c1.medium", "c1.xlarge",
"cc1.4xlarge" ],


"ConstraintDescription" : "must contain only alphanumeric characters."


},


"ChefServerURL" : {


"Description" : "URL of Chef Server",


"Type": "String"


},


"ChefServerPrivateKeyBucket" : {


"Description" : "S3 bucket containing validation private key for Chef Server",


"Type": "String"


},


"ChefServerSecurityGroup"
: {


"Description" : "Security group to get access to Opscode Chef Server",


"Type": "String"


}


},



"Mappings" : {


"AWSInstanceType2Arch" : {


"t1.micro" : { "Arch" : "32" },


"m1.small" : { "Arch" : "32" },


"m1.
large" : { "Arch" : "64" },


"m1.xlarge" : { "Arch" : "64" },


"m2.xlarge" : { "Arch" : "64" },


"m2.2xlarge" : { "Arch" : "64" },


"m2.4xlarge" : { "Arch" : "64" },


Integrating AWS CloudFormation with Opscode Chef

19


"c1.medium" : { "Arch" : "32" },


"c1.xlarge"
: { "Arch" : "64" },


"cc1.4xlarge" : { "Arch" : "64" }


},


"AWSRegionArch2AMI" : {


"us
-
east
-
1" : { "32" : "ami
-
06ad526f", "64" : "ami
-
1aad5273" },


"us
-
west
-
1" : { "32" : "ami
-
116f3c54", "64" : "ami
-
136f3c56" },


"eu
-
west
-
1" : { "32" : "ami
-
359ea941", "64" : "ami
-
379ea943" },


"ap
-
southeast
-
1" : { "32" : "ami
-
62582130", "64" : "ami
-
60582132" },


"ap
-
northeast
-
1" : { "32" : "ami
-
d8b812d9", "64" : "ami
-
dab812db" }


}


},



"Resources" : {




"ChefClientUser" : {


"Type" : "AWS::IAM::User",


"Properties" : {


"Path": "/",


"Policies": [{


"PolicyName": "root",


"PolicyDocument": { "Statement":[{


"Effect":"Allow",


"Action": [



"cloudformation:DescribeStackResource",


"s3:Get"


],


"Resource":"*"


}]}


}]


}


},



"HostKeys" : {


"Type" : "AWS::IAM::AccessKey",


"Properties" : {


"UserName" : {"Ref": "ChefClientUser"}


}


},



"BucketPolicy" : {


"Type" : "AWS::S3::BucketPolicy",


"Properties" : {


"PolicyDocument": {


"Version" : "2008
-
10
-
17",


"Id" : "ReadPoli
cy",


"Statement" : [{


"Sid" : "ReadAccess",


"Action" : ["s3:GetObject"],


"Effect" : "Allow",


"Resource" : { "Fn::Join" : ["", ["arn:aws:s3:::", {"Ref" : "ChefServerPrivateKeyBuck
et"} , "/*"]]},


"Principal" : { "AWS": {"Fn::GetAtt" : ["ChefClientUser", "Arn"]} }


}]


},


"Bucket" : {"Ref" : "ChefServerPrivateKeyBucket"}


}


},



"ChefClient": {


"Type": "AWS::EC2::Instance",



"DependsOn" : "BucketPolicy",


"Metadata" : {


"AWS::CloudFormation::Init" : {


"config" : {


"packages" : {


"rubygems" : {


"chef" : [],


"ohai" : ["0.6.4"]


},


"apt" : {


"ruby" : [],


"ruby
-
dev" : [],


"libopenssl
-
ruby" : [],


"rdoc" : [],


"ri" : [],


"irb"

: [],


"build
-
essential" : [],


"wget" : [],


"ssl
-
cert" : [],


"rubygems" : [],


"s3cmd" : []


}


},


"files
" : {


"/etc/chef/solo.rb" : {


"content" : { "Fn::Join" : ["
\
n", [


"file_cache_path
\
"/tmp/chef
-
solo
\
"",


"cookbook_path
\
"/tmp/chef
-
solo/cookbooks
\
""


]]},


"m
ode" : "000644",


"owner" : "root",


"group" : "root"


Integrating AWS CloudFormation with Opscode Chef

20


},


"/etc/chef/chef.json" : {


"content" : {


"chef_client": {


"server_url": { "Ref" : "ChefServerURL" }


},


"run_list": [ "recipe[chef
-
client::config]", "recipe[chef
-
client]" ]


},


"mode" : "000644",


"owner" : "root
",


"group" : "root"


},


"/etc/chef/roles.json" : {


"content" : {


"run_list": [ "role[wordpress]" ]


},


"mode" : "000644",


"owner" :

"root",


"group" : "root"


},


"/home/ubuntu/.s3cfg" : {


"content" : { "Fn::Join" : ["", [


"[default]
\
n",


"access_key = ", { "Ref" : "HostKeys" }, "
\
n",



"secret_key = ", {"Fn::GetAtt": ["HostKeys", "SecretAccessKey"]}, "
\
n",


"use_https = True
\
n"


]]},


"mode" : "000644",


"owner" : "ubuntu",


"group" : "ubuntu"



},


"/var/lib/gems/1.8/gems/ohai
-
0.6.4/lib/ohai/plugins/cfn.rb" : {


"source" : "https://s3.amazonaws.com/cloudformation
-
examples/cfn.rb",


"mode" : "000644",


"owner" : "root",


"group" : "root"


}


}


}


}


},



"Properties": {


"SecurityGroups": [ { "Ref" : "EC2SecurityGroup" }, { "Ref": "ChefServerSecurityGroup" } ],


"ImageId": { "Fn::FindInMap":
[ "AWSRegionArch2AMI", { "Ref": "AWS::Region" }, { "Fn::FindInMap": [ "AWSInstanceType2Arch", { "Ref":
"InstanceType" }, "Arch" ] } ]


},


"UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [


"#!/bin/bash
-
v
\
n",



"funct
ion error_exit
\
n",


"{
\
n",


" cfn
-
signal
-
e 1
-
r
\
"$1
\
" '", { "Ref" : "ChefClientWaitHandle" }, "'
\
n",


" exit 1
\
n",


"}
\
n",



"apt
-
get
-
y install python
-
setuptools
\
n",


"easy_install http
s://s3.amazonaws.com/cfn
-
init
-
demo/aws
-
cfn
-
bootstrap
-
1.0.tar.gz
\
n",


"cfn
-
init
--
region ", { "Ref" : "AWS::Region" },


"
-
s ", { "Ref" : "AWS::StackName" }, "
-
r ChefClient ",


"
--
access
-
key ", { "Ref" : "HostKe
ys" },


"
--
secret
-
key ", {"Fn::GetAtt": ["HostKeys", "SecretAccessKey"]},


"
--
region ", { "Ref" : "AWS::Region" }, " || error_exit 'Failed to run cfn
-
init'
\
n",



"# Fixup path and links for the bootstrap script
\
n",


"export PATH=$PATH:/var/lib/gems/1.8/bin
\
n",



"# Bootstrap chef
\
n",


"chef
-
solo
-
c /etc/chef/solo.rb
-
j /etc/chef/chef.json
-
r http://s3.amazonaws.com/chef
-
solo/bootstrap
-
latest.tar.gz > /tmp/chef_solo.log
2>&1 || error_exit 'Failed to bootstrap chef client'
\
n",



"# Fixup the server URL in client.rb
\
n",


"s3cmd
-
c /home/ubuntu/.s3cfg get s3://", { "Ref" : "ChefServerPrivateKeyBucket" }
, "/validation.pem /etc/chef/validation.pem >
/tmp/get_validation_key.log 2>&1 || error_exit 'Failed to get Chef Server validation key'
\
n",


"sed
-
i 's|http://localhost:4000|", { "Ref" : "ChefServerURL" }, "|g' /etc/chef/client.rb
\
n",



"chef
-
client
-
j /etc/chef/roles.json > /tmp/initialize_client.log 2>&1 || error_exit 'Failed to initialize host via chef client'
\
n",



"# If all went well, signal success
\
n",


"cfn
-
signal
-
e $?
-
r 'Chef Server configuration' '", {
"Ref" : "ChefClientWaitHandle" }, "'
\
n"


]]}},


"KeyName": { "Ref": "KeyName" },


"InstanceType": { "Ref": "InstanceType" }


}


},



"EC2SecurityGroup" : {


"Type" : "AWS::EC2::SecurityGroup",


"Properties" : {



"GroupDescription" : "Open up SSH access and HTTP over port 80",


"SecurityGroupIngress" : [


{ "IpProtocol": "tcp", "FromPort": "22", "ToPort": "22", "CidrIp": "0.0.0.0/0" },


{ "IpProtocol": "tcp", "FromPort": "80", "T
oPort": "80", "CidrIp": "0.0.0.0/0" }


Integrating AWS CloudFormation with Opscode Chef

21


]


}


},



"ChefClientWaitHandle" : {


"Type" : "AWS::CloudFormation::WaitConditionHandle"


},



"ChefClientWaitCondition" : {


"Type" : "AWS::CloudFormation::WaitCondition",


"DependsOn" : "ChefClient",


"Properties" : {


"Handle" : { "Ref" : "ChefClientWaitHandle" },


"Timeout" : "1200"


}


}


},



"Outputs": {


"WebsiteURL": {


"Value": { "Fn::Join": [ "", [ "http://", { "Fn::GetAtt
": [ "ChefClient", "PublicDnsName" ] }, "/" ] ] },


"Description" : "URL of the WordPress website"


},


"InstallURL": {


"Value": { "Fn::Join": [ "", [ "http://", { "Fn::GetAtt": [ "ChefClient", "PublicDnsName" ] }, "/wp
-
admin/install.php"
] ] },


"Description" : "URL to install WordPress"


}


}

}


Things to note about the template:



The user created in the template must have permissions to download the Chef Server
validation key
;

it is therefore given authorization to call both the CloudFormation
DescribeStackResource API and the S3 Get API.



The validation key is downloaded from S3 using the s3cmd package available from the
Ubuntu repository.



When the Chef Client is first installe
d, a new client is created using the validation key. In the
initial run, the run list for the node is empty since the newly created EC2 instance has no
configuration defined in the Chef Server.



To automatically set up the instance to with a specific role (
or recipe), the client is run in
the script directly, passing a configuration file
/etc/chef/roles/json.

This JSON file is defined
in the client template and contains the initial run list to be applied to the instance. When
this is executed, the client wil
l download the role information from the Chef Server and
initiate the installation. In addition, the role(s) applied to the client will be registered in the
Chef Server.

Ohai plug
-
in

Ohai is an open source utility that detects properties of your operating
environment. It is
distributed as a Ruby Gem and can be used standalone, but its primary purpose is to report data to
Chef via node attributes. This data can be referenced in cookbooks and recipes to create
customized deployments.

For more details of using

node attributes to configure Chef recipes see the
Opscode Chef wiki
.

Ohai

can be extended using plug
-
ins to detect and report any arbitrary attributes. AWS
CloudFormation provides a plug
-
in for Ohai that reports attributes defined in the template
resource metadata enabling you to define properties in the template that can be us
ed to customize
templates.


Integrating AWS CloudFormation with Opscode Chef

22

provides "cfn"



require 'rubygems'

require 'json'



filename = "/var/lib/cfn
-
init/data/metadata.json"



if not File.exist?(filename)


return

end



parsed = JSON.load(File.new(filename))

parsed.default = Hash.new



cfn Mash.ne
w(parsed["Chef"])


Once installed, you can define attributes in your template as follows:


"Resources" : {



"ChefClient": {


"Type": "AWS::EC2::Instance",


"Metadata" : {


"AWS::CloudFormation::Init" : {


"config" : {


:


}


},


"Chef" : {


"wordpress" : {


"db" : {


"database" : "foo",


"host" : "bar",


"user" : "dbuser",


"password" : "pass"


}


}


}


},


Your Chef recipes can then reference the metadata as follows:

template "#{node['wordpress']['dir']}/wp
-
config.php" do


source "wp
-
config.php.erb"


owner "root"


group "root"


mode "0644"


variables(


:database => node['cfn']['wordpress']['db']['database'],


:host => node['cfn']['wordpress']['db']['host'],


:user => node['cfn']['wordpress']['db']['user'],


:password => node['cfn']['wordpress']['db']['pass
word'],


:auth_key => node['wordpress']['keys']['auth'],


:secure_auth_key => node['wordpress']['keys']['secure_auth'],


:logged_in_key => node['wordpress']['keys']['logged_in'],


:nonce_key => node['wordpress']['keys']['nonce']


)


notifies :write, "log[Navigate to 'http://#{server_fqdn}/wp
-
admin/install.php' to complete wordpress installation]"

end

Summary of using Chef

The Chef Server and Chef Client templates presented allow you to create a Chef Server and
provision new EC2
instances. Not only are the clients bootstrapped with the Chef Client software,
but also with an initial set of roles. Once instances are running, they can be managed from the Chef
Server just like any other Chef
-
managed nodes. New roles may be applied, so
ftware updated, roles
removed and so on all from the Chef Server webui or through the knife command. AWS
CloudFormation does not get in your way.

While this section does not cover using Hosted Chef, it is very similar to using your own Chef
Server. For mor
e details see the
Opscode Chef wiki
.


Integrating AWS CloudFormation with Opscode Chef

23

Getting the AWS CloudFormation helper scripts

and templates

The AWS CloudFormation helper scripts are available from the following locations:



The latest v
ersion of the Amazon Linux AMI has the AWS CloudFormation helper scripts
installed by default in
/opt/aws/bin
.



The AWS helper scripts are available in the Amazon Linux
AMI y
um repository (the
package name is
aws
-
cfn
-
bootstrap
) for previous versions of
the
Amazon Linux

AMI
.



In addition, the helpers are available in other formats:

o

https://s3.amazonaws.com/cloudformation
-
examples/aws
-
cfn
-
bootstrap
-
1.0
-
4.noarch.r
pm

o

https://s3.amazonaws.com/cloudformation
-
examples/aws
-
cfn
-
bootstrap
-
1.0.tar.gz

to install
the helper scripts via the Python
easy
-
install

tools.

o

https://s3.amazonaws.com/cloudformation
-
examples/aws
-
cfn
-
bootstrap
-
1.0.zip

o

https://s3.amazonaws.com/cloudformation
-
examples/aws
-
cfn
-
bootstrap
-
1.0.win32.msi

for
installation on Windows.



The source
for the scripts is available at

https://s3.amazonaws.com/cloudformation
-
examples/aws
-
cfn
-
bootstrap
-
1.0
-
4.src.rpm



The Ohai plugin used in this document is available at
https://s3.amazonaws.com/cloudformation
-
examples/cfn.rb



The templates used in this document are available, along with many other sample
templates
,

on the AWS CloudFormation sample template site at
http://aws.amazon.com/cloudformation/aws
-
cloudformation
-
templates/