Download - Docs

boundlessbazaarServers

Dec 9, 2013 (3 years and 10 months ago)

118 views

Continuous Integration

in the Cloud

with Hudson

Kohsuke Kawaguchi

Jesse Glick

Sun Microsystems, Inc.

Hudson committers

Rise of Continuous Integration

>
Offload from people, push to computers

2

$

time

computers

us

Spend more CPU power to help you

>
… even if it only helps a little


>
First on your laptops and workstations


IDEs are at the forefront

>
And then to the servers


a.k.a. “Continuous Integration”


More frequent build/test executions


Static code analysis tools


And more to come

3

Hudson

>
Open
-
source CI server at java.net


>
Emphasis on ease of installation and use


“java
-
jar hudson.war” execution


Configure everything from browsers

>
Extensibility


140+ community
-
developed public plugins


By 150+ contributors


>
Estimated 13,000 installations

4

https://hudson.dev.java.net/

It basically does builds and tests

>
Check out the source code


Subversion, Perforce, Git, Mercurial, CVS, …

>
Do builds and/or tests


Ant, Maven, MSBuild, shell script, …

>
Record results


Binary, test results, code coverage, static analysis

>
Notify people


E
-
mail, IM, RSS, tray apps, IDEs

5

Localized to 8 languages

6

And hopefully more to come…

7

8

Adoption in all kinds of businesses

9

Why Distributed Builds?

>
You need to use multiple computers because…


You need different environments


You need isolation


>
There’s only so much you can do with 1 computer



10

Before we talk about clouds…

>
Going virtual doesn’t solve…


Software installation problem


Node failure problem


Remote maintenance problem





>
What does Hudson do to help you with these?




11

Installing new slaves

>
For first 20 or so slaves, we did it manually


Insert CD, click, type, click, type, click, …


But that doesn’t scale


>
Then we automated


Available as “Hudson PXE Plugin”

12

Automated System Installations

13

>
Slaves


Power on, hit F12


PC boots from network (PXE)


>
Hudson + PXE
plugin


ISO images of OS



Automated System Installations

14

>
Slaves


Power on, hit F12


PC boots from network (PXE)


Choose OS from menu


Installs non
-
interactively


>
Hudson + PXE
plugin


ISO images of OS



Your corporate IT guy
& his DHCP server

Automated System Installations

>
Supports OpenSolaris, Ubuntu, CentOS, Fedora


Trivial with most Linux

>
Cooperate with Windows, too


>
Quite useful outside Hudson, too


No more broken CD drives


No more CD
-
Rs


15

Distributed builds with Hudson

>
Master


Serves HTTP requests


Stores all important info

>
Slaves


170KB single JAR


Assumed to be unreliable


Scale to at least 100

>
Link


Single bi
-
di byte stream


No other requirements



16

Master

slave1

slave2

slave3

slave4

slave5

slave6

slave7

slave8

How master and slaves start talking

>
For Unix slaves, via SSH


Only need SSH and JRE on slaves


We just need a host name

17

How master and slaves start talking

>
For Windows, DCOM


We just need admin user name and password


No manual intervention


Works even from Unix masters

18

How master and slaves start talking

>
Via Java Web Start


When master cannot see slaves


A separate socket connection is made




19

Automating JNLP launch

>
Once started, can be installed as Windows
service

20

Automating JNLP launch

>
Emulate the JNLP client headless

21

$ java
-
jar slave.jar
-
jnlpUrl URL

Automated Tool Installation
-

JDK

>
JDK from http://java.sun.com/


Hudson automatically chooses the right bundle


Always up to date with new releases

22

Automated Tool Installation
-

Apache

>
Ant and Maven from Apache






23

Automated Tool Installation
-

Custom

>
Download arbitrary archive and unpack





24

Automated Tool Installation
-

Custom

>
Run arbitrary shell commands


Can have variants by OS

25

Automated Tool Installation
-

Extensible

>
Write your own


Simple Hudson extension


Just write Java code to create tool on slave

>
In progress:


SCMs


Mercurial, …


Install from Subversion

26

Heterogeneous Cluster Challenge

>
Your builds/tests need to run in specific
environment

>
Dependency on individual nodes hurts utilization

27

Wombat

Windows test

Hudson
Windows test

Windows
#1

jobs

slaves

GlassFish

Windows test

Windows
#2

Solaris

#1

Hudson
Solaris test

Labels to rescue

>
Label is a group of slaves

>
Tie jobs to labels


28

Wombat

Windows test

Hudson
Windows test

Windows
#1

jobs

slaves

GlassFish

Windows test

Windows
#2

Solaris

#1

Hudson
Solaris test

Windows

Solaris

Windows
#3

Forecasting failures

>
Hudson monitors key health metrics of slaves


Low disk space, insufficient swap


Clock out of synch


Extensible

>
Slaves go offline automatically

>
Catch problems before they break builds


30

Clean up mess after builds

>
Kill runaway processes


Daemons, background processes left by your build


Works on Windows, Linux, Mac, and Solaris



31

Load Statistics Monitoring

32

When it’s time to add more slaves

33

Hudson made this extensible

>
Hudson detects excessive workload

>
Hudson notifies plugins

>
Plugins can provision more slaves


… assuming that you have that infrastructure


34

35

Amazon EC2: The Good

>
Pay as you go (10¢/h or so)


Loads on Hudson tend to be spiky

>
Programmable API

>
Instances launch at machine
-
speed

>
EC2 instances are forgetful

36

Amazon EC2: The Bad

>
Your data is still inside your firewall


Takes time to check out code


… or to archive build artifacts


Some data just can’t be moved

>
EC2 instances are forgetful

>
Can your tests run in parallel?


37

Hudson EC2 plugin

>
Built on top of typica*

>
What does it do?


Automatically provisions slaves on EC2 on demand


Picks the right AMI depending on demand


Starts slave agent


Shuts down unused instances


38

* http://code.google.com/p/typica/

Putting it all together

39

time

# of executors

capacity

usage

queue length

Hudson “Appliance” on EC2

>
Run the master in the cloud too, if you like


Hudson on stock OpenSolaris AMI


Data stored persistently in Elastic Block Storage


Dynamically expandable thanks to ZFS


Online, too

>
Packaged as a wizard

40

41

42

>
Hudson Hadoop plugin


Just a few mouse clicks to install


Turn every Hudson slave into a Hadoop node


>
Distributed file system


Automatic data replication (fault tolerant)


Nice for storing old artifacts, logs, test records, …

>
Map/reduce framework


Large scale test results analysis / datamining


More interesting work to be done in the future


43

Hudson Selenium Plugin

>
Selenium


Tests webapps by scripting browsers

>
Selenium Grid


Runs Selenium over a grid of computers


>
Allow Hudson labels to specify where to start
browsers

>
Hudson & Selenium both need heterogeneous
cluster

44

Selenium Grid

45

Hudson master

(selenium hub)

Hudson slaves

Conclusion

>
CI is here to stay


We’ll continue to push more workload to servers

>
Hudson makes this easy for you

>
Reap the benefit of a cluster in multiple ways

46

Resources

>
http://hudson.dev.java.net/


>
BOF
-
5105 “Hudson Community Meet up”


Today 7:45pm same room

>
Hudson booth inside Sun Pavilion


>
Support Subscription


hudson@sun.com


47

48

Kohsuke Kawaguchi

Jesse Glick


kohsuke.kawaguchi@sun.com

jesse.glick@sun.com

http://hudson.dev.java.net/

Agenda

>
Quick Hudson introduction

>
Doing distributed builds

>
How Hudson make distributed builds easier

>
Taking infrastructure to cloud





49

Setting up slaves

>
Keeping slaves consistent is a good thing


Particularly hard on heterogeneous environment

>
General system administration tasks


Network configuration


Package installations for native tools


Tools like Puppet or cfEngine are supposed to help

>
Install build tools in the cluster


Prepare tools on one file system


rsync to everywhere

>
This part of Hudson needs improvements


50

Hudson EC2 plugin usage

>
Tell Hudson your AWS account information


51

Hudson EC2 plugin usage

>
Tell Hudson what AMIs you want to start

52