makefilex

californiamandrillΛογισμικό & κατασκευή λογ/κού

13 Δεκ 2013 (πριν από 3 χρόνια και 6 μήνες)

86 εμφανίσεις

Make
makefiles

pipelining for the masses

Make is based on set of rules

# this is a remark

target
:
prerequisites ...


recipe_line1


recipe_line2


...


<Tab>

Simple
example


Leg.pdf
:
Leg.tree


R
<
plotTree.R

Leg.tree

Leg.tree

:
Leg.aln


RAxML

-
f
a
-
m GTRGAMMA
-
s
Leg.aln

>
Leg.tree

Leg.aln

:
Lpne.aln

Ldra.aln

Lisr.aln

Ljor.aln


perl

fastaMsaConcat.pl

Lpn.aln

Ldra.aln

… >
Leg.aln

Lpne.aln

:
Lpne.fasta


prank
-
F
Lpne.fasta

>
Lpne.aln

Ldra.aln

:
Ldra.fasta


prank
-
F
Ldra.fasta

>
Ldra.aln

Lisr.aln

:
Lisr.fasta


prank
-
F
Lisr.fasta

>
Lisr.aln

Lsha.aln

:
Lsha.fasta


prank
-
F
Lsha.fasta

>
Lsha.aln

Makefile

is a tree

Leg.pdf

Leg.tree

Leg.aln

Lpne.aln

Lpne.fas

Ldra.aln

Ldra.fas

Lisr.aln

Lisr.fas

Lsha.aln

Lsha.fas

Simple
example


Leg.pdf
:
Leg.tree


R
<
plotTree.R

Leg.tree

Leg.tree

:
Leg.aln


RAxML

-
f
a
-
m GTRGAMMA
-
s
Leg.aln

>
Leg.tree

Leg.aln

:
Lpne.aln

Ldra.aln

Lisr.aln

Ljor.aln


perl

fastaMsaConcat.pl

Lpn.aln

Ldra.aln

… >
Leg.aln

Lpne.aln

:
Lpne.fasta


prank
-
F
Lpne.fasta

>
Lpne.aln

Ldra.aln

:
Ldra.fasta


prank
-
F
Ldra.fasta

>
Ldra.aln

Lisr.aln

:
Lisr.fasta


prank
-
F
Lisr.fasta

>
Lisr.aln

Lsha.aln

:
Lsha.fasta


prank
-
F
Lsha.fasta

>
Lsha.aln

Makefile

makes the first
target (the order of the
rest is not important

Variables



alns

=
Lpne.aln

Ldra.aln

Lisr.aln

Ljor.aln

Leg.tree.pdf

:
Leg.tree


R <
plotTree.R

Leg.tree

Leg.tree

:
Leg.aln


RAxML

-
f
a
-
m GTRGAMMA
-
s
Leg.aln

>
Leg.tree

Leg.aln

: $(
alns
)


perl

fastaMsaConcat.pl

$(
alns
) >
Leg.aln

Lpne.aln

:
Lpne.fasta


prank
-
F
Lpne.fasta

>
Lpne.aln

Ldra.aln

:
Ldra.fasta


prank
-
F
Ldra.fasta

>
Ldra.aln

Lisr.aln

:
Lisr.fasta


prank
-
F
Lisr.fasta

>
Lisr.aln

Lsha.aln

:
Lsha.fasta


prank
-
F
Lsha.fasta

>
Lsha.aln

Variables

dir

= ~/Legionella/Genomes/
phase1/

alns

= $(
dir
)
Lpne.aln

$(
dir
)
Ldra.aln

$(
dir
)
Lisr.aln



$(
dir
)
Leg.tree.pdf

:
$(
dir
)
Leg.tree


R <
plotTree.R

$(
dir
)
Leg.tree

$(
dir
)
Leg.tree

: $(
dir
)
Leg.aln


RAxML

-
f
a
-
s $(
dir
)
Leg.aln

> $(
dir
)
Leg.tree

$(
dir
)
Leg.aln

: $(
alns
)


perl

fastaMsaConcat.pl

$(
alns
) >
$(
dir
)
Leg.aln

$(
dir
)
Lpne.aln

:
$(
dir
)
Lpne.fasta


prank
-
F $(
dir
)
Lpne.fasta

>
$(
dir
)
Lpne.aln

$(
dir
)
Ldra.aln

:
$(
dir
)
Ldra.fasta


prank
-
F
$(
dir
)
Ldra.fasta

> $(
dir
)
Ldra.aln

$(
dir
)
Lisr.aln

:
$(
dir
)
Lisr.fasta


prank
-
F
$(
dir
)
Lisr.fasta

> $(
dir
)
Lisr.aln

$(
dir
)
Lsha.aln

:
$(
dir
)
Lsha.fasta


prank
-
F
$(
dir
)
Lsha.fasta

> $(
dir
)
Lsha.aln

Special Variables

dir

= ~/Legionella/Genomes/
phase1/

alns

= $(
dir
)
Lpne.aln

$(
dir
)
Ldra.aln

$(
dir
)
Lisr.aln



$(
dir
)
Leg.tree.pdf

:
$(
dir
)
Leg.tree


R <
plotTree.R

$^

$(
dir
)
Leg.tree

: $(
dir
)
Leg.aln


RAxML

-
f
a
-
s
$^

>
$@

$(
dir
)
Leg.aln

: $(
alns
)


perl

fastaMsaConcat.pl

$^

>
$@

$(
dir
)
Lpne.aln

: $(
dir
)
Lpne.fasta


prank
-
F
$^

>
$@

$
(
dir
)
Ldra.aln

:
$(
dir
)
Ldra.fasta


prank
-
F
$^

>
$@

$
(
dir
)
Lisr.aln

:
$(
dir
)
Lisr.fasta


prank
-
F
$^

>
$@

$
(
dir
)
Lsha.aln

:
$(
dir
)
Lsha.fasta


prank
-
F
$^

>
$@

Rulez

dir

= ~/Legionella/Genomes/
phase1/

alns

= $(
dir
)
Lpne.aln

$(
dir
)
Ldra.aln

$(
dir
)
Lisr.aln



$(
dir
)
Leg.tree.pdf

:
$(
dir
)
Leg.tree


R <
plotTree.R

$^

$(
dir
)
Leg.tree

: $(
dir
)
Leg.aln


RAxML

-
f
a
-
s
$^

>
$@

$(
dir
)
Leg.aln

: $(
alns
)


perl

fastaMsaConcat.pl

$^

>
$@

%
.
aln

:
%
.
fasta


prank
-
F
$^

>
$
@

Rules will simplify your
makefile

and raise level
of abstraction

Rulez

dir

= ~/Legionella/Genomes/
phase1/

alns

= $(
dir
)
Lpne.aln

$(
dir
)
Ldra.aln

$(
dir
)
Lisr.aln



%
.
tree.pdf

:
%
.tree


R <
plotTree.R

$^

%
.tree
:
%
.
aln


RAxML

-
f
a
-
s
$^

>
$@

$(
dir
)
Leg.aln

: $(
alns
)


perl

fastaMsaConcat.pl

$^

>
$@

%
.
aln

:
%
.
fasta


prank
-
F
$^

>
$
@

General rules can be
transferred among
makefiles

Some more variables

dir

= ~/Legionella/Genomes/
phase1/

alns

= $(
dir
)
Lpne.aln

$(
dir
)
Ldra.aln

$(
dir
)
Lisr.aln



ALIGN

= prank
-
F

RECONSTRUCT

=
RAxML

-
f a
-
s

%
.
tree.pdf

:
%
.tree


R <
plotTree.R

$^

%
.tree
:
%
.
aln



$(
RECONSTRUCT
)
$
^

>
$@

$(
dir
)
Leg.aln

: $(
alns
)


perl

fastaMsaConcat.pl

$^

>
$@

%
.
aln

:
%
.
fasta



$(
ALIGN
)
$
^

>
$
@

Some functions
(there are many many more…)

$
(
addsuffix

<SUFF>,<LIST>)


$(
addsuffix

.
ext,a

dir
/b)

=>

a.ext

dir
/
b.ext

$(
addprefix

<PREF>,<LIST>)


$(
addprefix

dir
/,a
b.ext
)

=>

dir
/a
dir
/
b.ext

$
(
dir

<LIST>)


$
(
dir

tmp
/a
dir
/
b.ext

c)

=>

temp/
dir
/ ./

$(
notdir

<LIST>)


$
(
notdir

dir
/
b.ext

c)

=>

b.ext

c

$
(
basename

<LIST>)


$
(
notdir

a.R

dir
/
b.ext

c
)

=>

a
dir
/b
c

$
(
shell

<command>
)


Function usage

dir

= ~/Legionella/Genomes/
phase1/

alns

= $(
addprefix

$(
dir
),
Lpne.aln

Ldra.aln

Lisr.aln

…)

ALIGN

= prank
-
F

RECONSTRUCT

=
RAxML

-
f a
-
s

%
.
tree.pdf

:
%
.tree


R <
plotTree.R

$^

%
.tree
:
%
.
aln



$(
RECONSTRUCT
)
$
^

>
$@

$(
dir
)
Leg.aln

: $(
alns
)


perl

fastaMsaConcat.pl

$^

>
$@

%
.
aln

:
%
.
fasta



$(
ALIGN
)
$
^

>
$
@

Function usage

dir

= ~/Legionella/Genomes/
phase1/

alns

= $(
addprefix

$(
dir
),$(
addsuffix

.
aln
,
Lpne

Ldra

Lisr

…))

ALIGN

= prank
-
F

RECONSTRUCT

=
RAxML

-
f a
-
s

%
.
tree.pdf

:
%
.tree


R <
plotTree.R

$^

%
.tree
:
%
.
aln



$(
RECONSTRUCT
)
$
^

>
$@

$(
dir
)
Leg.aln

: $(
alns
)


perl

fastaMsaConcat.pl

$^

>
$@

%
.
aln

:
%
.
fasta



$(
ALIGN
)
$
^

>
$
@

More functions

$
(
shell

<command>
)

Executes shell command and returns values as list

(newlines are turned to spaces).



$
(shell cat
b.ext
)

=>

the content of
b.ext






(as a list)


$(LIST:.
fas
=.
aln
)

Changes all extensions of LIST


Legs
=
Lpne.fas

Ldra.fas

Lisr.fas

Ljor.fas


$(Legs:.
aln
=.
fas
)
=>




Lpne.aln

Ldra.aln

Lisr.aln

Ljor.aln



Function usage

dir

= ~/Legionella/Genomes/
phase1
/

alns

= $($(shell
ls

$(
dir
)*.
fasta
):.
fasta
=.
aln
)

ALIGN

= prank
-
F

RECONSTRUCT

=
RAxML

-
f a
-
s

%
.
tree.pdf

:
%
.tree


R <
plotTree.R

$^

%
.tree
:
%
.
aln



$(
RECONSTRUCT
)
$
^

>
$@

$(
dir
)
Leg.aln

: $(
alns
)


perl

fastaMsaConcat.pl

$^

>
$@

%
.
aln

:
%
.
fasta



$(
ALIGN
)
$
^

>
$
@

Special targets

dir

= ~/Legionella/Genomes/
phase1
/

alns

= $($(shell
ls

$(
dir
)*.
fasta
):.
fasta
=.
aln
)

ALIGN

= prank
-
F

RECONSTRUCT

=
RAxML

-
f a
-
s

All :
$(
dir
)
Leg.tree.pdf


%
.
tree.pdf

:
%
.tree


R <
plotTree.R

$^

%
.tree
:
%
.
aln



$(
RECONSTRUCT
)
$
^

>
$@

$(
dir
)
Leg.aln

: $(
alns
)


perl

fastaMsaConcat.pl

$^

>
$@

%
.
aln

:
%
.
fasta



$(
ALIGN
)
$
^

>
$
@

clean:


rm

-
vf

$(
dir
)
Leg.tree.pdf

$(
dir
)
Leg.tree

$(
dir
)*.
aln


Invoking
makefile

By default run
make

to execute the first target of the file called
makefile


> make

You can invoke a specific target in a specific file:

> make
-
f <
makefile

name> <TARGET>


make
-
f
makeTree.mk

clean

You can
also pass values of variables (will override variables with in the
makefile

> make
-
f <
makefile

name
> <VAR>=<value>


make
-
f
makeTree.mk

DIR=/
mydir
/


Useful flags for make file

-
j <n>: use n threads (without n
-

as many as possible
)



as
makefile

knows the dependencies he can run


the non
-
dependent recipes in parallel…


-
n

: just echo commands, do not
excute


-
d

: debug (shows you what
makefile

thinks)


REAL
-
LIFE EXAMPLES

Let's check out some

Here are some
makefiles

I did


A real
makefile

example:

http
://www.tau.ac.il/~davidbur/
makePhylogeny.mk




Huge (but well documented)
makefile
:

http
://www.tau.ac.il/~davidbur/
makeCompileFeaturesByPtts.makefile



I use such a
makefile

to run the one in (2):

http://www.tau.ac.il/~davidbur/
makefile



This
makefile

creates a
makefile

similar to (3) and executes it
(getting weird, isn't it…):

http://www.tau.ac.il/~davidbur/
makeLearningHappen.makefile


useful tips and hints


-

before the
commnad

will tell make to continue even if the
execution of that line fails


When using "$", for example in a Perl
oneliner
, use
$
$
to let
makefile

know you mean the character '$'


Beware of extra spaces at the end of lines


You can use the program/script itself as
part of
prerequisit



Some time it is useful to invoke make from within a
makefile


You can use
include FILE
to add the content of FILE to this
point in the
makefile


References

GNU make tutorial, everything you want to know about make:

http
://www.gnu.org/software/make/manual/
make.html


Eyali's

Blog in the post (my initial inspiration):

http
://lifeofadataminer.blogspot.co.il/2009/08/writing
-
makefile
-
instead
-
of
-
pipeline.html


A pipeline is a make (I think his initial inspiration):

http
://archive.nodalpoint.org/2007/03/18/a_pipeline_is_a_makefilehttp://archive.n
odalpoint.org/2007/03/18/
a_pipeline_is_a_makefile