A Randomized Dynamic

eyelashesnectarineΛογισμικό & κατασκευή λογ/κού

3 Νοε 2013 (πριν από 4 χρόνια και 4 μέρες)

87 εμφανίσεις

A Randomized Dynamic
Program Analysis for
Detecting Real Deadlocks

Koushik

Sen

CS 265


What is a Deadlock?


An unintended condition in a shared
-
memory, multi
-
threaded program in which:


a set of threads blocks forever


because each thread in the set waits to acquire a

lock being held by another thread in the set


This work: ignore other causes (e.g., wait/notify)



Example



// thread t1

sync (l1) {


sync (l2) {





}

}

// thread t2

sync (l2) {


sync (l1) {





}

}

l1

t1

l2

t2

Motivation


Today’s concurrent programs are rife with
deadlocks


6,500/198,000 (~ 3%) of bug reports in Sun’s
bug database at http://
bugs.sun.com

are
deadlocks


Deadlocks are difficult to detect


Usually triggered non
-
deterministically, on
specific thread schedules


Fail
-
stop behavior not guaranteed (some
threads may be deadlocked while others
continue to run
)


Fixing other concurrency bugs like races can
introduce new deadlocks


Our past experience with reporting races:
developers often ask for deadlock checker

Our Goal


Build a deadlock detection tool that


Scales to very large programs


Finds deadlocks quickly


Has no false positives


Shows a real execution exhibiting a deadlock


Related Work: Program Analysis


Static program analysis (e.g.,
Engler

& Ashcraft
SOSP’03; Williams
-
Thies
-
Ernst ECOOP’05,
Naik

et al.
ICSE’09)

+
Examines all possible program behavior

-
Often reports many false positives


Type systems (e.g.,
Boyapati
-
Lee
-
Rinard

OOPSLA’02)


-

Annotation burden often significant


Model checking (e.g., SPIN,
Verisoft
, Java Pathfinder)


-

Does not currently scale beyond few KLOC


-

Not “directed”


Dynamic program analysis (e.g.
Goodlock

Havelund

et al.,
Agrawal

et al.)

+
Usually reports lesser false positives

-

Has false negatives


Related Work: Testing


Testing

+ Easy to deploy

+ Scales to large programs

+ No false positives

-
Same schedule gets tested many times


No effort to control thread scheduler

-

Often subtle thread schedules that exhibit
deadlocks are not tested


In Summary


Static and dynamic program analyses
have false positives


Testing is simple


No false positives


But, may miss subtle thread schedules that
result in deadlocks

In Summary


Static and dynamic program analyses
have false positives


Testing is simple


No false positives


But, may miss subtle thread schedules that
result in deadlocks


Can we leverage program analysis to
make testing quickly find real
deadlocks?



Our Approach


Active Testing


Phase 1
: Use imprecise static or dynamic
program analysis to find “abstract”
states where a potential deadlock can
happen


Phase 2
: “Direct” testing (or model
checking) based on the “abstract”
states obtained from phase 1

Example

Thread1

foo
(o1,o2,true)

Thread2

foo
(o2,o1,false)



void
foo
(Object

l
1
,
Object l2,
boolean

flag
) {



if(flag) {


// Long running computations


s1: f1();


s2: f2();


}


s3: synchronized(l1){


s4: synchronized(l
2
){


}


}


}

Thread 1

Thread 2

Thread1

foo
(o1,o2,true)

Thread2

foo
(o2,o1,false)



void
foo
(Object

l
1
,
Object l2,
boolean

flag
) {



if(flag) {


// Long running computations


s1: f1();


s2: f2();


}


s3: synchronized(l1){


s4: synchronized(l
2
){


}


}


}

Testing

f1()

f2()

Lock(o2)

Lock(o1)

Unlock(o1)

Unlock(o2)

Thread 1

Thread 2

Thread1

foo
(o1,o2,true)

Thread2

foo
(o2,o1,false)



void
foo
(Object

l
1
,
Object l2,
boolean

flag
) {



if(flag) {


// Long running computations


s1: f1();


s2: f2();


}


s3: synchronized(l1){


s4: synchronized(l
2
){


}


}


}

Testing

Lock(o1)

Lock(o2)

Unlock(o2)

Unlock(o1)

Lock(o2)

Lock(o1)

Unlock(o1)

Unlock(o2)

f1()

f2()

Thread 1

Thread 2

Thread1

foo
(o1,o2,true)

Thread2

foo
(o2,o1,false)



void
foo
(Object

l
1
,
Object l2,
boolean

flag
) {



if(flag) {


// Long running computations


s1: f1();


s2: f2();


}


s3: synchronized(l1){


s4: synchronized(l
2
){


}


}


}

Lock(o1)

Lock(o2)

Unlock(o2)

Unlock(o1)

Lock(o2)

Lock(o1)

Unlock(o1)

Unlock(o2)

f1()

f2()

No deadlock
detected

Testing

Thread 1

Thread 2

Thread1

foo
(o1,o2,true)

Thread2

foo
(o2,o1,false)



void
foo
(Object

l
1
,
Object l2,
boolean

flag
) {



if(flag) {


// Long running computations


f1();


f2();


}


synchronized(l1){


synchronized(l
2
){


}


}


}

Deadlock Directed Testing

Lock(o2)

Lock(o1)

Paused

f1()

f2()

Thread 1

Thread 2

Thread1

foo
(o1,o2,true)

Thread2

foo
(o2,o1,false)



void
foo
(Object

l
1
,
Object l2,
boolean

flag
) {



if(flag) {


// Long running computations


f1();


f2();


}


synchronized(l1){


synchronized(l
2
){


}


}


}

Deadlock Directed Testing

Lock(o2)

Lock(o1)

Paused

f1()

f2()

Lock(o1)

Lock(o2)

Paused

Thread 1

Thread 2

Thread1

foo
(o1,o2,true)

Thread2

foo
(o2,o1,false)



void
foo
(Object

l
1
,
Object l2,
boolean

flag
) {



if(flag) {


// Long running computations


f1();


f2();


}


synchronized(l1){


synchronized(l
2
){


}


}


}

Deadlock Directed Testing

Lock(o2)

Lock(o1)

Paused

f1()

f2()

Lock(o1)

Lock(o2)

Paused

Thread 1

Thread 2

Thread1

foo
(o1,o2,true)

Thread2

foo
(o2,o1,false)



void
foo
(Object

l
1
,
Object l2,
boolean

flag
) {



if(flag) {


// Long running computations


f1();


f2();


}


synchronized(l1){


synchronized(l
2
){


}


}


}

Deadlock Directed Testing

Lock(o2)

Lock(o1)

Paused

f1()

f2()

Lock(o1)

Lock(o2)

Paused

Deadlock
detected !

Preempting threads


How do we know where to pause a
thread ?

Preempting threads


How do we know where to pause a thread ?


Use existing static or dynamic analyses to find
potential deadlock cycles


Note that these analyses may report false deadlock
cycles


Use “information” recorded for a deadlock
cycle to decide where to pause a thread


We use a modified version of the
Goodlock

algorithm (
iGoodlock
) [
Havelund

et al,
Agarwal

et al]

iGoodlock


We consider dynamic instances of the
following statements


c
:
Acquire(t,l
)


c
:
Release(t,l
)


c
:
Call(t,m
)


c
:
Release(t,m
)


c
:
o

= new (
t,o’,T
)

iGoodlock


We consider dynamic instances of the
following statements


c
:
Acquire(t,l
)


c
:
Release(t,l
)


c
:
Call(t,m
)


c
:
Release(t,m
)


c
:
o

= new (
t,o’,T
)

Lock Dependency Relation


D is a subset of (T X 2
L

X L X C*)


(
t,L,l,C
) is in the lock dependency
relation if

Lock Dependency Relation


D is a subset of (T X 2
L

X L X C*)


(
t,L,l,C
) is in the lock dependency
relation if


thread
t

acquires lock
l

while holding the
locks in the set L, and C is the sequence of
labels of Acquire statements that were
executed by
t

to acquire the locks in
L


{
l
}.

Lock Dependency Chain

(t
1
,L
1
,l
1
,C
1
), (t
2
,L
2
,l
2
,C
2
), …, (
t
m
,L
m
,l
m
,C
m
) is a
lock dependency chain if



Lock Dependency Chain

(t
1
,L
1
,l
1
,C
1
), (t
2
,L
2
,l
2
,C
2
), …, (
t
m
,L
m
,l
m
,C
m
) is a
lock dependency chain if


for all distinct
i
, j

[1,
m
],
t
i


t
j



for all distinct
i
, j

[1,
m
],
l
i


l
j



for all
i



[1,m − 1],
l
i



L
i+1
,


for all distinct
i,j



[1,m], L
i



L
j

=

.

Lock Dependency Chain

(t
1
,L
1
,l
1
,C
1
), (t
2
,L
2
,l
2
,C
2
), …, (
t
m
,L
m
,l
m
,C
m
) is a
lock dependency chain if


for all distinct
i
, j

[1,
m
],
t
i


t
j



for all distinct
i
, j

[1,
m
],
l
i


l
j



for all
i



[1,m − 1],
l
i



L
i+1
,


for all distinct
i,j



[1,m], L
i



L
j

=

.


A chain is a deadlock if

Lock Dependency Chain

(t
1
,L
1
,l
1
,C
1
), (t
2
,L
2
,l
2
,C
2
), …, (
t
m
,L
m
,l
m
,C
m
) is a
lock dependency chain if


for all distinct
i
, j

[1,
m
],
t
i


t
j



for all distinct
i
, j

[1,
m
],
l
i


l
j



for all
i



[1,m − 1],
l
i



L
i+1
,


for all distinct
i,j



[1,m], L
i



L
j

=

.


A chain is a deadlock if


l
m



L
1

Thread 1

Thread 2

Thread1

foo
(o1,o2,true)

Thread2

foo
(o2,o1,false)



void
foo
(Object

l
1
,
Object l2,
boolean

flag
) {



if(flag) {


// Long running computations


s1: f1();


s2: f2();


}


s3: synchronized(l1){


s4: synchronized(l
2
){


}


}


}

Acquire(o1)

Acquire(o2)

Release(o2)

Release(o1)

Acquire(o2)

Acquire(o1)

Release(o1)

Release(o2)

Compute D on this trace

f1()

f2()

Computing lock dependency
chains


Compute
D
k

where D
1

= D


for each
d


D and
τ



D
i



if
d
,
τ

is a lock dependency chain and a
deadlock cycle then report a potential
deadlock


if
d
,
τ

is a lock dependency chain and not
a deadlock cycle


add
d
,
τ

to D
i+1


Preempting threads


T1, o1, T2, o2 are runtime objects


How do we identify them across
executions ?


Preempting threads


T1, o1, T2, o2 are runtime objects


How do we identify them across
executions ?


Runtime addresses?


Preempting threads


T1, o1, T2, o2 are runtime objects


How do we identify them across
executions ?


Runtime addresses?


No, they change across executions !


Preempting threads


T1, o1, T2, o2 are runtime objects


How do we identify them across
executions ?


Runtime addresses?


No, they change across executions !


Allocation sites where they were created ?


Yes, but different objects created at the same
allocation site will all be treated as the same
object



Preempting threads


T1, o1, T2, o2 are runtime objects


How do we identify them across
executions ?


Runtime addresses?


No, they change across executions !


Allocation sites where they were created ?


Yes, but different objects created at the same
allocation site will all be treated as the same
object


Can we do better?



Abstractions


Based on k
-
object sensitivity


[
Milanova

et al]


Based on light
-
weight execution indexing
[
Xin

et al]


Abstractions




7 class A {


8 void bar() {


9 for(
int

j = 0; j < 5; j++)

10 Object l = new Object();

11 }

12 }


1 class C {

2 void
foo
() {

3 A
a

= new A();

4 a.bar();

5
}

6
}

13 main() {

14 C
c

= new C();

15 for(
int

i

= 0;
i

< 5;
i
++)

16 c.foo();

17 }

Abstractions




7 class A {


8 void bar() {


9 for(
int

j = 0; j < 5; j++)

10 Object l = new Object();

11 }

12 }


1 class C {

2 void
foo
() {

3 A
a

= new A();

4 a.bar();

5
}

6
}

13 main() {

14 C
c

= new C();

15 for(
int

i

= 0;
i

< 5;
i
++)

16 c.foo();

17 }

Abstraction of last

object created (k
-

object sensitivity) :
[10,3, 14]



Abstraction of last

object created

(execution indexing) :

[(10,5),(4,1),(16,5)]



Implementation


Implemented in a prototype tool called
DEADLOCKFUZZER for Java


Part of the CALFUZZER framework


Active testing [
Sen

PLDI 08, Park et al.
FSE 08, Joshi et al. CAV 09] of concurrent
programs


Instrument Java
bytecode

to


observe events during execution


control scheduler



Evaluation

Program
Name

Lines Of
Code

Average Runtime in milliseconds

Normal

iGoodlock

DeadlockFuzzer

cache4j

3,897

2,045

3,409

N.A.

sor

17,718

163

396

N.A.

hedc

25,024

165

1,668

N.A.

jspider

10,252

4,622

5,020

N.A.

Jigsaw

160,338

-

-

-

Java Logging

4,248

166

272

493

Java Swing

337,291

4,694

9,563

28,052

DBCP

27,194

603

1,393

1,393

Synchronized
Lists

17,633

2,862

3,244

7,070

Synchronized
Maps

18,911

2,295

2,596

2898

Evaluation

Program
Name

Lines Of
Code

# Deadlock Cycles

Probability
of
Reproduction

iGoodlock

Real

Reproduced

cache4j

3,897

0

0

N.A.

N.A.

sor

17,718

0

0

N.A.

N.A.

hedc

25,024

0

0

N.A.

N.A.

jspider

10,252

0

0

N.A.

N.A.

Jigsaw

160,338

283



29

29

0.214

Java Logging

4,248

3

3

3

1.00

Java Swing

337,291

1

1

1

1.00

DBCP

27,194

2

2

2

1.00

Synchronized
Lists

17,633

27

27

27

0.99

Synchronized
Maps

18,911

20

20

20

0.52

Evaluation

Limitations


If DEADLOCKFUZZER does not
reproduce a deadlock, we cannot say if
the deadlock is a false positive or not


Jigsaw


Deadlocks reported by
iGoodlock

: 283


Deadlocks reproduced by
DEADLOCKFUZZER : 29


Deadlocks confirmed as false positives : 18


Rest : 236


Limitations


If DEADLOCKFUZZER does not
reproduce a deadlock, we cannot say if
the deadlock is a false positive or not


Jigsaw


Deadlocks reported by
iGoodlock

: 283


Deadlocks reproduced by
DEADLOCKFUZZER : 29


Deadlocks confirmed as false positives : 18


Rest : 236


Cannot say if they are real deadlocks or


false warnings

Conclusion


DEADLOCKFUZZER is a practical
deadlock detection tool that


Scales to large programs


Finds deadlock quickly


Finds real deadlocks


Complements static and dynamic
analyses


Automatically confirms some of the real
deadlocks