Developing Natural Language-based Software Analyses and Tools to Expedite Software Maintenance

erminerebelΤεχνίτη Νοημοσύνη και Ρομποτική

15 Νοε 2013 (πριν από 3 χρόνια και 7 μήνες)

68 εμφανίσεις

Developing Natural Language
-
based

Software Analyses and Tools

to Expedite Software Maintenance

Lori Pollock

Collaborators: K. Vijay
-
Shanker, Emily Hill,
GiriprasadSridhara

Past collaborators: David Shepherd, Zachary P. Fry,
KishenMaloor

Problem

Modern software is large and complex

object oriented class hierarchy

Softwaremaintenance:


-

search/locate


-

navigate


-

understand


-

modify


-
> Automated Support

Successes
in Software Maintenance Tools

object oriented class hierarchy


Good with local
tasks


Good with
traditional structure

object oriented class hierarchy


Scattered tasks
are difficult


Programmers
use more than
traditional
program
structure

Challenges
in Software Maintenance Tools

public interface Storable{...

activate tool

save drawing

update drawing

undo action

public void Circle.save()

//Store the fields in a file....

object oriented system

Key Insight:

Programmers leave
natural language clues
that
can
benefit

softwaremaintenance
tools

Observations

Studies on choosing identifiers


Impact of human cognition on names
[Liblit et al. PPIG 06]


Metaphors, morphology, scope, part of speech hints


Hints for understanding code


Analysis of Function identifiers
[Caprile and Tonella WCRE 99]


Lexical, syntactic, semantic analysis


Use for software tools: metrics, traceability, program understanding

Carla, the compiler writer

Pete, the programmer

I don’t care

about names.

So, I could use

x, y, z. But, no one

will understand

my code.

NLPA

Our Research Focus and Impact

Software Maintenance Tools

Search

Understanding



Natural Language Analysis

Word relations

(synonyms,

antonyms, …

Part of speech

tagging



Abbreviations

Exploration

Our Research Contributions…

FindConcept

Concern location tool

Clue Extraction +

NL
-
based Program


Representation

Motivated use

of NL clues


during maintenance

Dora the

Program Explorer

Abbreviation

Expander

Word Relation Tool

Comparison Study

[MACS 05, LATE 05]

[AOSD 06, IET 08]

[AOSD 07, PASTE 07]

[ASE 05]

iTimna

Aspect Miner

[ASE 07]

[MSR 08]

[ICPC 08]

Automatic Natural Language Clue
Extraction from Source Code


Key Challenges
:


Decode name usage


Develop automatic NL clue
extraction process (focused on
Java)


Create NL
-
based program
representation

Molly, the Maintainer

What was Pete
thinking

when he wrote this code?

Natural Language: Which Clues to Use?


Software Maintenance


Typically focused on actions


Objects are well
-
modularized


Focus on actions


Correspond to verbs


Verbs need Direct Object


(DO)


Extract
verb
-
DO
pairs


Extracting Verb
-
DO Pairs

Two types of extraction


class Player{


/**


* Play a specified file with specified time interval


*/


public static boolean play(final File file,final float fPosition,final long length) {


fCurrent = file;


try {


playerImpl = null;


//make sure to stop non
-
fading players


stop(false);


//Choose the player


Class cPlayer = file.getTrack().getType().getPlayerImpl();




}

Extraction
from
comments

Extraction
from
method
signatures


public UserList getUserListFromFile( String path )
throws

IOException {


try

{


File tmpFile =
new

File( path );

return

parseFile(tmpFile);



}
catch
( java.io.IOException e ) {

thrownew

IOrException( ”UserList format issue" + path + " file " + e );


}


}

Extracting Clues from Signatures

1.
Part
-
of
-
speech tag method name

2.
Chunk method name

3.
Identify
Verb

and
Direct
-
Object

(DO)


get
<verb>

User
<adj>

List
<noun>
From
<prep>
File
<noun>

get
<verb phrase>

User List
<noun phrase>
FromFile
<prep phrase>

POS Tag

Chunk

Representing Verb
-
DO Pairs

Action
-
Oriented Identifier Graph (AOIG)

verb1

verb2

verb3

DO1

DO2

DO3

verb1, DO1

verb1, DO2

verb3, DO2

verb2, DO3

source code files

use

use

use

use

use

use

use

use

Action
-
Oriented Identifier Graph (AOIG
)

Example

play

add

remove

file

playlist

listener

play, file

play, playlist

remove, playlist

add, listener

source code files

use

use

use

use

use

use

use

use

Evaluation of Clue Extraction


Compared automatic vs ideal (human) extraction


300 methods from 6 medium open source programs


Annotated by 3 Java developers


Promising Results


Precision: 57%


Recall: 64%


Context of Results


Did not analyze trivial methods


On average, at least verb OR direct object obtained

Using AOIG in Concern Location

Find, collect, and understand all source
code related to a particular concept





Concerns are
often crosscutting

State of the Art for Concern Location


Mining Dynamic Information
[Wilde ICSM
00]


Program Structure Navigation
[Robillard
FSE 05,

FEAT, Schaefer ICSM 05]


Search
-
Based Approaches


RegExp
[grep, Aspect Mining Tool 00]


LSA
-
Based
[Marcus 04]


Word
-
Frequency Based
[GES 06]

Reduced to
similar problem

Slow

Fast

Fragile

Sensitive

No Semantics

Limitations of Search Techniques

1.
Return large
result sets

2.
Return irrelevant
results

3.
Return hard
-
to
-
interpret result
sets

Find
-
Concept Search Tool


concept

Find
-
Concept




query

Recommendations

Source Code

Method a

Method b

Method c

Method d

Method e

NL
-
based

Code Rep

Result
Graph

Natural
Language
Information

1. More
effective search

2. Improved
search terms

3. Understandable
results

Underlying Program Analysis


Word Recommendation Algorithm


Stemmed/Rooted:
complete, completing


Synonyms:
finish, complete


Co
-
location:
completeWord()


Uses traversals of Action
-
oriented
identifier graph (AOIG)


Experimental Evaluation


Research Questions


Which
search tool

is most
effective

at forming and
executing a query for concern location?


Which search tool requires the least human effort to
form an effective query?


Methodology:

18

developers
completenine

concern location
tasks
on
medium
-
sized (>
20KLOC
) programs



Measures:


Precision (quality), Recall (completeness),

F
-
Measure
(combination of both P & R)


Find Concept
, GES, ELex

Overall Results


Effectiveness


FC > Elex with statistical
significance


FC >= GES on 7/9 tasks


FC is more consistent than GES


Effort


FC = Elex = GES

FC is more consistent and more effective in
experimental study without requiring more effort

Across all tasks

NLPA

Our Research Focus and Impact

Software Maintenance Tools

Search

Understanding



Natural Language Analysis

Word relations

(synonyms,

antonyms, …

Part of speech

tagging



Abbreviations

Exploration

Dora the Program Explorer
*

*
Dora comes from
exploradora
, the Spanish word for a female explorer.

Dora

Natural Language Query



Maintenance request



Expert knowledge



Query expansion

Relevant
Neighborhood

Program Structure



Representation



Current: call graph



Seed starting point

Relevant Neighborhood



Subgraph relevant to query

Query

State of the Art in Exploration


Structural

(dependence, inheritance)


Slicing


Suade [Robillard 2005]



Lexical

(identifier names, comments)


Regular expressions
: grep, Eclipse search


Information Retrieval
: FindConcept [Shepherd
2007],

Google Eclipse Search [Poshyvanyk 2006]


Dora: Using Program Structure + Ids


Program:
JBidWatcher, an eBay auction sniping program


Bug:

User
-
triggered add auction event has no effect



Task:

Locate code related to
‘add auction’

trigger


Seed:
DoAction()

method, from prior knowledge


Example Scenario:

Key Insight:

Automated tools can use
program
structure
and

identifier names

to save the
developer time and effort

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

Using only structural information


DoAction()

has 38 callees,

only 2/38 are relevant

Relevant

Methods

Irrelevant Methods

Looking for:

‘add auction’ trigger

DoAction()

DoAdd()

DoPasteFromClipboard()

And what if you wanted to explore
more than one edge away?


Locates locally relevant items,

but many irrelevant

Using only lexical information


50/1812 methods contain
matches to ‘add*auction’
regular expression query



Only 2/50 are relevant



Locates globally relevant
items, but many irrelevant


DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

DoNada()

Combining Structural &

Lexical Information


Structural:

guides exploration
from seed

Looking for:

‘add auction’ trigger

Relevant

Neighborhood

DoAction()

DoPasteFromClipboard()

DoAdd()


Lexical:

prunes irrelevant edges

The Dora Approach


Determine method relevance to query


Calculate lexical
-
based relevance score


Prune low
-
scored methods from
neighborhood


Recursively explore

Prune irrelevant structural edges from seed

31

Evaluation of the Dora Approach


Evaluated on 9 concerns


Lexical + structural >
structural


However, success highly
dependent on lexical
scoring performance

Structure

only

NLPA

Our Research Focus and Impact

Software Maintenance Tools

Search

Understanding



Natural Language Analysis

Word relations

(synonyms,

antonyms, …

Part of speech

tagging



Abbreviations

Exploration

Automatic Abbreviation Expansion

1.
Split Identifiers:


Punctuation


Camel case


No boundary


e.g.,
strlen

2.
Identify non
-
dictionary
words

3.
Determine long form

non
-
dictionary word

no boundary


Don’t want to miss relevant code with abbreviations


Given a code segment, identify character sequences
that are short forms and determine long form


Approach:
Mine expansions from code
[MSR 08]

Simple Dictionary Approach


Manually create a lookup table of common
abbreviations in code

-
Vocabulary evolves over time, must maintain
table

-
Same abbreviation can have different
expansions depending on domain AND context:

cfg

?

Control Flow Graph

Context
-
Free Grammar

configuration

configure


Types of Non
-
Dictionary Words


Single
-
Word


Prefix

(attr, obj, param, i)


Dropped Letter

(src, evt, msg)


Multi
-
Word


Acronyms

(ftp, xml, [type names])


Combination

(println, doctype)


Others


No boundary

(saveas, filesize)


Misspelling

(instanciation,
zzzcatzzzdogzzz)

Long Form Search Patterns


Given short form
arg
, we search for
regular expressions matching long forms
in code:


Single
-
Word


Prefix
arg
ument


Dropped letter
a
ve
r
a
g
e


Multi
-
Word


Acronym
a
ttribute
r
andom
g
roup


Combination
a
ccess
r
i
g
hts

Search Pattern Order


Search by abbreviation type:

Acronym

Combination

Prefix

Dropped
Letter

Multi
-
Word

Single Word

Conservative

Greedy

How do we identify potential
long forms for each type?


Inspired by static scoping, start from method
containing abbreviation and search increasingly
broader “scopes” until clear winner:

1.
JavaDoc

2.
Type Names of declared variables

3.
Method Name

4.
Statements

5.
Referenced identifiers and string literals

6.
Method comments

7.
Class comments

Context
-
based Approach through
Scope

What if no long form found?


Fall back to Most Frequent Expansion
(MFE)


MFE leverages successful local
expansions and applies throughout the
program

1.
Program:

provides domain knowledge

2.
Java:

more general programming
knowledge


Scope 57% more accurate than
state of the art LFB


Scope 30% more accurate than
Java MFE


Program MFE acceptable
approximation when speed
more important than accuracy

Experimental Evaluation

Number of Correct Expansions

250

200

150

100

50

0

No

Exp

LFB

Java

MFE

Prog

MFE

Our

Scope

Accuracy:

22%

40%

45%

54%

59%

63%

In Conclusion…



Evaluation studies indicate


Natural language analysis has far more potential to
improve software maintenance tools than we
initially believed


Existing technology falls short

Synonyms, collocations, morphology, word
frequencies, part
-
of
-
speech tagging, AOIG


Keys to further success

Improve recall


Extract additional NL clues