Detecting Confusion Among Novice Programmers Using BlueJ Compile Logs

scacchicgardenΛογισμικό & κατασκευή λογ/κού

13 Δεκ 2013 (πριν από 3 χρόνια και 9 μήνες)

103 εμφανίσεις

Detecting Confusion Among Novice
Programmers Using BlueJ Compile Logs

By

Diane Lee

Aplusix, Scatterplot, Ecolab, BlueJ, etc.

Student
Interaction
Logs

Observation
Logs

Analysis


Affect detectors


Behavior detectors


Novice programmer errors


Interventions


Intelligent agents


Improved error
messages

Biometrics
Logs

Introduction


Researchers have acknowledged the role of affect in
decision
-
making.


Negative affect has a negative impact on learners


Boredom, confusion, frustration


Early detection of negative emotions might help detect at
-
risk students.

Previous Study


Focused on detecting the negative affects of confusion
and boredom from novice programmers’ BlueJ compile
logs


These are the affective states that have a negative
correlation with achievement and learning.



Previous Study


Compile Logs


All submissions made to a compiler; also known as online
protocols


Status Logs


Extension to BlueJ


Dropdown box with 7 Affective States


Frustrated, Surprised, Delighted, Neutral, Flow, Confused, Bored

Previous Study


Data Collection


Who: Students enrolled in CS21A (AY2009
-
2010)


What: 4 lab exercises


When: First sem of AY 2009
-
2010


Where: F227 and F228


How: BlueJ + LAN + SQLite Database

Previous Study

Cleaning and Sync

Analysis


Before the data set (from
the tables in the database)
can be used for feature
selection, synchronization
and analysis, irrelevant
records at the time of
data gathering must be
separated.


Time window synch



Supervised learning
techniques


inductive process


Examples: Gaussian Process
(GP) and Support Vector
Machine (SVM)


Good for labeled data (data
that have a relationship with
ones that we want to predict)


Test our model using k
-
fold
cross validation test


Total of 149 participants


Lots of missing compilation logs from sections at different
sessions


Lack of status logs


Confused = most frequent


Boredom = least frequent


WEKA Analysis


RapidMiner Analysis

Results


Feature set:


# of compilations, Time per session (mins), Error quotient, # of
errors, # of consec. pairs of compilations with errors in the
same line number, # of consec. pairs of compilations with
errors of the same type, ave. time between compilations


Implementation 1: weak GP models, no SVMs


Implementation 2: weak GP models, no SVMs


Implementation 3: weak GP models and SVMs


Primary performance criteria: correlation coefficient and
mean abs. error

WEKA Analysis


Feature set


# of seconds since last action, # of compilations, # of errors, #
of consec. pairs of compilations with errors in the same line
number, # of consec. pairs of compilations with errors of the
same type, # of consec. pairs of compilations with errors in the
same edit location, ave. time bet. compilations, ave. time bet.
unsuccessful compilations


t
-
5 and t
-
2 minutes


J48 trees, logistic regression, step regression, boosted
decision stumps


Generally weak models all through the algorithms (look
at kappa value)

RapidMiner Analysis


Summary


Data insufficient for a fine or coarse
-
grained model


Too little features


Need for a richer attribute set


Status logs proved inadequate in labelling


GP and SVMs did not produce good results, as expected

Previous Study

Previous Study


Problems and issues


Too few status logs made it impossible to tag the appropriate
labels to compile logs


Weaknesses in definitions of terms


Small training set

This Study


is a continuation of prior work


But will now focus on confusion rather than boredom


Will not use the status log anymore




Objectives/Goal


To determine the factors within the compile logs that
indicate instances of confusion.


To create a model that can automatically detect confusion
among novice programmers using the aforementioned
factors.

Significance


Can help instructors identify students who are having
difficulty learning the subject matter


Instructors


can adjust accordingly


know when to intervene to help the student.

Confusion


It is important to note that there are two kinds of
confusion


Helpful


Harmful

Negative Effects of Confusion


Affect Student Performance


Lower Achievement


Gaming The System


Solving Compilation Errors


Confusion can exhibit it self in the way students solve
compilation errors


Stoppers


Movers


Extreme Movers


Methodology


Data: same from previous study


Will only use the compilation logs from last year


Will use Decision Trees for Analysis

This time though…


Compilation logs where divided into clips


Had clips labeled by professionals


Confused


Not confused


Done to establish a basis for detecting confusion




So far…


Done a t
-
test between clips labeled as confused and clips
labeled not confused


0.086049767
-

Ave time bet compilations


0.454034255
-

Max time bet compilations


5.3901E
-
12
-

Ave time bet comps. With errors


3.14414E
-
11
-

Max time bet comp. with errors


Have another batch of clips being relabeled


To get Jadud’s Error Quotient (EQ)

Thank you.

Any questions?