Reflections on iCODE: using web technology and hands-on projects to engage urban youth in computer science and engineering


Nov 5, 2013 (4 years and 8 months ago)


Auton Robot (2011) 30:265–280
DOI 10.1007/s10514-011-9218-3
Reflections on iCODE:using web technology and hands-on
projects to engage urban youth in computer science
and engineering
Fred G.Martin · Michelle Scribner-MacLean ·
SamChristy · Ivan Rudnicki · Rucha Londhe ·
Colleen Manning · Irene F.Goodman
Received:11 December 2009/Accepted:6 January 2011/Published online:26 January 2011
©Springer Science+Business Media,LLC 2011
Abstract More than 200 middle school and high school
students from underserved urban communities in Boston,
Lowell,and Lawrence,Massachusetts,participated in after-
school and summer enrichment programs over a three-year
period,using hands-on learning materials and web resources
to complete hands-on microcontroller-based projects.Pro-
gram content was based on a suite of robotics and electron-
ics kits developed by the University of Massachusetts Low-
ell and Machine Science Inc.,together with on-line instruc-
tions,a web-based programming tool,and a shared elec-
tronic portfolio of student projects.Participating students
worked with classroomteachers and undergraduate mentors
This material is based upon work supported by the National Science
Foundation under Grant Numbers DRL-0624669 and DRL-0624631.
F.G.Martin (
) · M.Scribner-MacLean
University of Massachusetts Lowell,Lowell,USA
S.Christy · I.Rudnicki
Machine Science Inc.,Cambridge,USA
R.Londhe · C.Manning · I.F.Goodman
Goodman Research Group,Inc.,Cambridge,USA
to complete a series of projects,and took part each year in
a non-competitive robotics exhibition and a competitive ro-
bot sumo tournament.Goodman Research Group assessed
learning outcomes and attitudinal changes using a variety
of measures,including observations of program sessions,
group interviews with participating students,pre- and post-
program student surveys,and educator feedback.The pro-
gramwas found to effectively engage participants,give them
real engineering and programming skills,improve their at-
titudes toward science,technology,engineering,and mathe-
matics (STEM) subjects,and increase their interest in STEM
career pathways.These results are presented,along with
lessons learned from the program implementation,technol-
ogy development,and evaluation.
Keywords Robotics · Education · K-12 · Informal · After
school · Microcontroller · Programming · Logo · Sensors ·
Crafts · Evaluation · Competition · Career · Computer
science · Engineering
1 Introduction
From 2006 to 2009,the University of Massachusetts Low-
ell (UML) and a non-profit partner,Machine Science Inc.
of Cambridge,Massachusetts,put technology on the web to
support the growth of a community of young people using
tangible,microcontroller-based projects to learn about sci-
ence,technology,engineering,and mathematics (STEM).
This work,titled Building an Internet Community of De-
sign Engineers (iCODE),was funded by a three-year grant
from the National Science Foundation’s (NSF’s) Informa-
tion Technology Experiences for Students and Teachers
(ITEST) program.
266 Auton Robot (2011) 30:265–280
Fig.1 iCODE programparticipants
Working with an open-source course management plat-
form,the iCODE team developed a web system (www. offering dozens of illustrated design chal-
lenges,interactive quizzes,and other instructional materi-
als.The system features a unique web-based programming
tool—a Java applet that enables users to develop code for
microcontrollers in a web browser,without any locally in-
stalled software.The two organizations also developed a
suite of physical hardware,including electronics and robot-
ics kits,to work in conjunction with the iCODE web site.
Using the iCODE web system and tangible learning ma-
terials,the project partners facilitated year-long enrichment
programs for students from local schools and community
centers during the 2006–2007,2007–2008,and 2008–2009
academic years.Comprising weekly after-school sessions,
weekend robotics exhibitions and competitions,and inten-
sive summer camps,the programs were effective in engag-
ing the project’s inner-city youth population,teaching them
computer programming and STEMworkforce skills,and en-
hancing students’ interest in STEM subjects and careers.
Figure 1 shows participants in the program.
This paper describes the iCODE project in detail and
presents lessons learned from its implementation,address-
ing the following topics:
– Challenges associated with retaining middle school and
high school students in the enrichment program,during
the course of each year and fromyear-to-year.
– Issues associated with supporting program leaders in dif-
ferent venues,including both school sites and neighbor-
hood community centers.
– The use of web technology to deliver instructional materi-
als,enable programming of microcontroller projects,and
build a community of users.
– Student responses to the iCODE project’s two tangible
electronic hardware platforms:one a printed circuit board,
the other a built-it-yourself,breadboard-based system.
– A discussion of our community-building work,including
on-line and face-to-face components.
– Student learning outcomes and educator feedback,based
on the three years of analysis by an external project eval-
2 Background
The use of robotics in education has become widespread
in informal settings,with a blossoming of robotics kits
and related K-12 curriculum materials over the last decade.
Foundational work included Seymour Papert’s introduc-
tion of the robot turtle in his implementations of the Logo
programming language (Papert 1980).This line of work
led to commercial implementations of robotics materials
launched by the LEGO Group (Martin and Resnick 1993;
Martin et al.2000) and numerous follow-on products.
At the university level,robotics competitions are now
common.Early instances,such as the IEEE Micromouse
contest,have led to a wide range of national and interna-
tional contests,such as RoboCup,the Intelligent Ground
Vehicles Competition,and the Trinity College Home Fire-
fighting Robot Competition.The competition format has mi-
grated back to the K-12 learning environment,with exam-
ples such as RoboCupJunior (Sklar et al.2003),FIRST Ro-
botics,FIRST LEGO League,and Botball.
Over the past few years,the growing popularity of web
sites featuring user-generated content has led to the forma-
tion of “maker” communities—hobbyists who create elec-
tronics and crafts projects for fun,and share them with
each other at sites like the same
time,the decreased cost of microcontroller hardware and
related electronic components,coupled with the increased
availability of free compilers and other software tools,have
prompted a surge in user-inspired microcontroller-based
projects.This merging of craft and computing is a rich area
of study,and was underway even before the Web 2.0 phe-
nomenon gave it a boost (Eisenberg and Eisenberg 1998).
The iCODE project was inspired by these trends and
drew on the experience and respective strengths of the part-
nering organizations.Prior to devising iCODE,co-author
Martin conducted work in contemporary educational robot-
ics,contributing to in-school implementations and univer-
sity competition formats (Martin 1996).More recently,Mar-
tin developed the Super Cricket and promoted its use in K-
12 outreach programs at UML,including the university’s
highly successful summer DesignCamp program (Baker
Auton Robot (2011) 30:265–280 267
Co-author Christy served as manager of the first Com-
puter Clubhouse,a network of community centers for un-
derserved youth focusing on developing creative technical
skills (Kafai et al.2009).In 2001,Christy founded Machine
Science—a non-profit that operated year-long after-school
engineering programs for underserved Boston youth,using
a breadboard-based microcontroller development platform.
Previously,Machine Science had also developed a num-
ber of innovative web-based resources to support these pro-
grams,including both on-line design challenges and the on-
line code development environment.
3 Project goals
The iCODE project was one of more than 130 projects that
NSF has funded under the ITEST grant program,which was
launched in 2003 in response to concerns at the time about
the availability and skills of the domestic information tech-
nology and STEM workforce.As of 2009,NSF had dis-
bursed a total of more than $140 million through ITEST,
supporting technology development,program implementa-
tion,educational research,and scale-up of effective mod-
els.Having received its initial funding in 2006,iCODE was
in the fourth cohort of funded projects.It was among 22%
of ITEST projects with a primary focus on engineering.
Others have focused on environmental science (30%),com-
puter programming (28%),computer gaming and simulation
(10%),and bioscience (9%).
The iCODE project’s overall objective was to increase
the likelihood that participating middle school and high
school students would pursue IT and STEMcareers later in
life.Within that context,the project had the following spe-
cific goals:
– Goal#1:Enhance participating students’ information
technology fluency.Throughout the year,students com-
pleted a range of IT-intensive microcontroller-based pro-
jects,requiring real engineering and computer program-
ming skills.
– Goal#2:Increase awareness among participating stu-
dents about educational and career opportunities in IT and
STEM.Students visited university campuses during the
iCODE summer sessions and met with IT professionals
at biannual IT and STEMcareer events.
– Goal#3:Connect participating students to a commu-
nity of like-minded peers and adults.Participating stu-
dents worked side by side with undergraduate mentors in
the year-long iCODEafter-school programs.Participating
students were also encouraged to create on-line portfolios
of projects within an existing web database of inventions.
Table 1 presents the underlying theory of change for each
project goal.
4 Programstructure
The iCODE enrichment program was designed to meet
guidelines established by the ITEST grant program,which
mandated a minimum of 120 hours of contact time for each
participating student,including a summer component last-
ing at least two weeks.In keeping with these guidelines,the
iCODE experience encompassed the following elements:
– After-school sessions.Every participating school and
community center offered weekly after-school sessions,
led by a teacher or staff member,with assistance from
an undergraduate mentor,using the design challenges
and other resources available on the iCODE web system
(24 sessions @2 hours = 48 hours).
– Weekend events.In each grant year,iCODE students par-
ticipated in UML’s Botfest,a non-competitive exhibition
on the UML campus,and Machine Science’s Robot Sumo
Tournament,an autonomous robotics competition held at
the Museum of Science Boston (2 events @ 6 hours =
12 hours).
– Summer camps.The iCODE experience culminated in
two-week summer camps at the University of Massa-
chusetts campuses in Boston and Lowell.The camps gave
students the opportunity to take their computer program-
ming skills and creative abilities to the next level,com-
pleting collaborative projects of their own design (10 days
@6 hours = 60 hours).
Within this framework,UML and Machine Science of-
fered two separate,but parallel,iCODE content tracks.Mid-
dle schoolers from Lowell and Lawrence,Massachusetts,
worked with the Super Cricket,a printed circuit board con-
troller programmed in Logo.High school students from
Boston,Massachusetts,worked with Machine Science’s
breadboard development kits,physically wiring components
to a PIC microcontroller and programming the chip in C.
Despite these differences in the hands-on materials,students
in both content tracks were challenged to master a similar
progression of knowledge and skills (Table 2).Moreover,
twice a year,participants from both tracks came together to
exhibit their robots and competed against one another in the
robot sumo contest on equal footing.
5 Student recruitment and retention
The iCODE enrichment programs were run at five school
and community center sites in Boston and Lowell/Lawrence
during 2006–2007,and at 10 sites during 2007–2008 and
2008–2009.In each year,the programs served a racially
diverse and economically disadvantaged urban population.
Over the three-year grant period,a total of nearly 250 stu-
dents participated,with many staying involved for multiple
268 Auton Robot (2011) 30:265–280
Table 1 iCODE theory of change
years.Recruitment of students was left to the teachers at
participating schools.Teachers were encouraged to identify
students with an interest in engineering and technology,but
received no formal recruitment guidelines.
In every year of the grant,male students outnumbered fe-
male students by a ratio of 2:1 or greater.This enrollment
pattern was a disappointment to the project team,but com-
parable to other programs with a strong focus on engineer-
ing,computer science,and robotics.Among the relatively
small number of female students who joined the program,
many excelled.Girls consistently ranked among the top per-
formers in the annual sumo robotics tournament.In addi-
tion,in their responses to pre- and post-program surveys,
some female students appeared to make stronger gains than
their male counterparts on items dealing with attitudes to-
ward STEM subjects and careers.Table 3 presents the de-
mographic characteristics of the iCODE student population.
In planning for the project,the iCODE project team had
predicted that roughly half of each year’s participants would
return to the program the following year,producing the
target retention figures shown in Table 4.This prediction
proved to be roughly accurate,and the project met its overall
student recruitment goals.
For both middle school and high school students,
120 hours of participation represented a considerable time
commitment.Although nearly all students enjoyed and ben-
Auton Robot (2011) 30:265–280 269
Table 2 iCODE program:
required skills
Level Objective Skills
Learn to use tools and
build simple circuits.
– identify basic components:e.g.,wires,switches,
– construct basic circuits:e.g.,simple circuit
– understand basic concepts:e.g.,power,ground,
short circuit,voltage,resistance,amperage
– use tools properly and safely
Build programmable
circuits and write simple
– identify components in programmable circuits:
e.g.,microcontrollers,programming linkages,
USB cables
– connect input and output devices to the micro-
– write a program and download it to the micro-
– understand persistence of program in the con-
Complete guided
building and
programming projects.
– write programs to control output devices:e.g.,
– write programs to use input from sensors:e.g.,
temperature,light,infrared distance sensors
– declare and manipulate variables
Apply building and
programming skills to
structured design
– write programs that use advanced structures:
e.g.,sub-procedures,digital and analog in-
puts,loops and conditionals,logical operators
– develop a working solution to a structured de-
sign challenge,using an appropriate subset of
the above programming elements
Apply skills in
independent projects and
mentor peers.
– formulate and complete a novel design chal-
lenge,connecting hardware in new ways
– teach circuit building and programming to oth-
– render project guidance or debugging assistance
to peers
– teach others how to use tools
efited from the program experience,some found it difficult
to maintain their attendance in weekly after-school sessions
through both the fall and spring semesters,as well as a two-
week summer session.At some sites,the full-year iCODE
programcommitment conflicted with seasonal sports sched-
ules and other extracurricular activities.Convincing stu-
dents from the after-school sessions to attend the program’s
summer component was challenging,particularly for the
Boston-based programs,whose low-income,high school-
aged participants were strongly motivated to seek summer
employment.For some students,mandatory summer school
precluded their participation in the summer program.
The iCODE program teachers were asked to be some-
what flexible about attendance,so as not to lose students
with extracurricular conflicts.However,this was a delicate
balance to strike,as students were expected to complete
an ambitious sequence of increasingly complex hands-on
projects and to acquire a substantial set of new engineer-
ing and programming skills.Students who missed sessions
would get out of step with their peers,and require extra as-
sistance to catch up.
The project team did anticipate attrition over the course
of the school year;a recent Massachusetts study estimated
that the typical after-school program has an attendance rate
of 59% (United Way of Massachusetts Bay & Merrimack
Valley 2005).Program teachers over-recruited at the be-
ginning of each academic year in anticipation of this,and
planned recruitment goals were met (Table 4).
To analyze retention rates for males and females,reten-
tion was defined as the number of programparticipants who
completed both the pre- and post- programsurveys,as com-
pared to the total number students who completed the pre-
survey.With this definition,the overall retention of students
from the iCODE school-year program was 47%.This esti-
270 Auton Robot (2011) 30:265–280
Table 3 iCODE student demographics:2006 to 2009
Characteristic Attribute 2006–2007 2007–2008 2008–2009
Gender Female 32% 18% 21%
Male 68% 82% 79%
Grade 6
6% 2% 7%
27% 35% 37%
12% 13% 23%
17% 10% 10%
25% 14% 3%
8% 19% 0%
6% 8% 11%
Ethnic and American Indian or 3% 2% 0%
racial Alaska Native
background Asian 19% 20% 25%
Black or 35% 23% 24%
Spanish/Hispanic 19% 31% 37%
or Latino
White 26% 26% 22%
Table 4 iCODE student recruitment and retention:2006 to 2009
Student 2006–2007 2007–2008 2008–2009 Total
Group Target Actual Target Actual Target Actual Target Actual
year 50 40 75 66 51 103 176 209
year – – 25 12 37 23 62 35
year – – – – 12 5 12 5
Total 50 40 100 88 100 131 250 249
mate is likely to be low,because it does not take into account
students who may have been absent from the program on
the days during which the post-program survey was admin-
istered.The retention rate was comparable for males (46%)
and females (49%).
6 Support for programsites
In rolling out the iCODE program,UML and Machine Sci-
ence found few teachers at the partnering program sites
with knowledge of the program’s core content:electronic
circuitry,embedded computing,and C and Logo program-
ming.The iCODE program did not simply ask teachers
to teach their STEM subjects in a new,computer-intensive
way;it asked them to teach an entirely new body of knowl-
edge and skills.As a result,most teachers required sig-
nificant training and support to implement the programs.
Since the partnering schools and community centers were
all located in Boston,Lowell,and Lawrence,UML and Ma-
chine Science were able to provide this support directly.
Teachers received individual on-site guidance in setting
up the programs prior to the start of the academic year,
and assistance from undergraduate mentors throughout the
The iCODE partners had more success with programs
that were based in schools than with those that were based
in community centers,such as Boys and Girls Clubs.The
classroomteachers who led the school-based programs were
able to draw on their history of interactions with particu-
lar students during the school day when recruiting partici-
pants for the program.They also were also well-positioned
to remind students about upcoming iCODE program ses-
sions and events,and experienced at encouraging students
to persevere when confronted with academically challeng-
ing material.In contrast,at community center sites,the re-
lationships between staff and youth were generally more in-
formal,and youth visitors were not always accustomed to
attending after-school programs on a consistent basis from
Auton Robot (2011) 30:265–280 271
Fig.2 On-line programming
week to week.To succeed in this environment,programs
like iCODE would need to articulate very clear expectations
in terms of student attendance and benchmarks.Also,they
might benefit froma more-intensive schedule,with meetings
held on more days of the week for a shorter overall time pe-
7 Web technology
A major goal of the iCODE project was to use web tech-
nology to facilitate the enrichment programs at each partner
site.Students used the iCODE web systemto access project
instructions,as well as the on-line programming tool.Avail-
able from any Java-enabled web browser,the programming
tool presents students with a full-featured code develop-
ment environment,complete with syntax highlighting for
supported languages (C and Logo).With a single mouse
click,a student can send a code file to the server to be com-
piled.The server instantly returns the compiled code,along
with any compiler messages.At that point,the student either
modifies the code or,with another mouse click,downloads
it to the microcontroller.The applet communicates directly
with a bootloader on the chip through a virtual serial port on
the student’s computer.The programming window supports
multiple hardware platforms.Using the options menu,stu-
dents can move seamlessly between developing programs in
Logo for the Super Cricket and developing programs in C
for the PIC microcontroller.Figure 2 shows the on-line pro-
gramming window.
After students had completed projects,they were encour-
aged to post images,videos,and descriptions to the iCODE
system’s invention database,an on-line portfolio of projects.
The project database enabled students to showoff their work
to friends,siblings,and parents,comment on one another’s
finished projects,and find inspiration for future projects.
Moodle,an open-source course management system
originally developed for universities,provided the frame-
work for student account management and web content de-
livery.It was well suited to this purpose,offering a range of
useful built-in features,including a WYSIWYG editor for
Fig.3 Super Cricket (left) and Machine Science breadboard kit (right)
course content,quiz development tools,threaded discussion
forums,and a highly configurable scheme for setting teacher
and student permissions.
The iCODE team had hoped that using the Java-based
programming tool would avoid some of the difficulties asso-
ciated with installing and maintaining software in schools,
including the need to support multiple operating systems,
the logistical problem of distributing software updates,and
the understandable reluctance of some school administrators
to permit the installation of large software packages (like C
compilers).In practice,the use of the web-based program-
mer involved challenges of its own.Frequent updates were
needed to keep up with Java releases and newversions of the
PC and Mac operating systems.Also,it was not completely
free of installation hassles:teachers needed an administra-
tive log-in in to install a small DLL library file and the USB
drivers that create the virtual serial port for communication
with the microcontroller.
8 Hands-on materials
The iCODE project’s hands-on materials included two
unique microcontroller development platforms (Fig.3),to-
gether with related robotics hardware and building mate-
rials.UML’s Super Cricket is a printed circuit board,pro-
grammed in Logo;Machine Science’s platform is a bread-
board development kit,with a PIC microcontroller intended
to be programmed in C.
The physical design of the Cricket and the breadboard
were thought to be suitable for the different grade levels in-
272 Auton Robot (2011) 30:265–280
Fig.4 iCODE student projects
volved in the iCODE project.The Cricket’s dedicated con-
nectors make circuit-building relatively straightforward—
students simply plug in sensors and actuators to the appro-
priate jacks.Machine Science’s breadboard approach chal-
lenges students to build circuits by hand,physically wiring
components to the microcontroller.The computer code for
the two devices is another key differentiator:Logo is a sim-
ple programming language whose commands and syntax
can be quickly learned by novice programmers;C is a pow-
erful language widely used by professionals for embedded
computing applications.
In general,the iCODEprogram’s hands-on materials held
high appeal for participants.Students seemed intrigued by
the opportunity to learn about technologies that have be-
come ubiquitous in modern life,embodied in devices such
as cell phones,MP3 players,handheld electronic games,
GPS receivers,etc.Some high schoolers found Machine
Science’s breadboard-based projects frustrating,since these
projects involved both wiring and coding challenges.As
anyone with experience in embedded computing will at-
test,it can be very difficult to troubleshoot a project without
knowing whether the hardware,the software,or both are the
source of the problem.At the same time,many students ap-
preciated having the chance to work with real,off-the-shelf
electronic components,and felt that this made the program
experience more authentic.
During the project’s third year,students who had worked
in previous years with the Cricket and Logo programming
chose to transition to the breadboard platform and C pro-
gramming,as a learning challenge.The mentor working
with these students reported that students’ work was suc-
Each partner brought a pre-existing hardware platformto
the iCODE collaboration.The research team did not inves-
tigate whether the two different hardware platforms resulted
in differences in student learning and outcomes.
Figure 4 illustrates the range of projects completed by
students in the iCODE program,including robots,games,
electronic fashion accessories,and other programmable cre-
9 Community building
Various aspects of the iCODE project were intended to build
a sense of community among students and connect them
with adult mentors and role models.Participating middle
and high school students were given the chance to work on
their iCODE projects with help from undergraduate engi-
neering and computer science students.They met with col-
lege students and admissions staff at IT and STEM-focused
career events,and shared their work with parents,siblings,
and the broader community at UML’s BotFest and Machine
Science’s Robot Sumo Tournament.Also,participants were
encouraged to form an on-line community through use of
a threaded discussion forum and the web-based invention
According to educator and student feedback,the presence
of undergraduate mentors in iCODE classrooms was one of
the more successful program elements.Depending on the
host teacher’s abilities,the mentor’s role ranged fromclass-
room assistant to lead instructor.Mentors helped explain
the iCODE content,assisted students with their hands-on
projects,challenged students to expand their project ideas,
and provided programming and troubleshooting expertise,
as needed.Mentors were particularly adept at gauging the
potential of individual students and challenging them with
questions and activities to help them go beyond their ac-
quired expertise.Because the mentors were relatively close
in age to the participating students,and because the mentors
were themselves still in school,the iCODE program’s mid-
dle school and high school students were inquisitive about
the mentors’ college classwork and career plans.
Machine Science’s annual Robot Sumo Tournament was
consistently well attended and cited by educators as a strong
motivator for students to build sophisticated autonomous ro-
bots.Holding this event in exhibit space at the Museum of
Science Boston provided visibility for the iCODE program,
attracted news media attention,and increased the excitement
level for competitors.Over the three-grant period,more than
120 robots were entered in the competition.
As for the on-line invention database,students made
some limited use of this system feature,posting more than
Auton Robot (2011) 30:265–280 273
200 project images and descriptions,but it never became
a popular gathering point for iCODE participants.The in-
vention database suffered from security and other technical
problems,and its appeal to students may have paled in com-
parison to other social networking alternatives,such as Face-
book and MySpace,which grewrapidly in popularity during
the three-year grant period.
10 Programevaluation
Goodman Research Group (GRG) of Cambridge,Massa-
chusetts,served as the project’s external evaluator.Over the
course of the project,student learning outcomes were as-
sessed using a variety of measures:structured observations
of iCODE program sessions;group interviews with partici-
pating students;pre- and post-programstudent surveys,con-
taining both Likert-scale and open-response items;educator
surveys and interviews;and content-specific quizzes,admin-
istered three times annually.Results were reported in three
broad areas:(1) appeal and effectiveness of iCODE pro-
gram components;(2) gains in engineering and program-
ming skills;and (3) preparation for STEMcareers.
The qualitative data from open-ended survey questions,
interviews,and observations were coded.Themes were clus-
tered and,where appropriate,counted.Then narrative ac-
counts emphasizing patterns,commonalities,and differ-
ences were generated.Where possible,data from different
sources were triangulated to corroborate findings.Results
regarding students’ understanding of the work of engineers,
presented in Sect.10.3,are an example of insights gleaned
fromthe qualitative evaluation work.
10.1 Appeal and effectiveness of programcomponents
In their survey responses,students rated the major compo-
nents of the iCODE programhighly (Table 5).Students par-
ticularly enjoyed completing hands-on technology projects
and working closely with the program’s undergraduate men-
tors.Both of these components earned a mean rating near
or above 4.0 on the survey’s five-point scale (1 = Poor,
2 =Fair,3 =Good,4 =Very Good,5 =Excellent).
Based on discussions with educators and students,GRG
reported that the following program components were most
– Hands-on activities.A common theme throughout the
three years of the iCODE program was the high appeal
among the students of the hands-on aspect of the various
projects.The students enjoyed working on the projects
and experiencing success with a project.The projects
gave them an opportunity to be creative within the struc-
tured programcurriculum.
– Project progression.Participating educators were satis-
fied with the structure of the program,which involved a
gradual progression from introductory activities to more
open-ended projects.Educators encouraged students to
reflect on their own work,communicate with others,and
document their progress,leading to an increase in student
– Balance between autonomy and collaborative work.
There was consensus among the educators that the iCODE
system’s on-line guides and programming portal allowed
the students to be independent.During the after-school
sessions,students closely followed instructions from the
project guides and loaded code files from the guides into
their projects.Nevertheless,students felt responsible and
took ownership for their projects.At the same time,the
students had a number of opportunities to work in small or
large groups and with the mentors,leading to community
building among students and the mentors.
– Culminating events.The participation in Botfest and the
Robot Sumo competition,which signified the evolution
of students’ hands-on projects,were important aspects of
the program,according to the educators.The summer ses-
sion,being more intensive,increased the students’ en-
gineering knowledge and skills to a large extent.Work-
Table 5 Student ratings of
iCODE programcomponents
Programcomponent 2006–2007 2007–2008 2008–2009
(N =43) (N =52) (N =69)
Hands-on projects 4.40 4.50 4.30
On-line resources 3.70 4.05 3.60
In-person visits from
engineering and technology 3.86 3.90 3.60
Internet-based interactions 3.70 3.90 3.50
with industry mentors
Collaboration with 4.10 4.12 3.80
student mentors
274 Auton Robot (2011) 30:265–280
A focused group of mostly girls using creativity and imagination to engineer high-tech
garments and accessories.
A GRG researcher observed a summer camp session for about an hour.Nine girls and three
boys,all from the 9
grade,participated.An educator and an undergraduate mentor were
present to instruct the students.The students worked on projects that had been approved by
the educator earlier in the week.Projects included:lamps with LED lights,LED night-lights,
picture frames with etched Plexiglas and LEDs,and purses and T-shirts embellished with
LEDs.The students had new computers and ample supplies.There was some structure to the
session,but the teacher gave the students a lot of flexibility,which they seemed to appreci-
During the observation period,the educator supervised the class and helped one-on-one when
needed.The educator explained to the GRG researcher that,during a previous session,stu-
dents had struggled to embellish T-shirts with LEDs.After that day,the students learned the
value of making a design plan and using it to progress step-by-step through a project.They
researched and developed plans for their projects,and the educator approved each plan indi-
vidually.By the day of the observation,the students were all a few steps into their projects,
revising their design plans as needed.
The GRG researcher observed students wiring lights,wiring their project items,creating
the structure for their projects,testing lights,and writing code to control their lights.They
seemed to be enjoying the activity and the fact that they got to choose a project and make
their own personal design choices.The students also got to take home whatever they made,
which seemed to have a positive influence on their progress and enthusiasm for finishing the
Fig.5 Observation of UMass Lowell summer camp session
ing on goal-oriented projects,whether for the competi-
tion or through the summer camps,added immensely to
the learning experience of the students.
Figures 5 and 6 encapsulate evaluator observations of the
summer camp sessions.
10.2 Gains in engineering and programming skills
For most students,the iCODEexperience provided their first
opportunity ever to engage in this type of activity (Table 6).
The vast majority of participants (80%) had never written
a computer program prior to joining the iCODE program.
There were statistically significant differences between the
baseline experience of the middle school students involved
in the programs run by UML and that of their high school
counterparts in the Machine Science programs.
By the end of the program,nearly all students reported
at least a little increase in their understanding of computer
programming and electronic devices.In both of these ar-
eas,59% of students felt their understanding had increased
a great deal (Table 7).
Notably,students’ previous experience did not affect
their perceptions of what they learned from iCODE.Stu-
dents who had previous experience building electronic cir-
cuits were just as likely to report gains in their understanding
of electronic devices at the end of the programas those who
had no prior experience.The same was true with respect to
writing computer programs.
On average,participants in each iCODE cohort left their
summer camp feeling that they could perform the required
engineering and programming skills pretty well and,with
enough time to review,they could lead friends in performing
the skills.Table 8 shows mean student responses on a five-
point scale (1 =I cannot do this,2 =I can do this but only
with assistance,3 =I can do this well enough for my own
personal use,4 =I can do this pretty well,and could showa
friend how to do it if I had time to review,and 5 =I can do
this very well and could show a friend how to do it).
10.3 STEMcareer preparation
The iCODE program provided students with opportunities
to work together,increased their perceived problem-solving
Auton Robot (2011) 30:265–280 275
Students building and programming wireless text messaging devices,supported effec-
tively by their teachers and strongly motivated by the project technology.
A GRG researcher observed a session during the summer camp at the Machine Science site.
Three members of Machine Science staff were present as instructors.There were a total of six
male and one female high school students,representing grades 10 to 12.There were plenty
of computers and space for everyone.Each student had a computer to work on and the other
required supplies.There were two projectors at the front of the room for the teacher to post
directions and demonstrate code development.There was also a white board for the teacher
to use for additional visual aids.The web technology seemed to be working well.
On the day of the observation,students were starting a new project:building wireless text
message devices.The educator introduced the code and the instructions,wrote notes on the
white board,and talked through some of the web site content.Then,the students went to
work on their own,going through the instructions with the machines.During this time,Ma-
chine Science staff members went around the roommultiple times helping students and doing
individualized instruction.
The educator had a very gentle and supportive approach with the students.She seemed to
encourage the students to figure out their own problems by doing a combination of asking
scaffolding questions and explaining new concepts.
The students were at the computers with their machines,writing the code and trying it out.
They mostly used the directions on the Machine Science website,although they also got
assistance from the instructors.They were learning about how phones use codes to translate
numbers into letters to do text messaging.Later that day,they were going to learn to add a
radio frequency component to the machines and try sending texts to each other.The students
were very engaged and focused on learning the code and programming their text messengers.
Fig.6 Observation of Machine Science summer camp session
Table 6 Relevant experience
prior to iCODE
p <.001
p <.05
Experience UML Machine Science Total
N =154–157 N =104–107 N =261–262
Completed an engineering
project before iCODE
27% 52% 37%
Built electronic circuits
before iCODE
41% 41% 41%
Wrote a computer program
before iCODE
15% 27% 20%
Table 7 Perceived extent to
which iCODE increased
students’ IT understanding
N =145
Content Area Not at all A little Some A great deal
Computer programming 2% 6% 32% 59%
Electronic devices 1% 9% 31% 59%
276 Auton Robot (2011) 30:265–280
Table 8 Students’ perceived skill level post summer camp
Rating UML Machine Science
2007 2008 2009 2007 2008 2009
(N =11) (N =34) (N =35) (N =11) (N =13) (N =7)
Mean 3.74 4.25 3.64 3.56 3.79 4.07
Minimum 2.30 2.64 1.18 2.22 2.30 3.00
Maximum 4.70 5.00 5.00 4.80 4.80 5.00
Table 9 Educator and student
comments on iCODE program
Source Comment
Educator “The iCODE programis a total package of designing,building,coding,testing,
documenting,and assessment.This approach is very effective in accomplishing
the programobjectives.”
Educator “The online guides allow the students to be very independent (learn without a
teacher),and also encourage themto work with their peers.”
Student “I liked the fact [that] no matter what we did,it was ours,whether we failed or
achieved,it was what we did,there was no handholder or anything,aside from
the basic explanation,so it was kind of being your own boss and getting what
you wanted to get done.”
Educator “I enjoyed having the mentor at the program.She was knowledgeable about the
projects,and could help troubleshoot when students were having problems.
The students felt comfortable asking her questions,and she was very good at
explaining the concepts in the project.”
Student “If you’re at all interested in electronics or anything,it’s [iCODE] something
hands-on.Instead of looking it up on the internet and having to read everything
about it,you can actually learn about it [electronics] here.”
Student “iCODE far exceeded my expectations.I was looking for a programto allow
me to build,and program.iCODE is far more than that.”
Table 10 Perceived extent to
which iCODE increased
students’ workforce skills
N =143–145
Not at all A little Some A great deal
Provided students with opportunities to
work together with other students
(N =97)
5% 12% 29% 55%
Increased students’ problemsolving
abilities (N =99)
9% 17% 34% 40%
Connected students with professionals in
the fields of engineering and technology
(N =145)
8% 18% 27% 48%
abilities,and connected themwith professionals in the fields
of engineering and technology.All of these are impor-
tant factors in encouraging workforce readiness.Qualita-
tive evidence in the form of selected educator and student
comments are presented in Table 9,and quantitative evi-
dence from analysis of survey replies is presented in Ta-
ble 10.
As shown in Table 11,more than three-quarters of par-
ticipants reported that the iCODE programexposed themto
information about careers in science and technology either
some (30%) or a great deal (48%).Likewise,a strong ma-
jority of respondents indicated that the program experience
improved their attitudes about careers in science and tech-
nology either some (36%) or a great deal (40%).
As an additional indicator of iCODE effectiveness in ex-
posing students to STEM careers,on the pre- and post-
program surveys each year students were asked “What do
engineers do?” There were 101 cases in which students an-
swered at both pre and post.These responses were coded
for accuracy,completion,and sophistication.As Table 12
illustrates,about half of the students (N = 50) gave more
accurate or sophisticated responses after the program than
they did before it.In most of these cases (N =35),the post-
response was substantially better than the pre-response;in
Auton Robot (2011) 30:265–280 277
Table 11 Perceived extent to
which iCODE increased STEM
career preparation
N =144–145
Not at all A little Some A great deal
Exposed students to information about
careers in science and technology
6% 16% 30% 48%
Improved students’ attitudes about
careers in science and technology
8% 17% 36% 40%
Table 12 Student responses to
“What does an engineer do?”
N =101
Post-response Post-response Post-response Post-response
less adequate same as marginally better substantially better
than pre- pre- than pre- than pre-
10%(N =10) 40%(N =41) 15%(N =15) 35%(N =35)
the other cases (N =15),the post-response was marginally
better than the pre-response.
Marginal improvements were defined as slightly more ac-
curate,complete,or sophisticated responses at post- than at
pre-.Examples included:
– Changes fromone- or two-dimensional responses at pre to
two- or multi-dimensional responses at post (e.g.,“build”
to “build or create”;“create and fix things” to “design,
create,and fix things”)
– Identification of a tool at post (e.g.,“build stuff” to “create
things with computers”)
– Identification of a purpose at post (e.g.,“They fix stuff or
make stuff” to “They design and build stuff for people’s
lives”) or identification of the focus of building at post
(e.g.,“design stuff” to “design bridges,roads”)
Substantial improvements fell into similar categories but
were quite a bit or a great deal more accurate or com-
plete/sophisticated responses at post- than at pre-.These in-
cluded eight students’ responses that changed from “I don’t
know” at pre- to,at post-,giving an explanation of what an
engineer does (e.g.,design,build,program computers,fix
things,work with technology).
Other changes that qualified as quite a bit or a great deal
more sophisticated started with simple responses such as
“make stuff” and ended with somewhat more elaborate re-
sponses,such as “design things and solve problems to make
human life easier.” The other half of students gave fairly
similar responses pre- and post- or gave a scantier response
at post-.
11 Discussion
The iCODE project produced significant outcomes for par-
ticipating students.The programeffectively engaged partic-
ipants in meaningful design projects.In completing these
projects,students developed real engineering and program-
ming skills,and their attitudes toward STEM subjects im-
proved.The iCODE program also provided students oppor-
tunities to work closely with undergraduates and to spend
two weeks per year at a college campus.Taken together,all
of these program elements increased many students’ inter-
est in STEM career pathways.To conclude,we reflect on
the challenge of evaluating programs like iCODE,the sus-
tainability of the project,and some larger issues raised by
this work.
11.1 Reflections on evaluation
As in other studies on informal education with robotics
(Nourbakhsh et al.2004;Nugent et al.2009),the iCODE
evaluation examined student learning in the areas of engi-
neering and programming,teamwork and other workforce
skills,and attitudes toward STEMsubjects and careers.Stu-
dents self reported their learning in written and online sur-
veys.The iCODE study also tested two other data-collection
methods.During the first summer camp,students completed
journals and during the second and third years of the school-
year program,students had the opportunity to complete on-
line quizzes.
Similar to other informal robotics programstudies (Wein-
berg et al.2007),the iCODE evaluation featured a pre- and
post-test design in which participating students completed
on-line or written surveys at the beginning and end of the
program.Had budget permitted,the inclusion of a compari-
son or control group would have allowed for stronger attri-
bution of observed outcomes to the program,as in the Nu-
gent et al.(2009).
The iCODE study did not include any mid-point or
follow-up surveys,as some other studies have (Nourbakhsh
et al.2004),although a small number of students partici-
pated in the program for more than one year,thus provid-
ing a limited longitudinal data set.In general,the more data
collection points,the better for determining the trajectory of
278 Auton Robot (2011) 30:265–280
student learning.In addition,follow-up surveys are desirable
for examining maintenance of programeffects or for observ-
ing effects beyond the boundaries of program participation
(e.g.,pursuit of relevant activities).
The iCODE evaluation was “whole-program” focused:it
included all students and educators at all sites.There was no
in-depth evaluation of specific students or sites as has been
done in other research.Had it been possible,such a supple-
ment may have resulted in more detailed understanding of
student learning.In addition,iCODE program recruitment,
particularly for the Machine Science curriculum,did not ex-
clude students who had previously participated in the pro-
gram.The presence of experienced students may have lim-
ited the observed impact of the program.
11.2 Sustaining iCODE
Going forward,the iCODE partners are committed to main-
taining the iCODE web site as a resource for educators,
students,hobbyists,and others interested in learning about
microcontroller-based projects.In addition to the more than
300 teachers and students from the 2006–2009 iCODE en-
richment programs,some 1,800 additional users have reg-
istered accounts on the web site,in order to access instruc-
tional materials and the on-line programming tool.This reg-
istered user base includes educators and students from pub-
lic and private secondary schools,as well as colleges and
Machine Science’s breadboard kits,in particular,have
attracted interest for use as college-level teaching tools
at Northeastern University,MIT,Tufts University,Norfolk
State University,Henderson State University,and the Uni-
versity of North Texas,among others.UML and Machine
Science have also made the project’s tangible learning ma-
terials available for sale on the web.It is hoped that revenues
from sales of these materials will help sustain the project’s
momentum,and continue the enrichment programs in local
public schools.
11.3 Larger issues
The results of our work and those of many other studies
clearly demonstrate that students enjoy hands-on,design-
rich experiences with robotics,and show strong gains in
content knowledge as a result of these programs.Having
self-selected for participation,many students entered the
iCODE programwith generally positive attitudes toward en-
gineering and technology.This phenomenon has also been
reported by the other researchers previously cited (Nour-
bakhsh et al.2004;Miller et al.2005;Weinberg et al.2007).
In the case of iCODE,these attitudes were maintained,but
more importantly,they were also given much clearer shape
and definition.Students understood much better after com-
pleting the program what professional engineers and pro-
grammers actually do.
The iCODE program’s impact on career intentions var-
ied from student to student.For some participants,we be-
lieve the iCODE program was truly transformative,leading
to changes in their perceptions of technology,their educa-
tional aspirations,and their college and career plans.This
is evidenced by students who stayed with the program for
multiple years.When they finished middle school,a small
group of iCODE students organized a robotics club at their
high school to continue developing their interest.
It seems clear that the larger structural challenge is to pro-
vide students with a continuumof learning experiences such
as those developed by the iCODE project.Several of our
teachers have commented that they enjoy teaching iCODE
because it allows them to engage students in technical con-
tent in a deeper fashion that is possible in the conventional
school day.In the American system of education,opportu-
nities for students to pursue interests in depth tend to occur
outside of the school day—for example,in sports practice
or music rehearsals that can consume 15 or more hours per
week.But there exist comparatively fewopportunities to en-
gage students in such a powerful fashion in science and tech-
Furthermore,the separation of academic preparatory
schools fromvocational institutions is problematic.Students
on the academic track are deprived of hands-on experiences,
while students in the vocational track are not offered the
opportunity to connect their concrete knowledge with the
symbolic skills with which they are surmised to struggle.
Neither group benefits fromthis arrangement.
We believe that offering a range of students the honest,
challenging,and sometimes frustrating experience of engi-
neering in a significant way is a critical need.Students who
never knew they had this interest will discover it.It is then
up to us to continue to provide them with opportunities to
nurture this interest towards meaningful,satisfying career
Baker,D.(2009).Design camp:an alternative model for D&T educa-
tion?D&T Practice.URL
Eisenberg,M.,& Eisenberg,A.N.(1998).Shop class for the
next millennium:education through computer-enriched handi-
crafts.Journal of Interactive Media in Education,98(8).URL
Kafai,Y.,Peppler,K.,& Chapman,R.(2009).The computer club-
house:constructionism and creativity in youth communities.
Teachers College Press,New York.
Martin,F.G.(1996).Ideal and real systems:a study of notions of
control in undergraduates who design robots.In Y.Kafai & M.
Resnick (Eds.),Constructionism in practice:designing,thinking,
and learning in a digital world.Mahwah:Lawrence ErlbaumAs-
Auton Robot (2011) 30:265–280 279
Martin,F.,& Resnick,M.(1993).LEGO/Logo and electronic bricks:
creating a scienceland for children.In D.Ferguson (Ed.),Ad-
vanced educational technologies for mathematics and science
(pp.61–90).Berlin Heidelberg:Springer.
To mindstorms and beyond:evolution of a construction kit for
magical machines.In A.Druin & J.Hendler (Eds.),Robots for
kids:exploring newtechnologies for learning (pp.9–33).San Ma-
teo:Morgan Kaufmann.
Miller,L.,Shearer,S.,& Moskal,B.(2005).Technology camp 101:
stimulating middle school students interests in computing.In
ASEE/IEEE frontiers in education conference,S1F–26.
mal measures of learning in a secondary school mobile robotics
course.In IEEEinternational conference on robotics and automa-
tion (Vol.2,pp.1831–1836).
Nugent,G.,Barker,B.,Grandgenett,N.,& Adamchuk,V.(2009).
The use of digital manipulatives in K-12:robotics,gps/gis and
programming.In ASEE/IEEE frontiers in education conference,
Papert,S.(1980).Mindstorms:children computers,and powerful
ideas.New York:Basic Books.
Sklar,E.,Eguchi,A.,& Johnson,J.(2003).Robocupjunior:learning
with educational robotics.In Lecture notes in computer science,
RoboCup 2002:robot soccer world cup VI (pp.238–253).Berlin:
United Way of Massachusetts Bay & Merrimack Valley (2005).Path-
ways to success for youth:what counts in after-school.Tech.
rep.,Intercultural Center for Research in Education and National
Institute on Out-of-School Time.URL
Stein,C.(2007).The impact of robot projects on girls’ attitudes
toward science and engineering.In Robotics science and sys-
tems (RSS) workshop on research in robots for education.URL
Fred G.Martin is an Associate
Professor in the Computer Science
department at the University of
Massachusetts Lowell.He directs
the Engaging Computing Group,
which develops and studies the use
of technological materials for engi-
neering and science education at the
K-12 and university levels.
Previously,Fred was a research sci-
entist at the MIT Media Laboratory,
where he developed a series of edu-
cational robotics materials,and laid
the foundation for the breakthrough
LEGO Mindstorms Robotics Inven-
tion System,which was launched in 1998.
Fred helped launch the worldwide robot contest phenomenon by co-
founding the MIT Autonomous Robot Design Competition,and by
publishing the textbook,“Robotics Explorations” (2001),along with
the Handy Board robot controller platform.
Fred holds a Computer Science (1986),an Mechanical
Engineering (1988),and a Media Arts in Sciences (1994),all
fromthe Massachusetts Institute of Technology.
Fred also contributes to Gleason Research,a robotics company he co-
founded with his wife Wanda Gleason,has consulted on a educational
projects in the United States and across the world,including Ireland,
Brazil,Thailand,Mexico,and Colombia.
Michelle Scribner-MacLean is an
assistant professor of science and
math education at the Graduate
School of Education at the Univer-
sity of Massachusetts Lowell.She
holds an Science and Math
Education,an Curriculum
and Instruction from the University
of MA Lowell and a Bi-
ology from the University of Ne-
braska Kearney.Her research inter-
ests include assessment and eval-
uation of science,math,and engi-
neering learning in elementary and
secondary students.She currently
serves as an assessment expert and curriculum developer on two
STEM-related NSF grants with Dr.Fred Martin.Dr.Scribner-MacLean
has extensive teaching experience in formal and informal settings.As
a science educator at the Boston Museumof Science for over 20 years,
she designed and implemented courses for preschoolers through adults,
designed teacher professional development experiences,and led safaris
to East Africa.She has also taught in the elementary and secondary
settings,in addition to her teaching experiences as a faculty member
specializing in curriculumand instruction and assessment.
Sam Christy is a co-founder of
Machine Science Inc.,overseeing
the organization’s technology de-
velopment and product commer-
cialization.Sam began his career
with the John F.Kennedy Library
Foundation,where he coordinated
community-service projects per-
formed by Boston-area teenagers.
In 1992,Mr.Christy founded Sci-
ence Bridge—an after-school sci-
ence workshop for high school stu-
dents.Operating from a storefront
in Chelsea,Massachusetts,Science
Bridge served over 50 students each
week with mentoring assistance from undergraduate and graduate stu-
dents.Mr.Christy went on to become the first manager of the Boston
Computer Museum’s Clubhouse Program,where he led the develop-
ment of a computer learning center visited annually by some 4,000
students.In 1996,Mr.Christy founded WordStream—a private com-
pany based on a language processing technology that he invented and
patented.As the company’s chairman,he raised over $5 million in ven-
ture capital funding and grew the company to more than 30 full-time
staff members.
Ivan Rudnicki is a co-founder of
Machine Science Inc.,where he su-
pervises the company’s curriculum
development,strategic partnerships,
and grant fundraising.Since 2001,
Machine Science has worked with
more than 25 schools and more than
100 teachers,serving a total of more
than 2,000 learners.Mr.Rudnicki
is part of the management team for
two National Science Foundation-
funded education research projects,
Building an Internet Community of
Design Engineers (iCODE) and the
Internet Systemfor Networked Sen-
280 Auton Robot (2011) 30:265–280
sor Experimentation (iSENSE),both collaborations with the University
of Massachusetts Lowell.Prior to joining Machine Science,Mr.Rud-
nicki held editorial and project management positions at two Boston-
area consulting firms.
Rucha Londhe Ph.D.,works as a
Research Associate at the Goodman
research Group (GRG),Inc.,a Cam-
bridge,MA based firm that special-
izes in evaluating programs,ma-
terials,and services.At GRG,Dr.
Londhe manages a variety of eval-
uation projects across a number of
content areas.Dr.Londhe received
her doctorate from the School of
Family Studies,University of Con-
necticut in Human Development
and Family Studies,with a special-
ization in child development.She
has a graduate degree in Psychol-
ogy from Bombay University,India,where she worked as a clinician
before moving to the U.S.She has also served as a visiting lecturer,
teaching undergraduate classes in Human Development and Psychol-
ogy,at the University of Connecticut,Storrs and the FraminghamState
Colleen Manning is Director of Re-
search at Goodman Research Group,
Inc.Her areas of expertise are eval-
uation of informal education pro-
grams and research on early care
and education.She received her Child Study from Tufts
University in 1994 and her
Psychology from Mount Holyoke
College in 1989.She is currently
completing her Public Pol-
icy at the University of Massa-
Irene F.Goodman the
founder and president of Goodman
Research Group,Inc.(GRG),a re-
search firm specializing in the eval-
uation of educational programs,ma-
terials,and services that has been
operating since 1989.Aside from
overseeing 40+ research evaluation
projects per year,Dr.Goodman pro-
vides consultation and gives lec-
tures and workshops on evaluation.
Currently,her particular areas of in-
terest are informal education oppor-
tunities for all ages,and the inter-
section between formal and infor-
mal science education for youth.Prior to founding the company,she
was an evaluation consultant to various local and national organiza-
tions,taught courses at Dartmouth College and Tufts University as
Visiting Lecturer,developed instructional materials,and carried out
regional training sessions on public policy issues.She also had stints
as senior research associate at other research institutes.In the mid
1970s,she was on the faculty at the University of Wisconsin-Madison
and was the statewide specialist in child development for University
of Wisconsin-Extension.She holds a BA in Psychology from UCLA
(1973),MA in Child Development from Washington State University
(1975) and a doctorate in Human Development and Psychology from
Harvard University (1984).