Use of Thermodynamics, Fluid Mechanics and Heat Transfer Concept Inventories to Assess Student Learning in Core Courses

sisterpurpleMechanics

Oct 24, 2013 (3 years and 5 months ago)

49 views

Use of Thermodynamics, Fluid Mechanics and Heat Transfer

Concept Inventories to Assess Student Learning in Core Courses


John W. Mitchell and Jay K. Martin

University of Wisconsin
-
Madison


The thermal
-
science concept inventories are assessment instruments

designed to evaluate
student understanding of concepts in thermodynamics, fluid mechanics, and heat transfer,
specifically for mechanical engineering undergraduate students. We have been involved
in the development of all three concept inventories, and ha
ve been the lead authors for the
fluid mechanics and heat transfer CI’s (along with Anthony Jacobi and Ty Newell of the
University of Illinois Urbana
-
Champaign, and Frank Pffeferkorn from our department.)


Each of the concept inventories have been through
a development cycle to improve the
inventory. At this point the primary feedback we are using to advance the inventories is
reliability analyses. This has resulted in significant revisions to the concept inventories.
The inventories get better with each re
vision, however, the more time we spend in
review, the more we find that can be improved.


Parallel with the development, the concept inventories have been used for assessment of
all of the mechanical engineering students in the three subjects in the Mech
anical
Engineering department at UW
-
Madison. Beginning in the Fall 2005, and again in the
Spring 2006, concept inventories were administered in all sections of thermodynamics,
heat transfer and fluid mechanics.


The concept inventories were administered in

a pre and post
-
course assessment method.
The purpose of this type of assessment is to observe how much the students learned of
specific concepts in each of the courses as a result of taking the course. With these
assessments, we gained additional experien
ce and data about how the concept inventory
results would be used by faculty and teaching staff, and generated specific questions
about course content and teaching methodologies that could be investigated.


Shown in Table 1 below is an example of the resul
ts from the Fall 2005 and Spring 2006.
For the Thermodynamics Concept Inventory, there are 32 questions, and both the Fluid
Mechanics Concept Inventory and the Heat Transfer Concept Inventory have 30
questions. As shown, typical of the experience of others

with this type of assessment,
generally the scores are not high before or after the course, and the gain runs somewhere
between 5 and 30 percent. The post
-
score range tends to be from 3
-
4 to 1 showing
significant differences in understanding of the studen
ts at the conclusion of a course. Not
shown in the table is that in all classes, some students (approximately 10 percent) have
negative gains. There are a number of issues that might cause this besides student
understanding. For example, some subset of stu
dents may not take the concept inventory
seriously.


Table 1
-

Summary Scores for Fall 2005 and Spring 2006

Course

Average
Pre
-
Course
Score

Average
Post
-
Course
Score

Gain
-

#
of
questions


Post
-
Course
Range

Gain (%)

Gain (%)=
(Post


Pre)/(#
Questions


Pre
)


Thermodynamics






Fall 2005

16.4

19.4

3.0

7
-

28

19

Spring 2006

16.3

19.2

2.9

5
-
24

18

Fluids






Fall 2005 Sect. 1

11.3

17.1

5.7

6
-

28

30

Fall 2005 Sect. 2

12.5

16.9

4.5

6
-

28

27

Sp 2006 Sect. 1

11.6

14.5

2.8

5


22

18

Sp 2006 Sect. 2

11.
6

15.3

3.6

4
-
21

25

Heat Transfer






Fall 2005 Sect. 1

14.4

17.2

2.9

6
-

29

19

Fall 2005 Sect. 2

15.4

20.0

4.5

8
-

26

30

Sp 2006 Sect. 1

13.4

14.3

0.9

3
-

23

5

Sp 2006 Sect. 2

13.9

18.4

4.5

7
-

28

28


It is instructive to look at the concept invent
ory results by topic and subject. As shown in
Table 2, scores on various thermodynamics topics or concepts have been examined. A
number of observations can be made:



There are definite differences in the gains for different topics, with a low of 5 % gain
and a high of 31 % gain.


Table 2
-

Results from Thermodynamics CI by Topic

Topic


Pre

Post

Gain (%)

Properties

Fall 2005

52

59

14


Spring 2006

51

57

11






Conservation of Mass

Fall 2005

65

78

37


Spring 2006

70

76

19






Conservation of Energy

F
all 2005

49

55

12


Spring 2006

49

51

4






Second Law

Fall 2005

31

51

28


Spring 2006

33

45

18






Work

Fall 2005

77

83

25


Spring 2006

72

84

42



The gain for Conservation of Mass is very low. For one question (16) there was a
decrease in score f
rom 82 % to 63 %



The gain for Properties is surprisingly low considering how much time is spent on
this subject. For three questions there were negative gains.



The gains for the Second Law are quite high. All of the questions showed positive
gains.



The s
cores for Work are high; apparently students know this topic pretty well.



Even though the questions on this Concept Inventory may not be reliable, the results
do demonstrate that the students have trouble conceptually with property ideas and
conservation o
f energy.

Table 3
-

Results from Fluids CI by Topic and Section


Topic

Section

Pre

Post

Gain* (%)


Basic Ideas

Fall 1

46

65

35


Fall 2

59

66

17


Spring 1

52

62

11


Spring 2

52

70

19






Hydrodynamics

Fall 1

48

67

35


Fall 2

46

54

15


Spring 1

46

47

0


Spring 2

46

56

10






Conservation of Mass

Fall 1

49

77

56


Fall 2

41

60

32


Spring 1

52

65

13


Spring 2

52

58

6






Cons Momentum


Ideal

Fall 1

35

58

35


Fall 2

35

59

37


Spring 1

25

42

6


Spring 2

35

38

3






Cons Momentum


Viscous

Fall 1

31

46

22


Fall 2

39

52

21


Spring 1

34

38

4


Spring 2

34

40

6






Conservation of Energy

Fall 1

22

41

24


Fall 2

19

44

32


Spring 1

15

15

0


Spring 2

15

22

8

Similar results are shown in Table 3 for Fluid Mechanics. From these results, t
he
following is apparent:



The average of all post course scores are highest for Basic Ideas, Hydrodynamics,
Conservation of Mass, and Conservation of Momentum


Ideal (about 60) and lowest
for Conservation of Momentum

Viscous and Conservation of Energy (
about 45).



There are definite differences in the gains for different topics, with a low of 0 % gain
and a high of 56 % gain.




The greatest gains are in Basic Concepts, Conservation of Mass, Conservation of
Momentum


Ideal



The lowest gains are Conservation

of Momentum


Viscous, Hydrodynamics, and
Conservation of Energy



There are significant differences in gains by section.

Table 4
-

Scores from the Heat Transfer CI by Topic and Class


T
he scores from Heat Transfer are shown in Table 4. In this case:



The average of all post course scores are highest for Energy Balance and lowest for
Radiation



Greatest gains are in Basic Ideas, Transient Conduction, and Convection

Topic

Section

Pre

Post

Gain* (%)


Energy Balance/Basic Ideas

Fall 05 1

5
3

72

40


Fall 05 2

65

84

55


Spring 06 1

47

53

10


Spring 06 2

47

64

31






Steady State Conduction

Fall 05 1

51

55

9


Fall 05 2

52

62

20


Spring 06 1

48

46

-
3


Spring 06 2

48

63

30






Transient Conduction

Fall 05 1

46

66

36


Fall 05 2

54

75

45


Spring 06 1

60

63

7


Spring 06 2

60

75

38






Convection

Fall 05 1

48

63

30


Fall 05 2

56

76

46


Spring 06 1

38

41

5


Spring 06 2

38

55

27






Radiation

Fall 05 1

36

34

3


Fall 05 2

28

43

21


Spring 06 1

42

48

10


Spring 06 2

42

56

24



Lowest gains are in stea
dy state conduction and radiation



Steady State Conduction has a surprisingly low post score and low gain even though
a lot of time is spent on the subject.



There is a negative gain for one class for radiation



There are again significant differences between

the gains for different sections.


As an additional means of studying the assessment results, individual student scores for
each of the subjects have been plotted in Figs. 1, 2, and 3. As illustrated, two results are
apparent. First, in general, for stu
dents that show improvement, there seems to be only a
small correlation between pre
-
course score and post
-
course score. Second, a significant
fraction of the students show no gain or negative gain.


Figure 1


Pre
-
Course Score vs. Post
-
Course Score for Th
ermodynamics


Fall 2005
and Spring 2006



Figure 2
-

Pre
-
Course Score vs. Post
-
Course Score for Fluid Mechanics


Fall 2005


Figure 3


Pre
-
Course Score vs. Post
-
Course Score


Heat Transfer


Fall 2005


As a means of assessing the relationship between
the grade received in the course and the
student results from the concept inventory at the completion of the course, Figs. 4, 5 and
6 illustrate this comparison for all three subjects. There appears to be a weak correlation
in all three subjects, with high
er grades being correlated with higher scores on the
inventory. The strongest correlation is in Thermodynamics, and the weakest in Heat
Transfer.


Figure 4
-

Post
-
Course Score vs. Final Grade


Thermodynamics


Fall 2005



Figure 5
-

Post
-
Course Score v
s. Final Grade
-

Fluid Mechanics


Fall 2005


Figure 6
-

Post
-
Course Score vs. Final Grade
-

Heat Transfer
-

Fall 2005


Use of the Concept Inventory Results


Following the development of the concept inventories and the collection of results from
students,

the faculty and teaching staff have been studying the results. At this stage in the
deliberation, the results have raised many questions. Some of the questions have focused
on the concept inventories themselves, and there remain questions as to how reliab
le or
valid these measurements are. For example, there has been a persistent question as to
whether or not the students would try to do their best on the inventories, given that they
were not being graded on how well they did (in general, they were given h
omework
credit for turning in an answer sheet.) To address this question, when the inventories were
administered at the end of the Spring semester, 2006, two additional questions were
added to the inventories.


The first question, shown below, was designed

to get a sense of how much time was
required for the inventories. The intent of the second question was to gage how the
students perceived the contents of the concept inventory and their understanding of the
subject.


Question 1: How much time did it take

you to complete the Concept Inventory?

1.

Less than 1/2 hour

2.

Between 1/2 and 1 hour


3.

Between 1 and 1 1/2 hours

4.

Over 1 and 1 1/2 hours


Question 2: How well do you think that your score on this Concept Inventory
reflects your understanding of Heat
Transfer?

1.

Not at all
-

I did not have time to give this an adequate effort

2.

Somewhat
-

I tried but was not really focused on providing the
best answers

3.

Reasonably accurate
-

I might have done better with more effort

4.

Quite well
-

I did my best to answer all
of the questions.


The relation between student score on the Post
-
Course Assessment and the Time and
Understanding responses are shown in Figs. 7 and 8. The students tended to take
somewhere between 30 minutes and 1.5 hours to complete the inventories. And
, in
general, they thought that the inventory score reflected with some accuracy their
understanding of the subject.


Figure 7
-

Time to Complete Inventory vs. Post
-
Course Score

Figure 8
-

Post
-
Course Score vs. Student Reported Reflection of Understand
ing


At this point, the faculty and teaching staff are continuing to refine the inventories to
improve reliability, the method of actually doing the assessment, and how to
appropriately use the results in improving course content and pedagogy. It is cle
ar that
this is a challenge, however, there is also agreement that this type of assessment holds
tremendous promise for informing the process of course improvement.