Face Recognitionbased Lecture Attendance System
Yohei KAWAGUCHI
y
Tetsuo SHOJI
yy
Weijane LIN
y
Koh KAKUSHO
yy
Michihiko MINOH
yy
y
Department of Intelligence Science and Technology,Graduate School of Informatics,Kyoto University
yy
Academic Center for Computing and Media Studies,Kyoto University
Abstract
In this paper,we propose a system that takes the attendance of students for classroom lecture.Our system takes the
attendance automatically using face recognition.However,it is diﬃcult to estimate the attendance precisely using
each result of face recognition independently because the face detection rate is not suﬃciently high.In this paper,
we propose a method for estimating the attendance precisely using all the results of face recognition obtained by
continuous observation.Continuous observation improves the performance for the estimation of the attendance We
constructed the lecture attendance systembased on face recognition,and applied the systemto classroomlecture.This
paper ﬁrst review the related works in the ﬁeld of attendance management and face recognition.Then,it introduces
our system structure and plan.Finally,experiments are implemented to provide as evidence to support our plan.The
result shows that continuous observation improved the performance for the estimation of the attendance.
1 Introduction
Though the video streaming service of lecture archive
is readily available in many systems,students have few
opportunities to view the lecture in this service because
lecture content is not summarized.If the attendance of
a student of classroom lecture is attached to the video
streaming service,it is possible to present the video of
the time when he was absent.It is important to take the
attendance of the students in the classroom automati
cally.
ID tag or other identiﬁcations such the record of log
in/out in most eLearning systems are not suﬃcient be
cause it does not represent students’ context in faceto
face classroom.It is also diﬃcult to grasp the contexts
by the data of a single moment.
student’s context such as presence,seat position,sta
tus,and comprehension are discussed in this paper.At
the same time face images reﬂect a lot about these con
text information.It is possible to estimate automatically
whether each student is present or absent and where each
student is sitting by using face recognition technology.It
is also possible to know whether students are awake or
sleeping and whether students are interested or bored in
lecture if face images are annotated with the students’
name,the time and the place.We are concerned with
the method to use face image processing technology.
By continuously observing of face information,our ap
proach can solve low eﬀectiveness of existing face detec
tion technology,and improve the accuracy of face recog
nition.
We propose a method that take the attendance using
face recognition based on continuous observation.In this
paper,our purpose is to obtain the attendance,positions
and images of students’ face,which are useful informa
tion in the classroom lecture.
2 Related work
Cheng,et al.[1] developed the system to manage the
context of the students for the classroomlecture by using
note PCs for all the students.Because this system uses
the note PC of each student,the attendance and the
position of the students are obtained.However,it is
diﬃcult to know the detailed situation of the lecture.
our system takes images of faces.
In recent decade,a number of algorithms for face
recognition have been proposed [2],but most of these
works deal with only single image of a face at a time.By
continuously observing of face information,our approach
can solve the problem of the face detection,and improve
the accuracy of face recognition.
1
Figure 1:Architecture of the system
3 Lecture attendance system
3.1 Architecture
In this paper,our system consists of two kinds of cam
eras.One is the sensing camera on the ceiling to obtain
the seats where the students are sitting.The other is the
capturing camera in front of the seats to capture images
of student’s face.The procedure of our system consists
the following steps (see Figure 1):
1.Seats information processing:this process deter
mines the target seat to direct the camera.We
adopt the approach called Active Student Detect
ing method (ASD) [3].The idea of this approach
is to estimate the existence of a student sitting on
the seat by using the background subtraction and
interframe subtraction of the image from the sens
ing camera on the ceiling.
2.Shooting plan:our system selects one seat from the
estimated sitting area obtained by ASD,directs the
camera to the seat and captures images.
3.The system processes the face images.the face im
ages are detected fromthe captured image,archived
and recognized.Face detection data and face recog
nition data are recorded into the database.
4.Attendance information processing:this process
estimates the attendance by interpreting the face
recognition data obtained by continuous observa
tion.The module obtains the most likely correspon
dence between the students and the seats under the
constrained condition.The system regards a stu
dent corresponded to each seat as present.The po
sition and attendance of the student are recorded
into the database.
The procedure is repeated during lecture,and estimated
the attendance of the students in real time.
3.2 Estimating students’ existence
We use the method of ASD to estimate the existence of
a student sitting on the seat.It is described in detail in
[3].In this approach,an observation camera with ﬁsh
eye lens is installed on the ceiling of the classroom and
looks down at the student area vertically.ASD estimates
students’ existence by using the background subtraction
and interframe subtraction of the images captured by
the sensing camera (see Figure 2).In the background
subtraction method,noise factors like bags and coats of
the students are also detected,and the students are not
detected if the color of clothes of them are similar the
seats.ASD makes use of the interframe subtraction to
detect the moving of the students.
2
Figure 2:Active Student Decting method
3.3 Shooting plan
Camera planning module selects one seat from the esti
mated sitting area in order to determine where to direct
the front camera.Actually,in this paper,the module se
lects a seat by scanning the seats sequentially.This ap
proach is insuﬃcient because it wastes time directing the
camera to where the studentandseat the seats the stu
dents correspondence is already decided In other words,
if we direct the camera to each seat with the same prob
ability,it is diﬃcult to detect the faces according to the
student or the seat,and the system judges the students
who are actually present to be absent consequently.In
order to solve this problem,it is important to the infor
mation of each student’s position.
The camera is directed to the selected seat using the
pan/tilt/zoomthat have been registered in the database.
The camera captures the image of the student.
3.4 Face detection and recognition
Face detection and recognition module detects faces from
the image captured by the camera,and the image of
the face is cropped and stored.The module recognizes
the images of student’s face,which have been regis
tered manually with their names and ID codes in the
database.Face detection data and face recognition data
are recorded into the database.
3.5 Estimating the seat of each student
In order to solve the problem of ineﬀectiveness,we inte
grated students’ seat information into the camera plan
Figure 3:The face of the student on the back seat is
detected.
ning.In this way,we can solve the problem such as
misrecognition of faces and seats by constraints of the
correspondence relationship between them.
The face detected from the captured image may be
another neighbor student’s face (see Figure 3).There
fore,it is necessary to consider the possibility that the
face image is the one of a neighbor student even if the
camera is directed to the target seat.
Considering the points we mentioned above,we pro
pose the following method.We assume that every seat
has a vector of values that represent the relationship be
tween the seat and each student.In the case that the
module of face image processing recognizes Student A’s
face from the image of Seat B,our module votes for Stu
dent A’s component of the vectors of the seats in the
neighborhood of Seat B.
We assume the voting weights in Figure 4.Each cell
means a seat,and the gray center cell means the focused
seat.This assumption means that,when Student A is
recognized at Seat B,0:24 is voted to Seat B,and 0:11 is
voted to the front seat of Seat B,and so on,for Student
A’s components.For example,Figure 5 shows Student
A’s components of each seat when Student A is recog
nized at the gray seat,and Figure 6 shows the case that
Student A is recognized at the gray seat in the next step.
Considering the bipartite graph of the students and
the seats,voting can be thought of as the addition to
the scores of the edges between the students and the
seats,and the cost of the edge is deﬁned as the inverse
of the score of the edge.
Before the seat information processing,we set two con
ditions as the premises:
² more than two students are not sitting on the same
seat,
² the students do not move to diﬀerent seats fre
quently.
The process of the seats information do not select inde
pendently the seat that has the highest score for each
3
Figure 4:An example of the voting weights
Figure 5:1) Student A’s component of each seat when
Student A is recognized at the gray seat
Figure 6:2) Student A’s component of each seat when
Student A is recognized at the gray seat after 1)
Figure 7:An example of 2 students and 2 seats
student but use the approach that ﬁnd the matching in
the bipartite graph such that the sum of the costs of
the edges are minimized where the premises are satis
ﬁed.Figure 7 shows an example of the bipartite graph
in the case that two students and two seats exist.In this
case,our approach obtains the two thick arrows as the
correspondence.Our process solves Linear sum assign
ment problem (LSAP) to estimate the correspondence.
We assume the assignment of student i to seat j incurs a
cost c
ij
.The problem is formulated as follows:
min
n
X
i=1
c
ij
x
ij
n
X
i=1
x
ij
= 1 j = 1;¢ ¢ ¢;n
n
X
j=1
x
ij
= 1 i = 1;¢ ¢ ¢;n
x
ij
2 f0;1g i;j = 1;¢ ¢ ¢;n (1)
The least complexity of the best sequential algorithms
for the LSAP is O(n
3
),where n is the larger one of the
numbers of the students or the seats[4].Thus,this prob
lem is solved in real time.
In this procedure,the systemregards the students cor
responded to the seats as present.
4 Experiment
4.1 Result of Estimating the seat of each
student
19 students existed in the center area,and we ran the
process of camera control and detection for 20 minutes.
We labeled the images of the detected faces with the
name of the students manually.The system detected
faces 186 times,and 15 students were detected.Table
1 shows the accuracy of seat estimation.We have com
pared the result of estimating the seat of each student
4
by using the method described in section 3.5.Method 1
is the method that corresponds each student to the seat
where the most faces of the student are detected.Method
2 is the method that corresponds each student to the seat
that has the lowest cost of the student.Method 3 is the
method of section 3.5.Denominator of fractions in this
table is the number of the facedetected students.This
table shows that accuracy are improved by the method
of section 3.5.
4.2 Result of Estimating the attendance
based on continuous observation
We compared the results one cycle only and continu
ous observation.12 students existed in the center area,
and 2 of them did not have their faces registered.In
this experiment of 79 minutes,8 scanning cycles were
completed during this period.Table 2 shows face detec
tion rate,and Table 3 shows the result of estimating the
attendance.In the case of 1 cycle only,we judge the
recognized students to be present.In the case of contin
uous observation,the system estimates the attendance
by the method of section 3.5 using the recognition data
obtained during 79 minutes.This table shows that con
tinuous observation improved the face detection rate and
improved Fscore of estimation of the attendance,which
is the harmonic mean of precision and recall.
5 Conclusion and future direc
tions
In this paper,in order to obtain the attendance,positions
and face images in classroomlecture,we proposed the at
tendance management system based on face recognition
in the classroom lecture.The system estimates the at
tendance and the position of each student by continuous
observation and recording.The result of our preliminary
experiment shows continuous observation improved the
performance for estimation of the attendance.
Current work is focused on the method to obtain the
diﬀerent weights of each focused seat (in section 3.5) ac
cording to its location.We also need to discuss the ap
proach of camera planning based on the result of the
Table 1:Result of estimating the seat of each student
Method
Accuracy
Method 1
60.0% (9/15)
Method 2
73.3% (11/15)
Method 3
80.0% (12/15)
Table 2:Face detection rate
Time
face detection rate
1 cycle only
37.5% (3.8/10)
79 min
80.0% (8/10)
Table 3:Result of estimating the attendance
Time
precision
recall
Fscore
1 cycle only
89.2%
33.8%
48.3%
79 min
70.0%
70.0%
70.0%
position estimation in order to improve face detection
eﬀectiveness.In further work,we intend to improve face
detection eﬀectiveness by using the interaction among
our system,the students and the teacher.
On the other hand,our system can be improved by in
tegrating videostreaming service and lecture archiving
system,to provide more profound applications in the
ﬁeld of distance education,course management system
(CMS) and support for faculty development (FD).
Acknowledgements
The authors would like to thank Omron Corporation for
their help to providing OKAO vision library used in face
detection and recognition in our system.
References
[1] K.Cheng,L.Xiang,T.Hirota and K.Ushijima,
“Eﬀective Teaching for Large Classes with Rental
PCs by Web System WTS,” in Proc.Data Engineer
ing Workshop 2005 (DEWS2005),2005,1Dd3 (in
Japanese).
[2] W.Zhao,R.Chellappa,P.J.Phillips,and A.Rosen
feld,“Face recognition:A literature survey,” ACM
Computing Surveys,2003,vol.35,no.4,pp.399458.
[3] S.Nishiguchi,K.Higashi,Y.Kameda and M.Minoh,
“A Sensorfusion Method of Detecting A Speaking
Student,” IEEE International Conference on Multi
media and Expo (ICME2003),2003,vol.2,pp.677
680.
[4] R.E.Burkard and E.C¸ela,“Linear Assignment Prob
lems and Extensions”,In Handbook of Combinato
rial Optimization,Du Z,Pardalos P (eds).Kluwer
Academic Publishers:Dordreck,1999,pp.75149.
5
Enter the password to open this PDF file:
File name:

File size:

Title:

Author:

Subject:

Keywords:

Creation Date:

Modification Date:

Creator:

PDF Producer:

PDF Version:

Page Count:

Preparing document for printing…
0%
Comments 0
Log in to post a comment