SEMANTIC VIDEO ANNOTATION IN E-LEARNING FRAMEWORK

drillchinchillaInternet and Web Development

Oct 21, 2013 (3 years and 5 months ago)

138 views

SEMANTIC VIDEO ANNOT
ATION

IN E
-
LEARNING FRAMEWORK

Antonella Carbonaro, Rodolfo Ferrini

Department of Computer Science

University of Bologna

Mura Anteo Zamboni 7, I
-
40127 Bologna, Italy

Tel.: +39 0547 338830 Fax: +39 0547 338990

e_mail:
carbonar@csr.unibo.
it, ferrini@csr.unibo.it

Abstract.

The aim of this paper is twofold. In the first part we will
consider the new technologies proposed as solution for the Semantic
Web and then we try to outline possible applications in different fields
of e
-
learning givin
g example of actual works. In the second part of
the paper our proposal of improvement of a
n

e
-
learning
with a
semantic video annotation module

is presented.


1.

Introduction

Nowadays
Web is maybe the
larger available repository of resources.

But when
one

ha
s

to look for within an enormous set of information,
the research process
i.e. learning material,
may look as too expensive without the help of machines.

The problem is that the Web was designed (and currently is) for human usage
and

its
resources are not m
achine
-
understandable.
It’s necessary adding some
machine understandable meta
-
data to resources in order to provide and build
services, agents or other kind of applications that help students in their tasks
.


In last

years we have had an improvement in th
e kind of resources used as
learning material. A tangible example of such trends are video resources.

Automatic systems for video segmentation and annotation are the requirements
of all digital video management systems. The goal is to find automatic and
ge
neral procedures to segment videos into scenes and to annotate them with
textual data or with metric information. Annotations could be useful for further
indexing, retrieval, recommendation and so on, performed both by human users
and by automated applicat
ions.

The Semantic Web
[Berners
-
Lee
et al
.
, 2001]

is
the propose
d solution in
semantic resource annotation perspective.


The Semantic Web can be defined as an extension of the
current

Web in which
meaning
is added to resources
so that machines are allowed
to understand
them better.

This new architecture is based on the annotation of web
documents with additional semantic data.
In these last years a number of new
languages have been proposed in order to carry out this task.

In [Bighini and
Carbonaro, 2004] i
s

introduced the InLinx (Intelligent Links)
system, a Web application that provides an on
-
line bookmarking service. The
overall system
it has been firstly improved

in [Carbonaro and
Ferrini, 2005],

introducing concepts for classification, recommendation an
d document sharing
to provide a better personalized semantic
-
based resource management.

Most
recently we have introduced the vid
eo annotation module Scout
-
v [
Carbonaro
et
al.
, 2006]. Scout
-
V performs automatic shot detection and supports user during
annota
tion phase in a collaborative framework proposing suggestion on the
basis of the effective user needs and modifiable user behaviour and interests
.

The paper is organized as follows. Section 2 provides an overview of semantic
web technology. Some examples o
f current research projects
are
given in
section 3. In section 4
an inLinx architectural overview is described
while a
description of
Scout
-
V module

is given in section 5.

2.

Semantic

web
technologies

In the above section we have
introduced the development o
f

languages for
resource annotation. But In the semantic web architecture there is another
important layer to take into consideration.



Figure
1
.

Semantic Web Layers


XML allows the definition of personalized tags while RDF all
ows
the metadata
realization
. But if you want to
univocally
describe a resource
,

you need to refer
your RDF statement to an ontology.
Such a term
is defined as a shared
specification of a conceptualization [
Gruber, 1993
]

and can be used as a
knowledge rep
resentation of a particular domain

by a
llowing applications to
communicate
e
ach other and to “understand” the meaning of the annotated
resources.

In the rest of this chapter we

will
briefly describe some of the main technologies
developed for the Semantic
Web. Most of them
have been

proposed as
standard by the WWW Consortium, the
leftover ones

are alternative solutions.


-

XML

[XML, 2004
]: is a well known Mark Up Language recommended by
W3C. This is a technology that allows to go deeper

into

the page struc
ture
provided by HTML.


-

RDF

[RDF, 2004
]: is the W3C recommendation for the creation of metadata
about resources. With RDF one can make statement
s

about a resource in
the form of

a

subject
-
predicate
-
object expression. The described resource is
the subject
of the statement, the predicate is a specified relation that links
the subject and the object that is the value assigned to the subject through
the predicate.


-

OWL

[OWL, 2004
]: is the W3C recommendation for the creation of new
ontology optimized for the
web. The Web Ontology Language OWL is a
semantic markup language for publishing and sharing ontologies on the
World Wide Web. OWL is developed as a vocabulary extension of RDF and
it
is derived from the DAML+OIL Web Ontology Language. For these
reasons
it
provides
a
greater machine interpretability of Web content than
that

one

supported by its predecessors. Essentially
,

with OWL one can
describe a specific domain in terms of class, properties and individuals
.

It
has three increasingly
-
expressive sublanguage
s: OWL Lite, OWL DL and
OWL Full.


-

UML

[UML, 1999]
: is a specification language originally designed
for
software engineering. UML provides a set of graphical notation
s

to describe
an abstract model of a system.
A lot of efforts have be recently done in or
der
to map UML notation with Semantic Web technology like
OWL.


-

RuleM
L
*
:
is a language defined by the Rule Markup Initiative for the
definition of XML rules.

The main objective of
this

project is to provide a
basis for an integrated rule
-
markup approach th
at will be useful for many
interconnected purposes.


3.

Semantic

web and e
-
learning

How can semantic web improve e
-
learning systems? This new web perspective
promises more reusability of resources as well as much more flexibility in
system
s

architecture and
in interoperability between them.
The research on e
-
learning and Web
-
based educational systems traditionally combines research
interests and efforts from various fields, in order to tailor the growing amount of
information

to the needs, goals, and tasks of

the specific individual users.
Semantic Web technologies may achieve improved adaptation and flexibility

of

e
-
learning systems
and new methods and types of courseware
which will be
compliant with the Semantic Web vision.

In
the following

sections we
will
describe some examples of existing projects thanks to
which
we
will be able to

outline what the current research on these fields offers.

3.1.


IMS Learning Design

IMS LD


楳i def楮id a猠a 污lguage 景爠mode汬楮i un楴猠o映獴sdy⸠
䥴猠
sée捩f楣慴楯n
污lguage 楳i found
ed on 瑨
e bdu捡瑩ona氠Mode氠ianguage


孋oée爬rOMMN
崮 䥍p
i䐠ex瑥nd猠bMi
by
捯ve物ng no琠on汹 瑨e
汥l牮楮g é牯捥獳smode汬ing
bu琠a汳l
楮ieg牡瑩rg 瑨e a獰e捴猠of o瑨e爠獰e捩f楣慴楯n猠
汩步
䥍匠捯n瑥n琠éa捫cg楮i and
䥍匠i位lme瑡data⸠周e 獴s牴rng éo楮i 楮i䥍匠
i䐠deve汯lmen琠
楳i
瑨e 獥éa牡瑩on
be瑷een a牮楮g⁡捴楶楴楥猠and⁴he楲 a牮楮g⁲e獯u牣敳r

LD doesn’t
completely
capture the aim of the semantic web but the conceptual
model expressed in UML and the constructs specification expressed in XML



*

http://www.ruleml.org/



h瑴t㨯W睷w.imsg汯b慬aorg⽬敡/n楮gd敳egn⽩/dex.捦m



h
瑴t㨯⽥m氮lu.n氯ln瑲odu捴楯n/do捳⽰敤
-
m整emod敬epdf.

can provide a s
emantic notation that e
-
learning systems could extend with their
preferred semantic web technologies.

3.2.

Edutella

Edutella
§

is defined as a multi
-
staged effort to scope, specify, architect and
implement an RDF
-
based metadata infrastructure for P2P
-
networks fo
r
exchanging information about learning objects. Edutella P2P architecture is
essentially
based on JXTA and RDF. JXTA
**

is an open source technology that
provides a set of XML based protocols
supporting

different kind of P2P
applications.

According
to
[Nej
dl
et al.
, 2002
]
three types of services, which a peer can offers,
are defined in an Edutella network:

-

Edutella Query Service: the basic service in the framework.
It p
rovide a
common, RDF
-
based query interface (the Query Exchange Language

RDF
-
QEL) for m
etadata providing and consuming through the Edutella
network;

-

Edutella Replication:
it
provide replication of data
to

additional peers to
ensure data persistence;

-

Edutella Mapping, Mediation, Clustering: this kind of services manages
metadata allowing sema
ntic functionality of the global infrastructure;

An important point to underline is that Edutella doesn’t share resource content
but only metadata.

3.3.

Smart Space for Learning

Smart Spaces for Learning is the result of the Elena project
††

wo牫⁛r業on
et al.
,
2003a
] and according
to
[Simon
et al.
, 2003b]

a Smart Space for Learning can
be defined as
a set of service mediators

which support the personalized
consumption of heterogeneous educational services provided by different
management systems. But what is a l
earning service? Learning services are
entities designed to satisfy a specific purpose (e.g. the delivery of a course).
They may use resources as learning objects (e.g
. exercises and exams) and
Web s
ervices
‡‡

瑯⁩ te牦r捥
瑨e forme牳

w楴i a牮r牳r

The 獹獴
em a牣桩瑥捴u牥rof pma牴riea牮楮g péa捥猠楳i
e獳sn瑩t汬y
捯méo獥d by
瑨楳⁴io⁢u楬d楮i⁢汯捫lW

J

an⁅dute汬ae瑷o牫r

J

a⁳ 琠of
onto汯l楥猻

䥮 a pma牴rpéa捥 fo爠iea牮楮g
I

é牯r楤i牳rof 汥l牮楮g 獥牶楣敳 a牥rconne捴cd to
a 汥l牮楮g managemen琠獹獴sm wh楣栠楳i bas
ed on bdu瑥汬a⸠ln瑯log楥猠ha
s

to
de獣物be 瑨e 汥l牮楮g doma楮猠 u獩湧 捯nceé瑳t and 牥污瑩tn猠 瑨a琠 may be
牥晥牲ed



楮 瑨e anno瑡瑩tns o映瑨e 汥lrn楮i 獥牶楣敳i A b物ef de獣s楰i楯i of an
on瑯logy nguage
瑨e teb⁏n瑯汯ly⁌anguage⁏ti⤠F猠s楶en⁩ ⁳ 捴楯
n′⸠

3.4.

HyCo




§

http://edutella.jxta.org/

**

http://www.jxta.org/

††

h瑴t㨯W睷w.敬ena
-
proj散琮trg/

‡‡

W3C’s
overview

on Web
services

at

http://www.w3.org/2002/ws/

HyCo [García
et al.
, 2004
] stands for Hypermedia Composer and

it

is a
multiplatform tool that supports the creation of learning materials. HyCo is the
result of the development of an authoring tool created with the intent

in order

to
define ALDs
.

According to [Berlanga and García, 2005
] ALDs
there
are

some

unit
s

of
learning that contains personalized behaviour in order to provide each student
with a learning flow

which is to be

adequate to her/him characteristics. ALDs are
semantically structure
d according to IMS LD (described above) in order to allow
reusability.

The last version of HyCo
also
manage a kind of resources named SLOs. A SLO
is a Learning Object
compliant with IMS Metadata
§§
. Every resource created
with HyCO is turned into a SLOs. Wh
en
ever

the conversion process is finished
an XML file is generated for the new SLO and stored in a repository.

4.

inLinx

In this section we

are going to

briefly introduce the InLinx (Intelligent Links)
system
[Bighini and
Carbonaro, 2004]
, a Web application
p
roviding

an on
-
line
bookmarking service. InLinx is the result
of three filtering components
integration

corresponding to the following functionalities:

-

bookmark classification (content
-
based filtering);

-

bookmark sharing (collaborative filtering);

-

paper rec
ommendation (content
-
based recommendation)
;

Over the years we have introduced several extensions
to

the original
architecture.

Generally, recommender systems, like inLinx, uses keywords to represent both
users and resources. Another way to handle such data

is using hierarchical
concept categories. This issue will enable users and the system to search,
handle or read only those concepts of interest in a more general manner. For
example, synonymy and hyponymy can reveal hidden similarities, by potentially
lea
ding to better classify and recommend. The advantages of a concept
-
based
document and user representation are: (i) ambiguous terms inside a resource
are disambiguated, allowing their correct interpretation, consequently, a better
precision about the user m
odel construction (e.g., if a student is interested in
computer science resources, a document containing the word ‘bank’ in the
financial context will not be relevant); (ii) synonymous words belonging to the
same synset can contribute to the user model def
inition (for example, both
‘mouse’ and ‘display’ brings evidences for computer science documents,
improving the coverage of the document retrieval); (iii) finally, classification,
recommendation and sharing phases take advantage from the word senses in
ord
er to classify, retrieve and suggest documents with high semantic relevance
with respect to the user and resource models.


5.

The Scout
-
V Module

The
Scout
-
V
module assists authors in the task of annotating video sequences.
Multimedia annotation systems need s
tandard output, compliant with other tools



§§

http://www.imsglobal.org/metadata/index.cfm

for browsing or indexing. MPEG
-
7 [ISO/IEC, 2002] standard was defined to this
purpose. It represents an elaborate standard in which a number of fields
ranging from low level encoding scheme descriptors to high lev
el content
descriptors are merged, useful for describing a video and part of it.

Each shot belonging to the video sequence can be annotated on the base of
underling ontology. These descriptions are labeled for each shot and are stored
as MPEG
-
7 descriptio
ns in the output XML file.
Scout
-
V
can also save, open,
and retrieve MPEG
-
7 files in order to display the annotations for corresponding
video sequences.

The
Scout
-
V
main page shows all the videos that should be elaborated
performing shot detection, editing

or remo
ving, as illustrated in Figure 2
.



Figure
2

System video processing: shot detection phase is selected.


Given the segmentation of video content into video shots, the second step is to
define the semantic lexicon in whic
h to label the shots. A video shot can
fundamentally be described using five basic classis: agents, objects, places,
times and events. These five types of lexicon define the initial vocabulary for our
video content; they correspond to the
SemanticBase
MPEG
-
7 tags.

We have also defined attributes to describe class characteristics. Each attribute
corresponds to a

specified MPEG
-
7 tag in storing phase.

Using the defined vocabulary for static agents, key objects, places, times and
events, the lexicon is importe
d into Scout
-
V for describing and labeling each
video shot. The shots are labeled for its content with respect to the selected
lexicon. Note that the lexicon definitions are database and application specific,
and can be easily modified and imported into th
e annotation tool.

Scout
-
V
annotation tool is divided into three graphical sect
ions, as illustrated in
Figure 3
. On the upper left
-
hand corner of the tool is located the
Scene
Matching
frame in which are specified the algorithms that can be used to obtain
video annotation recommendations (Block Truncation Coding, edge histogram,
colour histogram). On the bottom left
-
hand portion of the tool is placed the
Ontology Visualization
frame providing interactivity to assist authors of the
annotation tool. On the ri
ght, is the
Video Presentation
frame with key frame
image display and frame characteristics. The
Ontology Editor
module
allows to
modify the ontology tree creating needed classes and instances. The two
figures show how to create ontology structure introduc
ing new classes and their
hierarchy and how to populate created class with a specific instances. The aim
of the instance creation phase is to effectively represent the domain knowledge,
achieving better precision in the annotation task.



Figure
3

Scout
-
V annotation tool.


Figure 4

shows annotation procedure apply on the first scene using checkboxes
and comments to better describe the selected video. Annotations are then
stored and used by recommendation procedure to help users f
inding similar
frames annotated also b
y different users
.

When all the scene have been
annotated the system produces the MPEG
-
7 file.



Figure
4

Frame annotation phase
.


6.

Conclusions

The paper addresses a key of the semantic web t
echnology

usage

in e
-
learning. We have explored some of the most important W3C’s standard as
XML, RDF, OWL as well as some alternative solution
s

proposed in other
research field
s

like

RuleML.

We have tried to address our main purpose

by

giving examples of

our

current
research project
aiming

to extrapolate which and
how

the semantic web
technologies are involved in their architecture.


Subsequently a brief

introduction of inLinx and of its
new semantic video
annotation module
has been

given. inLinx is an hyb
rid recommender system
that provides an on
-
line bookmarking service.
Scout
-
V
performs automatic shot
detection and supports user during annotation phase in a collaborative
framework proposing suggestion on the basis of the effective user needs and
modifia
ble user behaviour and interests
.

7.

References

[
Berlanga and García, 2005
]
A. J.
Berlanga,

F. J.

García
.

IMS LD reusable elements for
adaptive learning designs
”.

Journal of Interactive Media in Education

(Advances in Learning
Design. Special Issue, eds. Col
in

Tattersall, Rob Koper)
, November 2005
.

[Berners
-
Lee
et al
.
, 2001
]

T. Berners
-
Lee, J. Hendler, O. Lassila.


The Semantic Web
”.

Scientific American
, 2001.


[Bighini and
Carbonaro, 2004]
Bighini C., Carbonaro A., 2004,
InLinx: Intelligent Agents for
Person
alized Classification, Sharing and Recommendation
, Int. Journal of Computational
Intelligence. Int. Computational Intelligence Society.


[Carbonaro and Ferrini, 2005] A. Carbonaro, R. Ferrini. “Considering semantic abilities to
improve a Web
-
Based Distance

Learning System”.
ACM International Workshop on Combining
Intelligent and Adaptive Hypermedia Methods/Techniques in Web
-
based Education Systems
,
2005.

[Carbonaro
et al.
, 2006] A. Carbonaro, G. Cavassi, R. Ferrini, M.Ghinelli. “Ontology
-
based video
annotat
ion and recommendation in a semantic education framework”.
SWET
, 2006.

[García
et al.
, 2004] F. J. García, A. J. Berlanga, M. N. Moreno, J. García, J. Carabias.
“HyCo
-

An Authoring Tool to Create Semantic Learning Objects for Web
-
Based E
-
learning Systems”
.
Lecture Notes on Computer Science
, Vol. 3140, pp 344


348, January 2004.

[Gruber, 1993] T. Gruber. “Conceptual Analysis and Knowledge Representation” chapter toward
Principles for the Design of Ontologies Used for Knowledge Sharing in Formal Ontology
, K
luwer
Press, 1993.

[ISO/IEC, 2002]. ISO/IEC. “Overview of the MPEG
-
7 Standard (version 8)”.
ISO/IEC
JTC1/SC29/WG11/N4980
, Klagenfurt, July 2002.


[Koper, 2001] R. Koper. “Modelling Units of Study from a Pedagogical Perspective. The
Pedagogical Metamodel be
hind EML”. 2001.

[Nejdl
et al.
, 2002] W. Nejdl, B. Wolf, C. Qu, S. Decker , M. Sintek, A. Naeve, M. Nilsson, M.
Palmér, T. Risch. “EDUTELLA: A P2P Networking Infrastructure Based on RDF”, 2002.

[OWL, 2004
]
M. Dean
,

G. Schreiber.

“Ontology Web Language (OWL
)”.

W3C Recommendation
,

10 February 2004,
http://www.w3.org/2004/OWL/.

[RDF, 2004
]
E. Miller
,

F.

Manola
,


Resource Description Framework (RDF)
”.

W3C
Recommendation
,

10 February 2004,
http://www.w3c.org/RDF/
.

[Simon
et al.
, 2003a
]
B.
Simon,

Z.

Miklos,

W.

Ne
jdl,

M.

Sintek,

J.

Salvachua
.


Elena: A
Mediation Infrastructure for Educational Services
”.
Twelfth International World Wide Web
Conference
, 2003.

[Simon
et al.
, 2003b
]
B.
Simon, Z. Miklos, W. Nejdl, M. Sintek, J. Salvachua.


Smart Space for
Learning: A Me
diation Infrastructure for Learning Services


Proceedings of the
Twelfth
International Conference on World Wide Web
, 2003.

[UML, 1999] G. Booch, J. Rumbaugh, I. Jacobson. “The Unified Modelling Language Use
Guide”
.
Addison
-
Wesley, 1999.

[XML, 2004
]
F.
Yerg
eau,

T.

Bray,

J.

Paoli,

C.

M.
Sperberg
-
McQueen,

E.

Maler
.


Extensible
Markup Language 1.0 (XML) (Third Edition)
”.

W3C Recommendation
,

4 February 2004,
http://www.w3c.org/XML/
.