A Semantic Middleware to enhance

blaredsnottyΤεχνίτη Νοημοσύνη και Ρομποτική

15 Νοε 2013 (πριν από 3 χρόνια και 6 μήνες)

278 εμφανίσεις

Computer Science Faculty.Computer Languages and Systems Department
A Semantic Middleware to enhance
current Multimedia Retrieval
Systems with Content-based
,a dissertation submitted to the Department of Computer Languages and
Systems of the Computer Science Faculty of the University of the Basque
Country in partial fulfillment of the requirements for the degree of Doctor
of Philosophy
by Gorka Marcos Ortego
This dissertation is supported by the following advisors
Dra.María Aranzazu Illarramendi Echabe and Dr.Julián Flórez Esnal
Donostia-San Sebastián,
© Servicio Editorial de la Universidad del País Vasco

Euskal Herriko Unibertsitateko Argitalpen Zerbitzua
ISBN: 978-84-694-6824-1
List of Tables ix
List of Figures xi
1.Introduction 7
1.1.Scope of this thesis..........................7
1.2.Problem identification and motivation of this thesis.........8
1.2.1.New needs in the content creation and consumption industry 8
1.2.2.Context and contributions of this thesis...........9
1.3.How to read this thesis work......................12
2.Technological Context 15
2.1.Multimedia in the information retrieval theory............16
2.1.1.Information versus Data retrieval...............17
2.1.2.Summary of a long history..................17
2.1.3.Information Retrieval Models:classical and modern....18
2.1.4.Multimedia Information Retrieval (MIR)............20
2.2.Semantic enhancement of MIR systems...............25
2.2.1.An intelligent media framework for Multimedia Content...26
2.2.2.Information Mediation Layer:a new component for the
digital libraries architecture..................28
2.2.3.A model for multimedia information retrieval.........31
2.2.4.A three layer infomediation architecture...........31
2.2.5.Ontology Based Information retrieval.............33
2.2.6.Ontology-enriched semantic space for Video Search....33
2.2.7.MPEG-7 driven multimedia retrieval.............34
2.3.Metadata models for multimedia...................36
2.3.1.Types of multimedia metadata................36
2.3.2.EBU P/Meta..........................38
2.3.3.Standard Media Exchange Format - SMEF.........40
2.3.4.Broadcast Exchange Metadata format - BMF........40
2.3.5.Dublin Core...........................41
2.3.6.TV Anytime...........................41
2.3.8.SMPTE Descriptive Metadata.................46
2.3.9.PB Core.............................48
2.3.11.Extensible Metadata Platform XMP..............50
2.3.12.Other Standards........................51
2.3.13.Criteria to choose the best standard.............52
2.4.Content based retrieval........................54
3.Semantic Middleware to enhance multimedia information retrieval
systems 59
3.1.Multimedia Information Retrieval Reference Model.........59
3.2.Semantic Middleware,a three Layered Architecture.........63
3.2.1.Requirements of the middleware...............63
3.2.2.Middleware Architecture....................64
3.2.3.Semantic Middleware Knowledge Base (SMD KB).....65
3.2.4.Semantic Middleware Intelligence Engine (SMD IE)....66
3.2.5.Semantic Middleware Gateway (SMD GW).........66
3.3.Key design criteria...........................67
3.3.1.Semantic Middleware Knowledge Base (SMD KB).....67
3.3.2.Semantic Middleware Intelligence Engine (SMD IE)....69
3.3.3.Semantic Middleware Gateway (SMD GW).........70
4.Other contributions 71
4.1.DMS-1 OWL ontology.........................71
5.WIDE use case:Semantic Middleware for multimedia retrieval from
multiple sources used by a multidisciplinary team in a car industry
domain 79
5.1.WIDE system..............................80
5.1.1.Motivation of the system....................80
5.1.2.Objectives of the system....................81
5.1.3.System architecture......................83
5.1.4.Search Workflow in WIDE...................83
5.1.5.Meta-Level,the SMD in WIDE................84
5.2.Description of WIDE SMD.......................85
5.2.1.WIDE SMD functionalities...................85
5.2.2.SMD functionalities in an scenario..............89
5.2.3.Summary of services.....................90
5.3.Key design criteria of the SMD....................91
5.3.1.WIDE SMD KB.........................91
5.3.2.WIDE SMD IE.........................94
5.3.3.WIDE SMD GW........................95
5.4.Implementation details of the SMD..................95
5.5.WIDE SMD Evaluation.........................120
6.RUSHES use case:Semantic Middleware to enable automatic
analysis techniques in large repositories of un-edited material in the
domain of a broadcaster 123
6.1.RUSHES system............................124
6.1.1.Motivation of the system....................124
6.1.2.Objectives of the system....................125
6.1.3.System architecture......................126
6.1.4.Metadata Model,the SMD in RUSHES............127
6.2.Description of the RUSHES SMD...................128
6.2.1.RUSHES SMD functionalities.................128
6.2.2.SMD functionalities in an scenario..............130
6.2.3.Summary of services.....................133
6.3.Key design criteria of the SMD....................134
6.3.1.RUSHES SMD KB.......................134
6.3.2.RUSHES SMD IE.......................135
6.3.3.RUSHES SMD GW......................136
6.4.Implementation details of the SMD..................137
6.4.1.MDM KB............................137
6.4.2.MDM IE.............................142
6.4.3.MDM GW............................144
6.5.RUSHES SMD Evaluation.......................146
7.Conclusions and future work 151
7.1.Summary of Conclusions.......................151
7.2.Future work...............................153
7.2.1.Architecture for semi-automatic multimedia analysis....153
7.2.2.Content-based retrieval functionalities in broadcast
7.3.Summary of publications........................157
7.3.1.Publication related to the contributions of this thesis....157
7.3.2.Publications of the future work................159
7.3.3.Other publications in the field.................159
ANNEX I:OWL-Rep structure 165
ANNEX II:BNF grammar 169
ANNEX III:Process Support File 173
ANNEX IV:Graph Format 175
ANNEX V:Result Format 177
Bibliography 183
List of Tables
1.1.Summary of contributions and information about their location in
the report................................12
2.1.Summary of the state of the art MPEG-7 based multimedia ontologies 46
2.2.Description of PB Content Classes..................48
2.3.XMP Rights Management Schema..................51
3.1.Examples of semantic services to be provided in a MIR system..62
List of Figures
2.1.Graphical summary of the Technological Context Chapter.....16
2.2.Model classification in modern Information Retrieval........19
2.3.Architecture of the Intelligent Media Framework Component....27
2.4.Model of the Knowledge Content Objects of the IMS........28
2.5.Reference Model for the Digital Libraries...............29
2.6.Query Decomposition process....................30
2.7.Layered Information Architecture & Processes...........32
2.8.View on ontology-based information retrieval............33
2.9.System overview of the infrastructure components for multimedia
2.10.BMF root nodes............................39
2.11.TVAnytime Metadata Model framework................40
2.12.TVAnytime metadata model summary................42
2.13.Multimedia Description Schemes...................44
2.14.Query by Example based on an MPEG-7 descriptor........47
2.15.Descriptive Metadata Frameworks and their Relationship to the
Content of an MXF File Body.....................49
2.16.Summary of Clip Framework Schema................50
3.1.Information Retrieval and Storage Reference Model by Soergel..60
3.2.Information Retrieval Reference Model................61
3.3.SMD three layered architecture....................65
4.1.Fragment of the implemented DMS-1 schema based on the
aggregation relation..........................72
4.2.JPSearch Architecture.........................74
5.1.WIDE problem statement.......................80
5.2.Screenshot of WIDE visual tool for domain browsing........81
5.3.Architecture of WIDE system.....................82
5.4.Classical search workflow.......................83
5.5.Search model implemented in WIDE.................84
5.6.Selection Panel for Task Types....................86
5.7.Protégé 2000 Annotation Panel...................87
5.8.Results browsing in WIDE.......................89
5.9.Meta Level Architecture........................96
5.10.Ontologies in WIDE ML KB......................98
5.11.Screenshot of ContentType ML Ontology..............100
5.12.Overview of relationships hierarchy.................102
5.13.ML approach for Process Context Management...........103
5.14.Input field of WIDE user interface front-end.............105
5.15.ASF interpretation of the query....................106
5.16.Example of RQL System Query...................113
5.17.Visualization of the instance view..................114
6.1.Architecture of RUSHES system...................127
6.2.Logical architecture of the CCR Service Domain..........129
6.3.Information storage and metadata generation in RUSHES.....131
6.4.RUSHES SMD architecture......................138
6.5.Protégé OWL editor..........................138
6.6.Approach to express the fuzziness by employing annotations...139
6.7.Partial view of the MDM GWinterfaces................145
7.1.Proposed Architecture for semi-automatic multimedia analysis by
hypothesis reinforcement.......................154
7.2.Preliminary results of the classification process...........155
7.3.Architecture for a location aware systemfor monitoring sports events156
Esta tesis ha sido posible por el soporte,la dedicación,el saber hacer y el tesón
de mis dos directores de tesis,Arantza y Julián,y por la confianza y el apoyo
que Vicomtech-IK4 ha depositado en mí.
En estos años,he ido tejiendo este trabajo en estrecha relación con muchas
personas.Esta tesis ha sido sin duda posible gracias a ellos.En los proyectos
que me han servido para validar este trabajo,he tenido la suerte de trabajar
con más de 60 expertos de distintos ámbitos y países.Estoy especialmente
agradecido a aquellos que generosamente han compartido su conocimiento y
debo recordar especialmente al equipo de compañeros de ETB.En Vicomtech-
IK4,también he estado muy bien acompañado.No habría podido realizar esta
tesis sin la ayuda y generosidad de Ivanjou,Kevin,Tim y Jorge.Y nunca me
podré olvidar del apoyo incondicional que en todo momento me han dado Igor,y
con él,todos mis compañeros del departamento,con los azules a la vanguardia.
Ahí va,de nuevo,un mila esker para todos vosotros.Y Petra,esto te incluye
también a ti.
Regarding some key contributions that I received,I can’t forget the kindness
of the Professor Ray Larson and the generosity of the NTUA team granting me
access to their reasoner.Phivos,thank you once again.I also want to thank
Oliver (Fraunhofer-HHI) and Sergio (University of Brescia) for the time they kindly
devoted to review this work.
Y los amigos de las cuadrillas de Donosti y Bilbao y"de"Vicomtech-
IK4.Gracias por estar ahí y por haber compartido conmigo los progresos y
A Iñigo,Maider,Laia y toda la familia de allá y de acá.En este camino
me habéis apoyado,comprendido y ayudado en todo lo que habéis podido,sin
preguntas ni condiciones.Soy feliz por seguir compartiendo el CAMINO con
Aita,Ama...esto,como tantas cosas,lo empezamos juntos.Sin vosotros,no
hubiera sido posible.¡ Gracias!
Beizama,Febrero de 2011.
2 Acknowledgment
This work reviews the information retrieval theory and focuses on the revolution
experimented in that field promoted by the digitalization and the widespread
use of the multimedia information.After analyzing the trends and promising
results in the main disciplines surrounding the content-based information retrieval
field,this thesis proposes a reference model for Multimedia Information Retrieval
that aims to contextualize the thesis contributions.According to this reference
model,this work proposes an architecture for a component named “Semantic
Middleware” that aims to centralize the main semantic services to be provided
during the indexing,storage,search,retrieval and consume of the multimedia
elements.This architecture has been designed from a pragmatic point of view,
aiming to facilitate the enhancement of the current systems with content-based
functionalities.The architecture proposal includes a set of key design criteria
for a right deployment.In order to validate this thesis,two real complementary
deployments have been performed and reported in this work.
4 Summary
Part I
1 Introduction
This chapters aims to describe the scope of this thesis,its motivation and
context.The chapter ends with a section that aims to facilitate the reading of
this document.
1.1 Scope of this thesis
Looking up in the IT business and computer industry dictionaries,is possible to
find diverse definitions of the term “Middleware”.One extended definition is the
one proposed by Kavanagh and Thite (2009):
general term for any programming that serves to “glue together” or
mediate between two separate and often already existing programs.A
common application of middleware is to allowprograms written for access
to a particular database to access other databases.
In a coherent way with this statement,we define the term “Semantic
Middleware” as that piece of software that semantically glues together
different existing programs that co-exist with a common target.Specifically,
this thesis work is related to the semantic tying among the different modules or
components that are frequently part of complex multimedia information retrieval
and management systems.
Therefore,this thesis,based on a diagnosis of the semantic needs of those
systems,propose a generic architecture to define a middleware that fulfills those
needs in a pragmatic,feasible and beneficial way from the programmatic point of
Along the following sections,we establish and define the problems and
motivation behind this work.Once this has been clarified,we describe the
structure of this thesis work at the end of the chapter.This structure has been
defined to expose and clarify the different details about the technological context,
the definition,implementation and validation of this work.
1.2 Problemidentification and motivation of this thesis
In this section we firstly present the revolution experienced by the media industry
in the last years.Then,we describe the impact of such revolution from the
perspective of the scope of this thesis work.
1.2.1 New needs in the content creation and consumption industry
During the last decade there have been different phenomenons that have led
to a deep and huge revolution in the way the content is created,managed and
exploited.In the following,we highlight the most significative ones:
 First of all,the digitalization of the content.The content is not just what
we can find inside a tape,book or disc that is stored in a specific shelf of
an archive,but an entity per se.The disappearance of the physical part of
the content has increased its protagonism.The expects and needs of the
users have changed.In many situations,the user is not looking just for an
identifier,reference number or a title (as in the traditional libraries) but for
a content that contains a specific piece of information,sentence,image or
piece of audio.
 Closely related to the previous item,the evolution of the information
management and retrieval systems has led to the appearance of a new
generation of products,such as the MAMs or “Media Asset Management”.
These products are not merely repositories of digital assets,but also aim
to digitalize the whole workflow of the content creation,generation and
exploitation.We would like to highlight three phenomenons linked to the
digitalization of those process.First of all,the migration from tape-based
archives to digital libraries accessible on the Intranet has changed the way
the metadata (information about the content) is generated (Avilés et al.,
2005).The metadata is not just generated in a specific point of the workflow
and is related to many different aspects of the asset (e.g.,legal,internal,
technical).Secondly,and related to the previous one,the content is not
just a unique entity,but a set of entities linked (e.g.,the video,several
audio tracks,the script,the metadata).It is only this set of entities the one
that conform what was previously understood as content.And finally,and
fulfilling the premonitions of Serb (1997),the roles of the people working in
organizations that handle content,have significantly changed.For instance
while in a broadcaster almost all the annotations were handled by the people
working in the archive,nowadays,due to the presence of the MAMsystems,
the journalists have the main role in the generation of those metadata.This
has an impact in the coherence and soundness of that information.
 The maturity achieved by the technologies for the storage and data network
has significantly contributed to the mentioned digitalization and therefore
in the accessability of the content.Nowadays the manufacturers of media
asset management systems include high resolution video storage solutions
that are feasible for small media producers.
 The explosion in the generation of content is not only due to a specific factor
but to a set of factors:the globalization of the society,the democratization
of the digital devices,the appearance of new communication platforms and
the consequent increase of media companies,etc.All these factors have
definitely contributed to the current situation,as the work of Pastra and
Piperidis (2006) corroborates.There is a need of handling or controlling this
digital content explosion.This is mainly due to the fact that the explosion
has occurred in a relatively short period of time,and many organizations
have not been able to either adapt the way they deal with the content,or
modify the business model in such a way that the establishment of new
ways of dealing with this content is a feasible task.
 The new communication channels (Internet,mobile networks),have also
contributed to this new scenario,where the content is created,accessed
and shared by users that were not active part of the content life-cycle.We
are not only referring to the generation of content made by final users (i.e.,
prosuming) but also to the fact that the investment required to make the
content accessible to the general public has decreased significantly.And the
involvement of such public,usually leads to the generation of new content
associated to that content.
This new context surrounding the generation and consumption of content and
its metadata has implied and will imply in the next years deep changes in almost
all the business processes of the media industry.This thesis focuses on a specific
aspect of this revolution.We are concerned about the way the content should be
managed in this context in order to take advantage of the semantic richness of the
content itself.In this sense,this thesis is a contribution in between the information
retrieval systems that have been adopted by the industry and the achievements
of the scientific field “Multimedia Semantics”.We cover this issue in the following
1.2.2 Context and contributions of this thesis
As we have stated,our contribution aims to support the media industry in order
to increase the semantic exploitation of their content.In order to do this from a
pragmatic perspective,this work is located in between the technology acquired
by the industry during the digitalization and the achievements of the scientific field
“Multimedia Semantics”.
Regarding the systems that drive the media storage and retrieval in the
industry,we highlight the following facts:
 Independently of the domain,the technology massively employed is the
relational database together within search algorithms of different nature.
This technology is mature,well established and,in fact,as we have
previously stated,has been and is one of the main drivers of the
digitalization process.
 Beside this,in those sectors with the highest amount of media
content generated (e.g.entertainment industry),most of the systems
are proprietary or customized solution (Datamonitor-Analysists,2007;
 Regarding the employment of common structure for the modeling of the
information,excepting some niche sectors (e.g.,libraries),there is an
important lack of homogeneity.Although there are multiple standards
coming from different forums,most of the companies organize their
information following their own internal criteria.This was one of the main
conclusions of the professional Workshop of Annotations and Metadata
models for Audiovisual/Multimedia hold in the context of the CHORUS forum
(Metadata-Professional-workshop,2007) and is also supported by the book
of Cox et al.(2006).
 We have had the chance to know the systems of seven Spanish
broadcasters (being five of them local broadcasters) and all the major
content producers of the Basque Audiovisual Cluster
.Many of them have
already faced this digitalization process,investing very important amounts
of money,but in most of the cases they have just replicated in a digital way
the organization schemas that they had in their analogue archives,without
taking use of the opportunities of the digitalization.In some cases,they
are already facing the upgrade and customization of those systems in order
to include some preliminary semantic functionalities (e.g.,automatic query
expansion based on synonyms).
In this context,and coming from the search,retrieval and image analysis
communities,a newscientific community has been devoted to the improvement of
the multimedia content retrieval by employing content-based functionalities.This
community,frequently tagged as “Multimedia Semantics”,aims,according to the
definition of Giorgos Stamou and Stefanos Kollias (Furht,2006),to deal with the
question how to conceptually index,search and retrieve the digital multimedia
content,which means how to extract and represent the semantics of the content
of the multimedia raw data in a human and machine-understandable way.
As we present later,this community is providing very interesting and promising
results aligned with that aim.They are bringing new means of extracting
information out of the multimedia content.The correct storage of this information
combined with new search techniques are presented as the basis for the
scenarios of the future multimedia information retrieval.
In this context,our aim is not the contribution to the generation of those
new systems in the long term,but the adaptation of the current retrieval and
storage technologies in order to increase their performance by the gradually
integration of the emerging content-based features.We are not proposing a
revolutionary paradigm for multimedia retrieval but a straightforward approach
based on the deployment of a middleware to enrich current Multimedia
Information Retrieval (MIR) systems with successful semantic applications that
benefit from the understanding of the multimedia asset.This middleware is a
three layered semantic middleware that has been designed to provide semantic
services needed by different content-based applications involved in conventional
multimedia retrieval workflow.The main feature of this middleware is that it
centralizes the semantic knowledge and the provision of semantic services
in the system.Below,we summarize the main advantages of our proposal:
 Outsourcing.The middleware facilitates the integration into existing systems
since the semantic services are outsourced from the retrieval engine(s).
 Uniqueness.The middleware avoids current semantic duplicities
imposed by the employment of satellite applications (e.g.,content-based
recommendation,ontology-based clustering).This simplifies the work of
knowledge engineers,since the upgrading of the knowledge representation
of the domain is performed in a single place.
 Semantic interoperability.The middleware includes a semantic
representation of the knowledge which is format-agnostic.In those
cases where the middleware is working with components or information
sources that employ different formats or languages,the architecture
of the middleware provides simple mechanisms to perform the needed
adaptations and carry out the upgrades derived from the evolution of each
of the peers.
We have also contextualized this middleware within a global multimedia
reference model and provided a set of key design parameters for its correct
deployment.Finally,we have validated this middleware by the implementation of
two deployments in two real complementary scenarios belonging to different
industrial sectors.
1.3 How to read this thesis work
In the chapter following this introductory one (Chapter 2),we take care of the
technological context of this thesis.In that chapter,we introduce the basis of the
Multimedia Information Retrieval (MIR) theory.Once this is clarified,we cover
three scientific and technological issues directly related to this thesis work:(i) the
initiatives that have similar aims or similar approaches to this work,(ii) a review
of the main multimedia metadata models and (iii) a summary of key contributions
in the field of the content-based retrieval.With this chapter,the introductory part
In the second part of this thesis work we deal with the contributions of this
thesis,which are summarized in Table 1.1.
To acquire a complete understanding of this semantic middleware,its design
and validation,the reader may turn to the Chapter 3,which covers the following
 First of all,in order to contextualize this thesis,we propose a reference
model for multimedia information retrieval.This can be found in Section
 Once we have defined the context of this thesis,the middleware or main
contribution of this thesis work,is described.Section 3.2 covers the
definition of each of the layers of the proposed architecture for this
 With the aim to support a right deployment of this middleware in a real
system Section 3.3 includes a list of key criteria to have in mind.
At the end of this part,Chapter 4 provides an overview of some other minor
contributions in this thesis.
Table 1.1:Summary of contributions and information about their location in the report
MIR Reference Model Proposal
Semantic Middleware:definition
Semantic Middleware:deployment design criteria
USE CASE WIDE:deployment implementation
USE CASE RUSHES:deployment implementation
DMS-1 OWL ontology
Contribution to JPSearch Standardization activity
Summary of Publications
In the third part of this thesis,we cover the validation of the proposed
semantic middleware architecture.We include two real deployments of the
middleware:WIDE (Chapter 5) and RUSHES (Chapter 6).For each of the use
cases the following aspects are covered:
 Contextualization:Each semantic middleware deployment has been
implemented within a global system.In order to contextualize the
development of the semantic middleware,we include a description of
the global system,the motivation that led to its development and its
 Functionalities:Once the context of the semantic middleware has
been described,a description of the functionalities implemented by the
middleware is included.
 Mapping with the proposed middleware:The middleware proposed in
the second part of this thesis,has three lines:Browsing Line,Search Line
and Storage Line (described in Figures 3.2 and 3.3)).Both scenarios cover
the Browsing Line included in that reference architecture.Regarding the
other two lines,the validation scenarios are complementary.On the one
hand,the first deployment provides a wide range of services for the Search
Line in order to cover different information sources.On the other hand
the second scenario implements different services devoted to support the
Storage Line,in order to provide advanced indexing mechanisms of the
multimedia assets.Therefore,the combination of the scenarios provide
a global overview of the provision of services for the three lines of the
reference model for multimedia retrieval.
 Identification of design criteria:In a coherent view with the requirements
identified in Section 3.3,we detail the decisions taken for each semantic
 Implementation details:We identify the main implementation details for
each of the three layers that compose each semantic middleware.
 Validation:Finally we provide information about the validation of both
semantic middleware.
The fourth part of this thesis work,covers the summary of the conclusions of
this thesis,the main future work action lines started and our publications (Chapter
At the end of this thesis,there are two parts including some annexes and the
list of referenced bibliography.
2 Technological Context
This chapter aims to provide a focused review of the technological context of this
thesis work.In the following section,we provide a general introduction to the
Information Retrieval (IR) field followed by a description of the impact in the field
promoted by the targeting of the multimedia assets.We also cover the significant
role played by the Multimedia Analysis community in such revolution.Accordingly
with this,we include a brief description of the current context,techniques and
challenges of that community.
After this introductory section,we concentrate on three topics that are deeply
related to the semantic middleware that we present.First of all we provide a
description of the IR field.Then we include a section focused on the relevant
contributions found in the literature that propose a system,architecture or
middleware to promote and enable the usage of content-based functionalities in
retrieval systems.Each contribution is reviewed and the main differences with
respect to this thesis work are highlighted.
Furthermore,in order to understand the important efforts that the scientific
and industrial communities are spending in facilitating the management,sharing
and retrieval of multimedia assets,a review of the main metadata models is
performed.From our point of view,due to the fact that the model is the main
element to provide the semantic services targeted,an understanding of the
current context of such models is a key criteria for a successful enhancement
of a MIR system with content-based functionalities.
Finally,in order to grasp the current status of the semantic-aware techniques
being developed by the scientific community regarding different stages of the MIR
process,a summary of several main contributions and key surveys in the content-
based multimedia information retrieval field is provided.
Figure 2.1 supports graphically the structure of this chapter.In this Figure
we show how the deployment of a semantic middleware (Section 2.2),the
employment of multimedia metadata models (Section 2.3) and the integration
of content-based techniques (Section 2.4) are key contributions to current MIR
systems (Section 2.1) with content-based functionalities.
Multimedia Metadata Models
Content-Based Multimedia Techniques
Semantic Middleware
Figure 2.1:Graphical summary of the Technological Context Chapter
2.1 Multimedia in the information retrieval theory
Information Retrieval (IR) can be understood as the field related to the storage,
organization,and searching of collections of data.But behind this simple
definition,there is a little confusion.As Styltsvig (2006),based on the work
of Lancaster (1968);van Rijsbergen (1979),remembers,Information Retrieval
systems do not actually retrieve information,but rather documents from which
the information can be obtained if they are read and understood.To be more
precise,that which is being retrieved is the system’s internal description of the
documents,thus as the process of fetching the documents being represented is
a separate process.Despite this loose definition,information retrieval is the term
commonly used to refer to this kind of process,and thus,whenever we used the
term Information Retrieval,it refers to this text-document-description retrieval
definition.Moreover,whenever the type of document that is retrieved is not only
a text-document but any kind of digital asset,we employ the term Multimedia
Information Retrieval.
In the following subsections,we first try to clarify the distinction between
Information Retrieval versus Data Retrieval.Secondly,we present a short
summary of the history of IR.After this,we provide an overview of the IR models.
Finally,we include a deeper analysis of the state of the art in multimedia retrieval.
This analysis includes some introductory concepts about the content analysis
field,which,as will be stated,is a key agent in the development of MIR systems.
2.1.1 Information versus Data retrieval
We find the inclusion of the distinction between Information and Data retrieval
proposed by Baeza-Yates and Ribeiro-Neto (1999) in the context of this thesis
convenient.The term Data retrieval should be employed whenever the main
objective is the determination of which documents of a collection contain the
keywords that the user employed in a query.However,most frequently,that it
is not enough to satisfy the user information need.In fact,the user of an IR
system is concerned more with retrieving information about a subject than with
retrieving data which satisfies a given query.A data retrieval language aims at
retrieving all objects which satisfy clearly defined conditions such as those in a
regular expression or in a relational algebra expression.Thus,for a data retrieval
system,a single erroneous object among a thousand retrieved objects means
total failure.For an information retrieval system,however,the retrieved objects
might be inaccurate and small errors are likely to go unnoticed.The main reason
for this difference is that information retrieval usually deals with natural language
text which is not always well structured and could be semantically ambiguous.On
the other hand,a data retrieval system(such as a relational database) deals with
data that has a well defined structure and semantics.
This difference is even more evident in the case of this work,when the assets
to be retrieved are multimedia ones.This,as stated Cusumano (2005) is even
noticed in the attitude of the user,which is usually more tolerant to the lack of
precision of the systems.
2.1.2 Summary of a long history
Singhal (2001) from Google refers to the Sumerians (3000 BC) to locate the
beginning of the practice of archive information.Professor Larson (2010)
also mentions the Sumerians but goes even backwards,considering that
the mnemonic systems probably developed in prehistoric times can also be
considered a form of mental IR.Although it is the aim of this thesis to discuss
the origin of the Information Retrieval,we share the idea that the need to archive
and retrieve information became more and more important during the centuries.
Even more,with the invention of the paper and the printing press.The computers
also were employed for this aim.
The article “As We May Think” written by Bush (1945) is considered as the
beginning of the automatic access to large amounts of data stored.In the fifties,
several works were developed about the basic idea of searching and finding
text with a computer.The work of Luhn (1957) is one of the key references
of that period.During the next decade,several key developments in the field
happened.Most notable were the SMART system developed by Salton (1971),
first at Harvard University and later at Cornell University.
Based on the work of this decade,during the 1970s and 1980s many
developments were built.Several models for retrieving documents were
developed and the progress in all the steps of the retrieval process were
important.The experiments were tested on small text collections (several
thousand articles) available to researchers.This lack of large collections was
solved with the 1992 Text Retrieval Conference or TREC,which established
objective methodologies and measurements for information retrieval systems,
that are employed nowadays.
The algorithms developed during those decades were the first ones to be
employed for searching the World Wide Web from 1996 to 1998.However,
the powerful provided by the cross linkage available on the web led to the
implementation of new approaches,which are out of the scope of this thesis.
As we analyze later in 2.1.4,the explosion of the multimedia asset has led,in
the recent years to a new revolution in the field.
2.1.3 Information Retrieval Models:classical and modern
As Larson (2010) defines,a model for IRis a specific and distinct approach for the
text processing and the ranking algorithms of the system.A shared agreement in
the key literature in IR is that the main classic information retrieval models are the
following:Boolean,Vector Space,and Probabilistic.In addition there are many
systems that are hybrids of two or more of these models (e.g.,a vector system
with Boolean result limiting features).
The earliest retrieval model is the Boolean model,described in the work
of Gudivada et al.(1997),f and is based on Boolean logic.Most of the
earliest commercial search services,local search engines or individual Web sites
implement this model.The Boolean model is a set-oriented model,where sets of
documents are defined by the presence or absence of an individual index term.If
the term is there,and the logic of the boolean (AND,OR...) query is fulfilled,the
document is retrieved.Boolean systems have several disadvantages.Perhaps
the most serious is that there is no inherent notion of ranking.
The vector space model,deeply described in the work of Salton et al.(1975),
represents a document as a vector of terms.Vector space IR systems base
implement ranking algorithms according to how close together the vector of
the query and the vector of all the documents are.So,the ranking is a kind
of similarity measure based on the terms employed in the query and in the
Figure 2.2:Model classification in modern Information Retrieval
documents archived.
The probabilistic model is based on what is called the Probabilistic Ranking
Principle (PRP):the documents of a collection should be ranked by decreasing
probability of their relevance to a query (Robertson,1997).Relevance is therefore
defined as a subjective assessment by a given user or machine of the value or
utility of a particular document in satisfying a particular need for information.
Baeza-Yates and Ribeiro-Neto (1999) go further and,as can be seen in Figure
2.2,beside the classical models,which are deeply explained,two newmodels are
included for the retrieval and three for the browsing.
The structured models are aware and make use of certain knowledge of
the structure of the document.Inside this category we distinguish the non-
overlapping lists and Proximal Nodes approaches.First of all,regarding the non-
overlapping lists approach,Burkowski (1992) proposes to divide the whole text
of each document in non-overlapping text regions which are collected in a list.
Since there are multiple ways to divide a text in non-overlapping regions,multiple
lists are generated.According to this,a book may be composed of a list of all
the chapters,a list of all the sections and a list of all the subsections.While the
text regions in the same (flat) list have no overlapping,text regions from distinct
lists might overlap.And,once these lists are created the approach is similar to
the one employed in the vector-space,but applying it for each list.Secondly,the
proximal nodes models proposed by Navarro and Baeza-Yates (1997) propose a
model which allows the definition of independent hierarchical (non-flat) indexing
structures over the same document text.Each of these indexing structures is
a strict hierarchy composed of chapters,sections,paragraphs,pages,and lines
which are called nodes.To each of these nodes is associated a text region.
Further,two distinct hierarchies might refer to overlapping text regions.Given
a user query which refers to distinct hierarchies,the compiled answer is formed
by nodes which all come from only one of them.Thus,an answer cannot be
composed of nodes which come from two distinct hierarchies (which allows for
faster query processing at the expense of less expressiveness).
Finally,regarding the browsing models,Baeza-Yates and Ribeiro-Neto (1999)
define three approaches:flat,structure guided browsing and hypertext.First of
all,with the flat model the idea here is that the user explores a document space
which has a flat organization.For instance,the documents might be represented
as dots in a (two dimensional) plan or as elements in a (single dimension) list.The
user then glances here and there looking for information within the documents
visited.Secondly,the structure guided browsing tries to facilitate the task of
browsing organizing the documents in a structure such as a directory.Directories
are hierarchies of classes which group documents covering related topics.Finally,
the hypertext is a high level interactive navigational structure which allows us to
browse text non-sequentially on a computer screen.It consists basically of nodes
which are correlated by directed links in a graph structure.
2.1.4 Multimedia Information Retrieval (MIR)
The explosion of multimedia content caused by the digitalization and the
convergence of the technology has conducted to a new revolution in the
information retrieval.This revolution has led to newtrends and techniques for very
diverse aspects of the retrieval (Tse,2008):object representation,architecture
for storage systems,data compression techniques,statistical placement of
discs,scheduling methods for disks requests,multimedia pipelining and stream
dependent caching among many others.
However,the main impact of this revolution in this work is related to the way
the metadata is created and made accessible for the search and retrieval.In this
context,the mentioned revolution has impacted on two scientific fields,blurring
the boundaries between them:Multimedia Information Retrieval (MIR) and Image
and Video Analysis.
On the one hand,in the IR field,the inclusion of the multimedia assets in the
information retrieval implies new means for the storage,retrieval,transportation,
and presentation of data with very heterogeneous features such as text,video,
images,graphs,and audio.Baeza-Yates and Ribeiro-Neto (1999) in their
book about the modern concept of Information Retrieval already include several
chapters focused on the techniques and approaches to retrieve multimedia
assets,as an emerging particularity of IR.The motivation behind this new
activity is due to the fact that traditional IR techniques are very efficient from
the performance and precision point of view when the fundamental unit is the
textual document and the search is based on text and carried out over simple
data types.However,in the case of multimedia information retrieval the underlying
data model,the query language,and the access and storage mechanisms must
be able to support objects with a very complex structure.Furthermore,the
scientific work devoted to establish the foundations of the next generation of
multimedia information retrieval systems,such as the remarkable contribution of
Meghini et al.(2001),are slowly having an impact in the commercial products.
An example of this preliminary deployment of such concepts is the last version
of the multimedia database of Oracle,which is able to handle and perform some
operations on new object types (e.g.,DICOM images from the medical sector).
On the other hand,the image and video community has spent remarkable
efforts during the last years to promote what they coin as “Content or
semantic based visual/multimedia information retrieval” (CBVIR) (Lew et al.,
2006;Naphade and Huang,2002).According to Zhang (2006),CBVIR has
already a history of fifteen years,but it is in the last years when the focus has
moved from extraction of low-level features from the multimedia assets (e.g.,
dominant colour in an image) to the resolution/minimization of the semantic
gap (e.g.,person recognition).The community is devoted to a higher level of
semantic abstraction.This is called by the author as Semantic-based Visual
Information Retrieval,and is leading to the application of such technologies for the
enhancement of current multimedia management and retrieval systems.Among
the processes being improved,we may highlight the followings:indexing and
retrieval,higher-level interpretation,video summarization,object classification
and annotation,and object recognition.We also want to notice that in all these
disciplines,the presence of the technologies developed by the semantic web
community has been significantly increasing during the last years.The book
edited by Stamou and Kollias (2005) provides a very complete summary of the
efforts made by the community to perform the semantic analysis required for the
multimedia information retrieval.
In this section,we try to provide an introductory explanation to some of the
most relevant techniques and current challenges for the main disciplines involved
in that task.The image,video and audio components are treated independently.
For each of them,we identify the key research issues and trends.In Section
2.4,we include a brief summary of the state the art in content based information
retrieval in the context of this thesis.That summary has a more technical and
specific perspective.
Image retrieval
When the retrieval is about images,the metadata is not the only valuable piece
of information,but also,thanks to the image analysis techniques,of the features
of the images (Zhang and Izquierdo,2008).According to Eakins and Graham
(1999),depending on the features of the image you employ for the retrieval,the
queries can be classified into three levels,being each of the level of different
 Level 1:Primitive features such as colour,texture,shape,or the spatial
location of image elements.
 Level 2:Derived features involving some degree of logical inference about
the identity of the objects depicted in the image.
 Level 3:Abstract attributes involving a significant amount of high-level
reasoning about the meaning and purpose of the objects or scenes
The first level of queries is the set of queries that are more easily solved.All
the information is gathered in the image and,therefore,there is not need of any
external intelligent resource.This type of queries is relatively easy to solve but is
largely limited to specialist applications.Levels 2 and 3,which are in fact most
widely demanded,together are commonly referred to as semantic-based visual
information retrieval.However,there is an important gap between Level 1 and 2,
referred to as the semantic gap (Smeulders et al.,2002).
The bridging of this semantic gap is the main objective of most of the
research activity of the scientific community working in image retrieval.Zhang
and Izquierdo (2008) group the efforts of this community according to two main
classifications.The first classification is based on the features exploited for the
retrieval.The second classification is made based according to the different
retrieval paradigms existing in the literature.
Concerning the first classification,these research works imply that general
visual information representation schemes that are employed to design image
retrieval algorithms can be categorized into the following three classes:
 Textual feature-based.This is based on the written metadata available about
the image,and is concerned with the classical retrieval technology already
 Visual feature-based.The paradigm behind this is to represent images
or video clips using their low-level attributes,such as color,texture,
shape,sketch and spatial layout,motion,audio features,which can be
automatically extracted from the multimedia content themselves.There are
many examples of this preliminary approach in the literature.For instance,
the pioneer QBIC (Faloutsos et al.,1994) and the more recent PicHunter
(Cox et al.,2000).
 Combined textual-visual feature-based methods.Many researchers have
investigated the possibility of combining the text-based and content-based
retrieval.For instance,the iFind (Lu et al.,2000) system approaches
the problem first by constructing a semantic network on top of an image
database,and then using relevant feedback based on low-level features to
further improve the semantic network and update the weights linking images
with keywords.Other approaches (e.g.,the work of Shi and Manduchi
(2003);Zhang and Izquierdo (2007)) just treat each feature individually
and fuse the lists to obtain the final results.However,this is still a remain
challenge for the community.
Regarding the second classification,related to the retrieval paradigms,the
main categories are the following:
 Region based representations.According to the current state of the art
in image analysis,it is difficult to go beyond the extraction of middle level
features.And these middle level features usually are not referred to the
whole image,but just to a part of it.For that reason,many research works
seek to use a combination of regional descriptions to represent an image,
because it is much more feasible to link those middle level features (e.g.,
vegetation,sky) to regions.In our opinion a reference work in the literature
is the work performed by Papadopoulos et al.(2007) in the ITI institute in
 Fusion of multiple features.This category compiles those works related
to the joint exploitation of different features of multimedia content.The
motivation behind this thesis is that different features and their respective
similarity measures are not designed to be combined naturally and
straightforwardly in a meaningful way.A large number of different features
can be used to obtain content representations that could potentially capture
or describe semantically meaningful objects in images.The challenge in
this type of work is the appropriate selection of those features.Zhang and
Izquierdo (2007) provide a review of this kind of approaches.
 Probabilistic inference for context exploitation.The current imprecision of
the image analysis algorithms (Santini,2003) and the aim to approach to
the way the human brain behaves are the main motivation of this kind
of approaches,where statistical methods are employed to learn and train
algorithms.Popular techniques related to storing and enforcing high-level
information include neural networks,expert systems,fuzzy logic,decision
trees,static and dynamic Bayesian networks,factor graphs,Markov random
fields,etc.A comprehensive literature review on these topics can be found
in the work of Naphade and Huang (2002).
 User relevance feedback.These approaches make use of the last stage of a
retrieval process,to employ the users’ judgement to influence the previous
steps.By doing so,the retrieval systems accept the user as the central
actor,which implies accepting the users’ interactions with information as
the central process.Rui et al.(1998) and Crucianu et al.(2004) provide a
complete introduction and short survey to this kind of techniques.
Video retrieval
From the visual perspective,a video can be understood as a consecutive set
of images or frames.However,the techniques for the automatic extraction of
features out of a video have some peculiarities that have not been covered in the
previous section.In this section we cover some particular research challenges
and subfields that are particular of video retrieval.First of all,the shot boundary
detection or scene segmentation,that can be understood as a continuous
action in an image sequence (Han et al.,2000).This is one of the first steps
to be applied in video processing.As a consequence,a video is divided into
a set of sub-videos or shots.There are different types of transitions between
shots.Depending on the transition,there are more suitable techniques.Geetha
and Narayanan (2008) provide an introduction to six families of algorithms.In
general this is a well solved issue being the results on the shot boundary detection
competition of TRECVID conference very appealing (close to 100%of precision).
Once the shots have been identified,it is very suitable to extract the set of
most representative frames in the shot.These frame or frames are named Key-
frames.Once this is done most of the techniques mentioned for image retrieval
can be applied to that shot straightforwardly.The simplest techniques are static
(e.g.,select the central frame of the shot) but there are also very challenging
unsupervised approaches e.g.,(Hafner et al.,1995;Hauptmann et al.,2003)
that automatically select the most suitable key-frame according to the established
Another distinction with respect with the image analysis is the existence of
the spatio-temporal relationship.Some relevant works on this are devoted to
the extraction of motion descriptors (Smeulders et al.,2002) and temporal
texture(Ngo et al.,2003).
The content of the video,specially if it is long,can be organized using
clustering techniques.The similar shots or frames are grouped on the same
cluster,simplifying the analysis and understanding of the video.Here,again,
the approaches are multiple (e.g.,hierarchical clustering (Fan et al.,2004) and
spectral clustering (Chasanis et al.,2008) among others).
Regarding the indexing and retrieval techniques,the approaches are in
general similar to the image retrieval.However,we would like to mention some
innovative approaches that make use of the peculiarities of the video in order to
enhance the automatic annotation of the asset.Hanjalic (2005) has developed
a system for adaptive extraction of highlights from a sport video based on
excitement modeling.Feldmann et al.(2008) have employed the motion of the
camera for the automatic detection and modeling of flat surfaces.Vasconcelos
and Lippman (1997) integrated shot length along with global motion activity to
characterize the video stream with properties such as violence,sex or profanity.
Finally we should notice that the videos and the multimedia assets,are
composed by one or more essences or modalities.The research works for
indexing and retrieval of multimedia assets employ more and more the different
modalities present in the asset to extract as rich annotations as possible in what
is called multimodal analysis (Lai et al.,2002;Snoek et al.,2007;Wu et al.,
Audio retrieval
The retrieval of Audio retrieval is a very wide scientific field and is not fully aligned
with the scope of this thesis.Therefore,it is completely out of our expertise and in
our validation scenarios we have never tackled this issue.Since the audio is not
covered in the rest of this thesis work,we do not go into detail of this subject.As
an introduction to the field,we would like to name some of the multiple disciplines
and key references behind the audio retrieval.The challenges and the techniques
employed for the retrieval of music (Byrd and Crawford,2002;Ellis,2006;Klapuri,
2004),notated music (Hoos et al.,2001) or human spoken audio (Peinado,2006;
Rabiner and Juang,1993) are totally different.In the work of (Spanias et al.,2007)
and (Zoelzer,2008) the reader may find support for a deeper understanding of
the field.
2.2 Approaches for the semantic improvement in the
multimedia retrieval workflow
In this section we aim to cover the relevant work in the literature related to the
improvement of the retrieval of multimedia assets by employing semantic-aware
technologies but from a global perspective.This means that the work that we
present in this section does not employ semantic technologies to improve just
some aspect of the retrieval process but,in a similar way to our approach,aims
to tackle this issue from a broader perspective.
This is,to our vision,the closest related work to our approach.According
to this,we dedicate a different subsection for each of them.Each subsection
includes not only the description of the work,but also the main differences among
the work presented by the authors and the work we present here.
2.2.1 An intelligent media framework for Multimedia Content
Bürger (2006) and Günter et al.(2007) also face the gap between the
heterogeneity of the information and the users.They state that the main
motivation behind their work is that in the current multimedia management
systems,users are supported by a wide range of features which are traditionally
based on full text search and metadata queries.However,generating metadata
is an error-prune and work-intensive task,that for multimedia content cannot yet
be made fully automatically.In this context,they define the Intelligent Media
Framework (IMF) to formalize and manage the semantic connections across
the system,semi-automatic annotation tools to index multiple incoming streams,
information databases and audiovisual archives,and a recommender system to
analyse and visualise consumer feedback that is delivered over a back channel
system (Messina et al.,2006).This framework provides the following services to
the rest of the components of the system.
 Services to create,annotate and manage the intelligent media assets that
make up the show under real-time conditions.These services operate on a
metadata level and do not actually store any raw video streams (they rather
reference these so called essence).
 A service to manage and deliver information about the staged events (e.g.,
the schedule of the contests and races,the participating athletes,the
 Aservice to manage and deliver information on the way howa live broadcast
of a sporting event is presented (e.g.,which types of switching concepts are
available and used in a certain concept of a show,which basic dramaturgic
concepts are appropriate according to the disposition of the production team
and/or a predefined mood of the show).
 A service to access the vocabularies and the terms of the controlled
vocabulary constituting the knowledge base of the live staging domain.
 A messaging systemto support the real-time aspects of the staging process
by offering subscription methods to other subsystems.
<<Service Layer>>
IMF Services
<<Business Layer>>
IMF Components
<<Storage Layer>>
IMF Storage
Figure 2.3:Architecture of the Intelligent Media Framework Component
This set of objectives makes the Intelligent Media Framework and its
applications very close to the our semantic middleware and specially to the
deployment of the RUSHES system.Beside this,its architecture (see Figure
2.3 is also based on a combination of a classical three-tier architecture with the
principles of Service Oriented Architectures (SOA).The main responsibilities are
addressed in different layers:
 The Service Layer:In a similar way to the semantic middleware gateway
that we present in Section 3.2,the services layer consists of the services
provided by the IMF to the other building blocks of the whole system and
external systems.These building blocks include components responsible
for the manipulation,semantic enrichment and recommendation of data.
 The Business Layer:this layer is designed to interact with the services
and therefore is in charge of handling the data in specified incoming data
formats.This includes the data formats specified by the IMF as well as
standard data formats such as MPEG-7 or NewsML.
 The Storage Layer:This layer is responsible for the transformation of data
into the data formats specified by the IMF data model and secondly for the
provision of a persistence layer for the whole system.
The lower the layer,the more differences are detected with respect to the
approach we present.The IMF relies on the storage layer for the transformation
of the data into the data formats specified by the IMF data model.This implies
that the storage of the semantic model (passive role) is not the main mission
of the layer but the transformation of the external data into a kind of internal
representation according to a model (active role).This is due to the fact that the
IMF relies on a specific data format to provide the mentioned semantic services.
In this approach,the assets are not just a passive representation of information
but complex objects.Those complex objects are called by the author Knowledge
© Salzburg Research 200
Figure 2.4:Model of the Knowledge Content Objects of the IMS
Content Objects and their model can be seen in Figure 2.4.The core parts of this
model are content annotations which provide information about the essence (i.e.
the raw video stream) and subject matter annotations which provide information
about the subject matter of the essence.
To summarize,the IMF,in the context of the work we present here,can be
understood as a kind of semantic middleware which is designed to work in a
specific environment where the multimedia assets are mapped into a new set
of multimedia assets that are able to perform by themselves some semantic
operations.This fact and the consequences in the design and implementation
derived from it are the main differences with this work.In our opinion,this
approach is not compatible with the motivation behind the work we present here.
The main reasons for this are the performance and cost consequences of such
replication and the techniques employed for the storage and management of the
Knowledge Content Objects.
2.2.2 Information Mediation Layer:a newcomponent for the digital libraries
Candela et al.(2006),in the context of a larger effort dedicated to the definition
of a reference model for the digital library (Candela et al.,2007),describe
the motivation and scope of the introduction of a new layer:“The Information
Mediation Layer”.
Their work relies on the idea that Digital Libraries are often built by exploiting
Figure 2.5:Reference Model for the Digital Libraries
already existing resources.According to them,the most frequently shared
resources are the documents of the archives,but many other type of resources,
such as authority files
,thesauri,language dependent resources,ontologies,
classification systems
,and gazetteers
.Those resources are mainly created
by third-parties and are heterogenous.
The need of handling this heterogeneity is the main motivation of their work,
the information mediation layer which is graphically summarized in Figure 2.5.
This layer implements the services required for the provision of virtual views
of what they name information spaces.The main idea is to improve the access to
the information by homogenizing it.The mediators that compose this area may
be classified according to the following:
 Information organization:These mediators are related to the semantic
representation of the information organization aspects.They may be related
to the problem of the heterogeneity (e.g.,the provider of a virtual view of
the information object model,which is able to provide information about
the multiple object manifestations,the object composition) or the problems
raised by large volumes of data (e.g.,the provider of a virtual collection view,
which is able to organize the information space in multiple sets of objects,
each capable to meet a different need).
Figure 2.6:Query Decomposition process
 Object manifestation:This kind of mediators provides a manifestation view.
The manifestation is the way through which the content of an information
object is perceived by the user.The functionalities provided by the services
of this area are:(i) to access the manifestation while hiding details about its
storage,and (ii) the dynamic generation of alternative and more profitable
manifestation formats.
 Metadata object manifestation:This class of mediators provides a metadata
view.The functionalities provided by the services of this area are:(i) the
presentation of the metadata in a required format,and (ii) the dynamic
generation of new metadata.
Thus,the Information Mediator Layer has a number of services that implement
the corresponding mediation functionality.Some of these services are mandatory
in any Digital Library (DL) system while others depend on the specific DL
application area.
Regarding the contextualizing of Candela et al.(2006) contribution with
respect to this thesis,their approach is in general very similar.However,the
lack of technical details of the mediators or information about any implementation
makes difficult the identification of the similarities and differences.
2.2.3 A model for multimedia information retrieval
The work of Meghini et al.(2001) is a very remarkable contribution in the literature
that handles the problem from a generic perspective.This theoretic work results
in a conceptual model that,according to the authors,encompasses in a unified
and coherent perspective the many efforts that are being produced under the
label of MIR.
The model is formulated in terms of a fuzzy description logic,which plays
a twofold role:(i) it directly models semantics-based retrieval,and (ii) it offers
an ideal framework for the integration of the multimedia and multidimensional
aspects of retrieval mentioned above.This scope is the reason because we have
included Meghini work in this section,in spite of the fact that the nature of this
work,as we do state later,is intrinsically different.Figure 2.6 aims to graphically
summarize the approach followed by the model to address the query processing.
The model presents a decomposition technique that reduces query evaluation
to the processing of simpler requests,each of which can be solved by means
of widely known methods for text and image retrieval,and semantic processing.
Each of the steps in the process has been expressed mathematically according to
the mentioned fuzzy description logic.Therefore the semantics,the current state
of the art in multimedia querying and the peculiarities of the multimedia retrieval
have been taken into account by the authors over the whole process.
This model shares our motivation of bringing semantics to MIR systems.
However,while our aim is to support system managers to complete their current
facilities,the scope of this model is to define guidelines for the design of systems
that are able to provide a generalized retrieval service.This fact and the nature of
the model are the most significant differences with respect to the work we present
2.2.4 A three layer infomediation architecture
Kerschberg and Weishar (2000) in their article about Conceptual Models and
Architectures for Advanced Information Systems present an approach about
how conceptual modeling of information resources can be used to integrate
information obtained from multiple data sources,including both internal and
external data.
Their work is based on a three-layer Reference Architecture consisting of
various types of mediation services,including facilitation and brokerage services,
Materialized and
Mediated View of
Processing and
Object Specification
Constraint Management
Information Integration
Data Quality
Data Lineage
Data Sources
Text Images SimluationsText Images
Figure 2.7:Layered Information Architecture & Processes
mediation and integration services,and wrapping and data access services.
Although their work is domain agnostic,in Figure 2.7 a particularization for the
logistics domain is shown.
The upper layer Information Interface layer,is the layer in charge of providing
the users with the available information.This layer must support scalable
organizatino,browsing and search.Some of the services provided by that layer
are the intelligent thesaurus or yellow pages.
The intermediate layer Information Management layer,is responsible for the
semantic integration,replication and catching of the information gathered by all
the information sources.
Finally,the bottom layer Information Gathering layer is responsible for
collecting and correlating the information frommany incomplete,inconsistent,and
heterogeneous repositories.
This short summary is enough to understand the main differences of this
approach with this thesis work.On the one hand,the focus of their work is
devoted in information integration.On the other hand,their approach relies on the
adaptation and replication of that information,instead of the provision of semantic
services (e.g.,terminological mapping,negotiation resources) to a main system
in order to perform searches over external repositories.
Query UI
annotation links
List of instances
Query UI
annotation links
List of instances
Figure 2.8:View on ontology-based information retrieval
2.2.5 Ontology Based Information retrieval
The work of Castells et al.(2007) is a relevant example of what can be understood
as ontology based retrieval system.Their proposal is a retrieval model meant
for the exploitation of full-fledged domain ontologies and knowledge bases,
supporting semantic search in document repositories.Castells et al.(2007)
in a coherent view with their understanding of semantic information retrieval
(see Figure 2.8),assume that each information source includes a knowledge
base (KB) which was built using one or several domain ontologies that describe
concepts appearing in the document text.The concepts and instances in the KB
are linked to the documents by means of explicit,non-embedded annotations to
the documents.
This work and its promising results shares the motivation of this thesis work
and provides semantic services over different steps of the multimedia retrieval
(e.g.,query processing,result ranking).However,it relies on a very specific
particularization:the existence of a knowledge base expressed in an XML
format (in this example RDF or Resource Definition Framework) for each of the
information sources.This is the main difference with this thesis work and is clearly
against our will to facilitate the integration of the semantic techniques in current
multimedia asset management systems.
2.2.6 Ontology-enriched semantic space for Video Search
Wei and Ngo (2007) share our aim of diminishing the semantic gap between
the low-level features available for the multimedia assets due to the analysis
algorithms and the high-level features demanded by the users.They propose
a novel model,namely Ontology-enriched Semantic Space (OSS),to provide
a computable platform for modeling and reasoning concepts in a linear space.
According to the authors,OSS enlightens the possibility of answering conceptual
questions such as a high coverage of semantic space with minimal set of
concepts,and the set of concepts to be developed for video search.
The basis of their work is a simplification,in the sense of performance and
computational resources consumed,of the comparison of concept pairs.The
OSS is composed by a semantic space that is linearly constructed to model the
available set of concepts.The expressive power of OSS is linguistically spanned
with a set of basis concepts,which is easier to generalize,not only to the available
concept detectors but also to the unseen concepts.
The main implications of this simplifications are the following:
 Query disambiguation:OSS facilitates the interpretation of the terms of the
query of the user.
 Query Concept Mapping:The comparison between the concepts is done
by ensuring the global consistency.
 Multi modality fusion:OSS is a key element for the generation of concept
clusters,and the authors demonstrate that those clusters allow to effectively
fuse the outcomes of concept based retrieval (visual) and text based
retrieval (keywords).
 Scalability:OSS facilitates the selection of concept detectors (e.g.,face
recognition) that result to be more useful for query answering in a domain.
We have included the work of Wei and Ngo in this section not because we
consider that it can be understood as a pure semantic middleware in a retrieval
system but because its defines a semantic infrastructure generic enough to be
employed to support several steps of the information retrieval process.The
authors do not focus of the integration of their technology into existing systems.
However,they describe their work as a useful semantic resources to empower
the analysis of the content,the implementation of navigation mechanisms (e.g.,
cluster construction) and the mapping of the queries into the internal vocabulary.
2.2.7 MPEG-7 driven multimedia retrieval
While there are profuse bibliography related to the development and use of the
MPEG-7 standard (Dasiopoulou et al.(2010) provide an extensive state of the
art just on MPEG-7 ontologies),it is not easy to find relevant references that
employ the MPEG-7 to bo beyond the building of a specific solution,generating a
framework or whole retrieval system.
In that scare bibliography references,the work of Schallauer et al.(2006),
a Description Infrastructure for Audiovisual Media Processing Based on MPEG-
7 and some complementary reports of the same authors Bailer and Schallauer
(2006);Bailer et al.(2007) are,from the perspective of this thesis,remarkable
Schallauer et al.(2006) work tackles,from a generic perspective,a big set
of aspects related to one of the key steps of the multimedia retrieval process,
the multimedia processing.According to this,their system is able to import
audiovisual data into the system and to perform and control automatic content-
analysis tools which extract a number of low- and mid-level metadata.But going
beyond that,as Figure 2.9 reflects,their contribution includes also the following
 A Manual Documentation Component used for textual descriptions and
description of high-level semantic information,which cannot be extracted
 A Search component for query formulation and result presentation,which
provides search options for both textual and content-based queries.
 A backend infrastructure providing storage and search functionalities
As a result of this,they propose a complete open (MPEG-7 based) multimedia
retrieval system that has been designed taking into account the difficulties and
peculiarities of the multimedia indexing.
This work shares the objective of the thesis work we present here but it
presents a difference:it does not complement a system,in fact it implements
a new one.However,if we focus on the multimedia processing component,for
one of the deployments that we present here,we share not only the approach but
also the ontology employed,the DAVP profile of MPEG-7.While Schallauer et al.
(2006) employ this ontology and a complete query,search and storage machinery
for that ontology,this thesis is more generic.Even for the case that part of our
middleware is composed by MPEG-7 ontologies,the storage and query facilities
are shared with the rest of the ontologies present in the middleware.
MPEG-7 Repository
Formulation &
Presentati on
media-anal yze
Medi a
Reposi tory
Essence &
Annotati on
medi a-summary
with XML Support
for CB
References to Essence
Figure 2.9:Systemoverview of the infrastructure components for multimedia description
2.3 Metadata models for multimedia
Bailer and Schallauer (2008) provide an overview of the role of the metadata in
the audiovisual media production process.The state one premise fully shared
with us:although there are multiple multimedia metadata standards,no single
standard fulfills all requirements required in complex real live applications.Both,
the middleware that we present in Section 3.2 and its deployments (see Chapters
5 and 6) rely on this assertion.In this section we provide an overview of relevant
references in the field of metadata models for the management (i.e.indexing,
processing,searching and so on) of multimedia assets.
In order to facilitate the comprehension of the different standards,their
differences and complementarities,we provide a summary of the different types
of metadata.
2.3.1 Types of multimedia metadata
There are many different types of metadata (Cox et al.,2006;Smith and Schirling,
2006).Not all of them are involved in the search process.However,the
digitalization has led to a system convergence in the companies and in order
to enhance the retrieval systems with content-based features,usually all the
metadata requirements of the company have to be taken into account.
Following the approach of the researchers of the Joanneum Research
Institution Bailer and Schallauer (2008),we can classify the metadata according
to three main parameters:the source of the metadata and its properties.
Types of metadata according to the source
 Capture.The capture metadata is mainly related to the technical description
of the asset and is created together with the asset.Some examples of
this metadata are the DMS-1 (SMPTE 380M-2004 - Descriptive Metadata
Scheme - 1),annotations provided by some broadcast cameras,the Exif
(Exchangeable image file format) information and so on.
 Legacy and Related Information.This metadata,sometimes generated
even before the assets itself,makes reference to the legal aspects of the
assets (e.g.production contracts) and audiovisual material that is related to
the asset (e.g.interview to the creator).
 Manual Annotation This metadata is very rich from the semantic point of
view,but very costly.In a professional environment,this information is
reliable and valuable.
 Content Analysis This source is derived from automatic analysis of the
content in order to extract metadata describing it.This metadata can be
related to very low level features (e.g.histograms of a key-frame),middle
level features (e.g.face identification) or high level features (e.g.face
recognition).The problem of extracting semantics from the low and middle
level features is known as the semantic gap (Santini and Jain,1998) and
is still not satisfactorily solved for open domains (Hauptmann et al.,2007).
This type of metadata is therefore more unprecise,but extremely cheaper
than the produced by the manual annotation.
 Text and semantic analysis.This includes recognition of references to named
entities (e.g.persons,organisations,places) as well as linking them to
ontological entities,the detection of topics and the classification of content
segments and linking content to legacy or related information.
Types of metadata according to the properties The nature of an asset
is usually complex.In a professional environment,an asset is composed of
different essences (e.g.several audio tracks,subtitles and so on).Each essence
consist on a dynamic representation of information that usually changes over the
time.Taken this into account,we can distinguish between the following types of
metadata according to its properties.
 Scope.A metadata unit may refer to the whole asset or just a segment of
one of the components of the assets.It can apply to a spatial,temporal or
spatiotemporal segment of the content.The same metadata elements may
exist in different scopes,such as the title of a movie and the title of a scene.
 Data type.The datatypes of the metadata may be diverse.First of all it
can be either textual or numerical.The textual metadata can be free text,
discrete set of values (e.g.thesauri,ontologies).The numerical metadata
can be composed of integer numbers,vectors,and so on.
 Time Dependency.Some metadata changes in the time (i.e.dynamic
metadata) while other pieces metadata are not altered (i.e.static metadata).
 Spatial dependency.This is the same that in the previous case but for the
spatial component.
 Modality channel dependency.Some metadata affects the whole asset
while some affects fully or partially just to one of the modalities of the asset
 Context dependency.There are metadata that depend highly in the context
in order to provide a meaningful interpretation.For instance,classifying a
segment as “frightening” is fully context dependent.
2.3.2 EBU P/Meta
The EBUor European Broadcasting Union self-defines as “the largest association
of national broadcasters in the world,built to promote cooperation between
broadcasters and facilitate the exchange of audiovisual content”.This has had
an impact in the work they have done regarding the metadata models and
schemas.This work has been mainly focused on the exchange of metadata.
This activity started in 1999,based on other works already in progress at the
British Broadcasting Corporation (BBC) on the schema SMEF (Standard Media
Exchange Format) and the RAI (Radio Televisione Italiana).
This work,tagged as P/Meta (EBU-Technical-Department,2001),is a flat list
of metadata entries focused on the commercial programme exchanged between
broadcasters.P/Meta defines syntactical rules that must be followed when the
generation of the metadata is done.
From the technological point of view,P/Meta does not constrain any
implementation,since it does not go beyond the definition of the terms.It
can be “materialized” as XML documents or Word docs or embedded in MXF
(SMPTE 377-1-2009 Material Exchange Format).As other schemas,it uses
numerical codes for attributes and standard values.This facilitates the machine
manipulation and the multilingual aspect.
Figure 2.10:BMF root nodes
Figure 2.11:TVAnytime Metadata Model framework
2.3.3 Standard Media Exchange Format - SMEF
SMEF (BBC,2000) is a standard for metadata modeling defined by the British
Broadcast Corporation.It covers the indexing of the assets from a very wide
perspective,going from the asset itself (media object) to the shot level and
the editorial objects (programmes).While P Meta was mainly defined for the
exchange,SMEF was defined for internal usage in the corporation.
2.3.4 Broadcast Exchange Metadata format - BMF
The Institut für Rundfunktechnik GmbH (IRT) has developed the Broadcast
exchange Metadata Format (BMF) that defines an uniform,generic model for