Diapositiva 1

roomagitatedInternet and Web Development

Dec 8, 2013 (3 years and 8 months ago)

79 views

M
-
Advantage


M
ultimedia
-

A
utomatic
D
igital
V
ideo &
A
udio
N
etwork
T
hrough
A
dvanced Publishin
g

E
uropean Service



Project proposal

Author: Andrea de Polo on the behalf of the M
-
ADVANTAGE Consortium

Presenter: Mari Partio, TUT, Finland

27th of August, 2005

IASW 2005

2

Outline


Motivation


State of the Art


Scientific and Technological Objectives


Outline Implementation Plan


Consortium


Conclusions


References

27th of August, 2005

IASW 2005

3

Motivation


Aim is to increase the competitiveness of Europe’s digital
content industries by semantic services across the
content value chain


Target is to build a service infrastructure for automated
semantic discovery, extraction, summarization, labeling,
composition and personalized delivery of content from
heterogeneous multimedia repositories


The project also involves merging multiple
heterogeneous datatypes into an integral representation


Goal is to make use of Semantic Web, multimedia
description and other standards to enable a broad
uptake of M
-
ADVANTAGE’s open source and non
proprietary technologies

27th of August, 2005

IASW 2005

4

Relation to State of the Art


Tasks, scientific and technological objectives of
M
-
ADVANTAGE can be grouped under
intelligent multimedia analysis and access with
the use of ontological information.


Thus, the most relevant state of the art is that
related to the development of ontological
knowledge representations for multimedia
applications as well as those related to
multimedia analysis and access approaches.

27th of August, 2005

IASW 2005

5

Description of Multimedia Content


As far as the representation is concerned, the MPEG
-
7
standard provides a rich set of standardized tools to
describe multimedia content.


To make MPEG
-
7 accessible, re
-
usable and
interoperable in many domains, the semantics of the
MPEG
-
7 metadata should be expressed in an ontology
using a machine
-
understandable language


Additionally there is an increasing need to allow some
degree of machine interpretation of multimedia
information’s meaning.


27th of August, 2005

IASW 2005

6

Representing Multimedia Ontology


In [1] Hunter represents multimedia ontology in RDF Schema and
demonstrates how this ontology can be exploited and reused by other
communities on the semantic web.


First basic multimedia entities and then their hierarchies from the
MPEG
-
7 Multimedia Description Scheme (MDS) basic entities are
determined


The RDF schema semantic definitions for MPEG
-
7 can be linked to
their corresponding pre
-
existing MPEG
-
7 XML schema definitions.


Additionally, the RDF Schema can be merged with RDF schemas
from other domains to generate a single "super
-
ontology" called
MetaNet.


This super
-
ontology can be used to enable the co
-
existence of
interoperability, extensibility and diversity within metadata descriptions
generated by integrating metadata terms from different domains.


The proposed method for building a multimedia ontology has been
applied to manage the manufacturing, performance and image data
captured from fuel cell components [8,9].



27th of August, 2005

IASW 2005

7

Using OWL Ontology


Future work plan of [1] includes the automatic semantic extraction from the
MPEG
-
7 XML schema document as well as linking of the semantics to the XML
schema document.


To couple with domain
-
specific and low
-
level description vocabularies, a similar
methodology for enabling interoperability of OWL domain
-
specific ontologies
with the complete MPEG
-
7 MDS is described in [10].


The approach is based on an OWL ontology, referred as a core ontology, which
fully captures the MPEG
-
7 MDS.


For the development of the core ontology, a set of rules is defined to map
particular MPEG
-
7 components to OWL statements.


The integration of the domain
-
specific knowledge is performed by considering
the domain specific ontologies as comprising the second layer of the semantic
metadata model used in the DS
-
MIRF framework.


Additionally, rules are provided for transforming the OWL/RDF metadata,
structured according to the core ontology and the domain
-
specific ontologies,
into MPEG
-
7 compliant metadata.


Following this approach proves advantageous for MPEG
-
7
-
based multimedia
content services, such as search and filtering services, since incorporating
semantics can lead to more accurate and meaningful results in terms of meeting
the user queries.


27th of August, 2005

IASW 2005

8

Semantic indexing


Semantic indexing aims at finding ”patterns” in
unstructured data and use these patterns to offer
more effective search and categorization
services [12]


Language independent


In order to understand multimedia content, one
necessary step is to identify objects within it
(similarity search on image parts immersed into
various contexts).





27th of August, 2005

IASW 2005

9

Semantic multimedia retrieval


Semantic multimedia retrieval requires the
presence of already annotated multimedia
content


Three types of semantic retrieval:


Direct description of semantic track (significance of
semantic features should be specified)


Defining semantic track of the target data by giving
same type of item as query (sample media file with a
set of extracted mathematical features)


Combination of the two earlier methods



27th of August, 2005

IASW 2005

10

Bayesian Inference


A mathematical technique for modeling the
significance of semantic concepts based of how
they occur in conjunction with other concepts.


Helps to extract the key conceptual aspects of
any piece of unstructural information

27th of August, 2005

IASW 2005

11

Pattern
-
matching technology


Information theory provides a mechanism for
being able to extract the most meaningful ideas
in documents
-
> pattern matching


Shannon’s theory: ”the less frequently a unit of
communication occurs, the more information it
conveys”


Pattern
-
matching approach has the additional
benefits:


robust to false positive matches


it can determine how similar documents are


27th of August, 2005

IASW 2005

12

Scientific & Technological Objectives


The M
-
ADVANTAGE project aims at developing an infrastructure capable of
delivering multimedia information and content customized to the needs of
end
-
users.


It focuses on building some specific components to provide the
functionalities necessary to facilitate the construction of advanced
multimedia content applications and the use of structured and unstructured
multimedia information.


The goal of the M
-
ADVANTAGE approach to the “delivering multimedia
information and content customized to the needs of end
-
users” is based on
three ambitious deliverables:


M
-
ADVANTAGE is able to automatically integrate heterogeneous multimedia
content.


360
°

Technology Approach: M
-
ADVANTAGE infrastructure is based on the more
up
-
to
-
date technology approaches for managing unstructured information:
Keyword, Semantic and Statistical (through a pattern matching system).


Develop specific application services to deliver the content managed by the M
-
ADVANTAGE back
-
end infrastructure


27th of August, 2005

IASW 2005

13


These features will enable the utilization of
digital content delivery systems distributed
across the computer network and will process
the information stored within these archives in
order to find dependencies, links and similarities
between various pieces of information.


It will also allow to automatically manage and
customize the available content for the needs of
end
-
user applications built on top of the M
-
ADVANTAGE infrastructure


Scientific & Technological Objectives

27th of August, 2005

IASW 2005

14

Scientific objectives


Automated (semantic) multimedia discovery, which concerns both retrieval,
i.e. search for multimedia files; and extraction, i.e. more focused search for
specific structural components of the multimedia: episodes, frames, images
(focuses), etc.


Advanced video summarization, i.e. general idea of the video content can
be obtained quickly.


Advanced techniques for semantic labeling, i.e. propagation of labels
through hierarchical database structures.


Automated multimedia integration / composition: real power is in
composition of different structural elements (episodes, frames, focuses)
extracted from heterogeneous multimedia files in a coherent track.


Semantic personalized delivery: based on semantic interactions of user
activities / actions on content and user's explicit preferences; proactive
supply to the user of relevant multimedia.


Interoperability between heterogeneous (web
-
) services and multimedia:
this is possible following Semantic Web's recommendations about common
(upper
-
) ontology or managing mapping between semantic concepts from
different ontologies.


27th of August, 2005

IASW 2005

15

Technical objectives


M
-
ADVANTAGE aims at creating a state
-
of
-
the
-
art cutting edge
technology that is going to serve public and business sector in the
knowledge management for multimedia. In that respect:


To enhance search behavior statistical search will be used as a super
set of the conventional methods to grasp concepts embedded in
images, text and videos


The combination of semantic / ontology methodologies and the
statistical one will offer users the possibility to have a much more
precise and to the point interaction with the KB. Users will be profiled
and grouped into communities according to their previous interactions
with the KB.


Splitting content into its fundamental parts (scene detection, object
extraction)


Utilization of speech from video and the most advanced speech to text
technology to search for the most meaningful frames related to search
argument

27th of August, 2005

IASW 2005

16

Outline Implementation Plan


M
-
ADVANTAGE platform intends to provide an
integrated solution for the B2B value chain starting from
the content owners passing through the added
-
value
content creators and arriving to the service providers


Market standard networks and devices are planned to be
used by the final users, accessing the services created
within the M
-
ADVANTAGE platform.


This can be broken down into the segmentations
described in Figure 1 representing a more in depth view
of the content value chain that M
-
ADVANTAGE intends
to address.

27th of August, 2005

IASW 2005

17

M
-
ADVANTAGE Services

Multimedia

Repository

Building blocks

of

Contextualised

Information

import

delivery

delivery

HW / SW

for MM

Displays

HW / SW

Technology for

Information

Delivery QoS,




Tools for

Authoring and

Value Adding

Content

Management

Systems

Content

Information

Design

Mobile Phone

Mobile Phone

PDA

PDA

Web Browser

Web Browser

Goe

-

ref

.

Data

Goe

-

ref

.

Data

MM

Assets

MM

Assets

Metadata

Metadata

place

subject

time

Tools for

Information

Acquisition,

Filtering,

Importing, etc.

(Technical point


of view)

27th of August, 2005

IASW 2005

18

M
-
ADVANTAGE Services


(User’s point


of view)

Audio/Video
content


Audio/Video
users


Content
-
Based
Indexing

Audio/Video
Database

Ontology
-
Based
Semantic
Indexing

Audio/Video
expert


Manual
annotation

Semantically
Enhanced
Content

MFO & SFO


Ontologies

Intelligent
wizard


Ontology
-
Based
Semantic
Tracking

User’s Semantic
Blog

User Profile
Ontology

Proactive
behaviour

Semantics &
content mining
tools & services


External
Ontology

Semantic
enrichment

Learning

Automated
annotation

27th of August, 2005

IASW 2005

19

M
-
ADVANTAGE’s Improvements to the
Industrial Processes:


Provides access to a larger amount of multimedia
information


Simplifies search activities


Offers an integrated Digital Rights Management System
(DRMS)


Provides a Customized Multi Licensee Service


Offers a secure online payment mechanism


Pay per View


Automatic tools to enrich ”poor” or unclassified items
with enriched multimedia features


Offering customized and personalized search
environments

27th of August, 2005

IASW 2005

20

M
-
ADVANTAGE Platform


M
-
ADVANTAGE platform is intended to be a
basis for a wide set of tools, which satisfy
different needs of different actors:


different business approaches


different technological situations


different vocations


To satisfy all these aspects, M
-
ADVANTAGE
needs to integrate and/or develop wide range of
tools and services, as briefly summarized in
following figure

27th of August, 2005

IASW 2005

21

Business services included in M
-
ADVANTAGE

27th of August, 2005

IASW 2005

22

M
-
ADVANTAGE Architecture

User

Raw Media

Content

Storage

Systems

Unification

Raw Media

Content

Storage

Systems

Unification

Automated

Annotation

Collection

Management

User

Monitoring


Web

Interface

Access

Mechanisms


User





Profiles

Knowledge





Base

Indices

Ontological Information

User


Monitoring

A

B

C

D



User

27th of August, 2005

IASW 2005

23

Development component 1


One of the main objectives is to enable the access and
consideration of heterogeneous archives


Thus the first service is the analysis of existing content
storage systems and the specification of a generic
querying and access interface capable to support and
serve all existing content


Based on this generic interface, it will be possible to
create software interfaces, custom to each archive,
which allow for the automatic connection of the archive
with the overall M
-
ADVANTAGE system

27th of August, 2005

IASW 2005

24

Development component 2


Most important and challenging objective of M
-
ADVANTAGE is to contribute to the effort to
bridge the semantic gap (knowledge based
approaches to semi
-
automatic and fully
automatic media annotation)


Complex ontological data models will be
developed


Various methodologies will be utilized (manual
annotation to semiautomatic, retainable and
adaptive computer assisted annotation and fully
automated knowledge driven annotation)



27th of August, 2005

IASW 2005

25

Development component 3


The meta
-
publication description format used to store
the analysis results in the knowledge base will also be
designed so that it provides optimal balance between
effectiveness and efficiency for storage and consequent
processes.


While the aim is to provide fundamental measurements
of the collection and properties, M
-
ADVANTAGE will
also address the issue in concrete terms of clustering
and informative sampling


The aim is to instantiate the concept of collection guiding
that extends classical browsing by creating exploration
strategies around the document collection and therefore
literally guide through it.


27th of August, 2005

IASW 2005

26

Development component 4


M
-
ADVANTAGE aims to offer innovative, intelligent,
personalized multimedia search and access services to
end users


State of the art content management system will be
integrated in the overall platform, allowing for simple,
semantic and statistical search; in all of these
approaches, knowledge contained in ontological
databases will also be considered


User interactions will be analyzed in order to extract user
profiles that can be fed back into the system thus
enhancing the quality of services offered to each specific
user

27th of August, 2005

IASW 2005

27

Consortium

Partner

name

Partner

type

Fratelli Alinari IDEA SpA (
Alinari
)

SME

Autonomy

Industrial (Private Commercial Organization, Ltd.)

Viper group, CVML, University of Geneva (
UniGE)

University

National Technical University of Athens (
NTUA
)

University

Poznan Supercomputing and Networking Center (
PSNC
)

Research centre, part of Polish Academy of
Sciences

Tampere University of Technology (
TUT
)

University

Ansa

Cooperative of newspapers

Contentmine International AG (
Contentmine
)

SME

Getty Images (
Getty
)

Industrial (Private Commercial Organization)

Italian State Library of modern and contemporary history (
BSMC
)

National Governmental Institution

the International Centre for Information Management Systems
and Services (
ICIMSS
)

Private non profit Organization

MENON

European Economic Interest Group (EEIG)

Salzburg NewMediaLab (
SNML
)

University

SWORD IT Solutions S.A. (
SWORD
)

Industrial (Private Commercial Organization, S.A.)

X
-
ART

SME (Private Commercial Organization)

27th of August, 2005

IASW 2005

28

Conclusions


The aim of M
-
ADVANTAGE is to deliver a first
version of infrastructure capable of delivering
multimedia information content customized to
user’s needs


It will develop new formal models for knowledge
representation with major focus being placed on
multimedia ontological knowledge presentation


It will generate an ontology infrastructure
containing all the knowledge needed for the
analysis in the main three ontologies: MFO, SFO
and UPO


27th of August, 2005

IASW 2005

29

Conclusions


A formal data model for integration of diverse
multimedia content (meta
-
publication) will be
designed


New tools to support automatic analysis,
annotation, filtering and visualization of
multimedia content will be provided to that extent
it is possible


27th of August, 2005

IASW 2005

30

References

[1]

J. Hunter, "Adding Multimedia to the Semantic Web
-

Building an MPEG
-
7Ontology", International



Semantic Web Working Symposium (SWWS), Stanford,July 30
-

August 1, 2001

[2]

RDF Schema Specification 1.0, W3C Candidate Recommendation 27 March 2000.
http://www.w3.org/TR/rdf
-
schema/

[3]

TV
-
Anytime Forum, http://www.tv
-
anytime.org/

[4]

MPEG
-
21 Multimedia Framework, http://www.cselt.it/mpeg/public/mpeg
-
21_pdtr.zip

[5]

NewsML http://www.newsml.org/

[6]

ISO/IEC 15938
-
5 FCD Information Technology
-

Multimedia Content Description Interface
-

Part 5:
Multimedia Description Schemes, March 2001, Singapore

[7]

DAML+OIL Revised Language Specification, March 2001. http://www.daml.org/2001/03/daml+oil
-
index

[8]

J. Hunter, J. Drennan, S. Little "Realizing the Hydrogen Economy through Semantic Web
Technologies", IEEE Intelligent Systems Journal
-

Special Issue on eScience, January 2004

[9]

Little, S., and Hunter, J., “Rules
-
B
-
Example


a Novel Approach to Semantic Indexing and Querying
of Images”, In 3rd International Semantic Web Conference (ISWC2004), Hiroshima, Japan,
November 2004.

[10] Tsinaraki, C., Polydoros, P., Christodoulakis, S., “Interoperability support for Ontology
-
based Video
Retrieval Applications”, In Proceedings of Third International Conference on Image and Video
Retrieval (CIVR), Dublin, Ireland, July 21
-
23, pp 582
-
591, 2004.

[11] Baeza
-
Yates, R.A., Ribeiro
-
Neto, B.A. (1999) Modern Information Retrieval. ACM Press / Addison
-
Wesley.

[12] Zhao, R., W.I. Grosky (2002) Narrowing the Semantic Gap
-
Improved Text
-
Based Web Document
Retrieval Using Visual Features. IEEE Transactions on Multimedia 4(2).