Android Multimedia Overview

gatecircleMobile - Wireless

Dec 10, 2013 (3 years and 6 months ago)

252 views

Android

Multimedia
Overview




February 11th 2009

David Gottardo, System SW Architecture at Texas Instruments France

Email: d
-
gottardo@ti.com


Android

Architecture


Table of Contents



Audio Flinger: Audio manager


Surface Flinger: Surface manager /
composite


Multimedia Framework


MM Playback


(MM Record)


Android

Components

Red border is corresponding to components not supported by TI SW foundation (Modem vendors, etc.)

WLAN
GPS
USB
BT
Vibrator
R
-
Touch
NAND
MMC
/
SD
Keyboard
Camera
Display
TV Out
Peripherals
Peripherals
Core HW Platform

OMAP
3630
ARM Cortex A
8
TWL

5030
SDRAM
Linux Kernels
Bluetooth
Driver
Camera
Driver
Power
Management
WiFi Driver
Keypad
Driver
USB Driver
Sensors
Driver
Led Driver
GPS Driver
Flashlight
Driver
Android Hardware Abstraction Layer
Graphics
Audio
Camera
Bluetooth
WiFi
Flashlight
GPS
Led
Sensors
Mount
Service
Power
Alarm
&
RTC
Logging
Telephony
Radio
Layer
Battery
3430
Machine
Support
MM ARM
Codecs
MM DSP
Codecs
Applications
LED
Touch Driver
C
-
Touch
IVA
2
+
DSP
MM ARM
Codecs
MM DSP
Codecs
Nand
support
Backlight
FM Radio
UART
UART
Driver
Backlight
Driver
FM Radio
Driver
Audio Driver
NL
5500
WL
1271
MM Audio
Battery
Flash
SPI
SPI Driver
Libraries
Surface Flinger
OpenGL
/
ES
SGL
Media
Framework
FreeType
SSL
SQLite
WbKit
Libc
Android Runtime
Dalvik Virtual
Machine
Core Library
Vibrator
Home
Contacs
Dailer
Voice Dail
SMS
/
MMS
Email
IM
Calendar
Browser
Media Player
Camera
Albums
Alarm
Clock
Calculator
Etc
.
C
64
x DSP
Binder
(
IPC
)
Driver
Application Framework
Content
Providers
View
System
Resource
Manager
Location
Manager
Etc
.
RIL Library
System Services
Activity
Manager
Window
Manager
Package
Manager
Notification
Manager
Telephony
Manager
App
launcher
Status Bar
GFX
(
SGX
)
Driver
Display
Driver
Audio Flinger
Media
Android

Surface

Manager

Client/Server Architecture


Android graphics system is based on a Client/Server Architecture


Server side: Surface Flinger is a
service coded in C++


Manages access to display subsystem


Seamlessly composites 2D and 3D graphic layers from multiple
applications


Client side: applications can draw on surfaces using two paths:


Use widgets/View/Canvas provided by the JAVA APIs and managed
by C++ external Skia library


Access OpenGL through JNI to directly create 3D surfaces

OpenGL

Java Native
Interface

2D Apps / 3D Apps

Java View/

Widget/

Canvas

Surface

SKIA (SGL)

SurfaceFlinger


Server side native library (C++):
frameworks/base/libs/surfaceflinger/


Provides system
-
wide surface “composer”, handling all surface
rendering to frame buffer device.


Can combine 2D and 3D surfaces (Z
-
buffer) from multiple
Applications


Can use OpenGL ES and 2D hardware accelerator for its
compositions.


Double
-
buffering using page
-
flip.


Surfaces passed as buffers via Binder IPC calls (Android’s interprocess
communication pipe).

SurfaceFlinger

(libsurfaceflinger)

Frame

Buffer

3D Surface

2D Surface

3D Surface

OpenGL ES (HW or SW)

Surface Management


Android apps are built using View and ViewGroup objects:


Basic UI building blocks (text, buttons, checkboxes, etc)


Responsible for handling drawing, events, measuing.


Each View is tied to a Surface, which is double buffered.


Surfaces are managed as follows:


ViewRoot (app window) asks SurfaceFlinger for Canvas for View to draw on.


View draws its data on the Canvas.


ViewRoot calls
surface.unlockCanvasAndPost(canvas)

to schedule
surfaceFlinger::composeSurfaces()


SurfaceFlinger::composeSurfaces()

does actual display on panel.


SurfaceFlinger handles transfer of drawn data in canvas to surface front
buffer or backbuffer.



Surface Interface: framework/base/include/ui/ISurface.h


Surface management: framework/base/view/surface.java

Android

Graphics System

SurfaceFlinger Architecture

Surface
-
Flinger
Overlay
-
Obj
1
Overlay
-
Obj
2
OpenGL
|
ES
1
.
1
FBDev
Framebuffer device
JVM
Jsr
239


OpenGL
|
ES
Java Binding
3
D
-
Java Apps
Surface
-
composer
client
Android Java Apps
V
4
L
2
-
vid pipe
1
V
4
L
2
-
vid pipe
2
Display Subsystem
video pipe
1
video pipe
2
Graphics pipe
LCD
Android
Surface Output
Video MIO
(
TI Adaptation
)
PVMF Video
Decoder OMX
TI OMX
Video Decoder
PV Player
Driver
Android Java Apps
JNI Media Class
TI DSP
Codecs
GFX Driver
SGX
Camera HAL
Camera Service
V
4
L
2
-
camera
driver
TI s
/
w
PV s
/
w
Google s
/
w
Open source
TI H
/
w
Alpha


color key
Android


Audio Framework




Android

Multimedia


AudioFlinger Architecture


Manages all audio output devices


Processes multiple audio streams into PCM audio out paths


Handles audio routing to various outputs


Available in:
libs/external/audioFlinger


HAL available in
: hardware/alsa_sound/AudioHardwareALSA.h

APP

APP

Game

Audio

Media

Player

Tone

Audio

Audio

Flinger

Bluetooth

Speaker

Earpeace

Android

Multimedia


Audio Flinger Interaction Layer

Android

Multimedia


Audio Architecture overview

UI

AudioTrack.java

AudioFlinger.cpp

Audio Policy Service

Audio Policy

ALSA Audio HAL

OMAP3 ALSA lib

ALSA lib

Kernel

Java code

Start Stream

(openTrack)

Streams

Most of changes

applied for Éclair by FMA

Minor changes

applied for Éclair by FMA

./ti/omap3/modules/alsa

./hardware/alsa_sound

AudioPolicyManagerALSA.cpp


./frameworks/base/media/java/android/media/AudioTrack.java

IN

OUT

Android

Multimedia


Audio Framework Components


Stream
: digital bits that represent a sound


Stream

VOICE_CALL (used for Voice record)


Stream

SYSTEM (for system sounds)


Stream

RING


Stream

MUSIC


Stream

ALARM


Stream

NOTIFICATION


Audio Flinger
: mixer (mix sound, adjust soft volumes, …): take several streams, mix
them + apply volume on each stream. These streams are entering (
IN
’s stream) or
outputing (
OUT
’s stream).


OUT
’s &
IN
’s are physical HW channels


These
OUT
’s &
IN
’s are attached to
devices
.


Device
= headset, handset, …


Streams

are grouped into
strategy


Strategy
:


Phone
: VOICE_CALL, BT_SCO


Sonification: RING, NOTIFICATION, ALARM


ENFORCE_ENABLED: unmutable strings (exple: in some country it is mandatory to produce a
sound when recording with a camera )


Android

Multimedia


Audio HAL


HAL definition:


Interface: AudioHardwareInterface.h


Stub version of the HAL is located into the framework directory:

framework/base/libs/audioflinger/AudioHardwareStub.h



Platform implementation is located into hardware directory
:


TI HAL solution (depreciated):
hardware/omap3/libaudio/audioHardwareOmap.cpp




ALSA HAL solution (in used):


hardware/alsa_sound/AndroidHardwareAlsa.cpp

Android

Multimedia




Android

Multimedia

Components Diagram

Media Framework
OpenCore
libopencorehw
TI DSP
/
OMX
DSP Bridge
OpenMax IL
Camera
Service
OMAP Video Overlay
OMAP Camer
Interface
OMAP Camera
Overlay
ALSA
OMAP Audio
libaudio
Media Server
PV
/
OMX
ARM Codec
DSP Accelerated
OpenMax codec
libOverlay
OMAP Video Overlay
ARM
<
-
>
DSP
TI DSP Firmware
Executed on DSP Core
PackeVideo
TI OpenMax IL solution
Android
Framework
TI Audio HAL
TI Display HAL
TI Camera HAL
StageFright
TIHardware
Renderer
OpenMax IL
StageFright
Android

Multimedia

Android stack overview

Java UI Application
(
mediaPlayer
)
Libmedia
_
jni
MediaPlayerInterface
Libmedia
.
so
JAVA JNI interface
Media library
Libmediaplayerservice
.
so
Binder
MidiFile
Vorbis
Codec
Libopencore
.
so
OpenCore PVPlayer
Application framework
as part of the
Android framework
Delivered by Google Android package
PV OpenCore
is an opensource multimedia framework
handling multimedia content except Vorbis and MIDI
External delivery from PacketVideo
.
StageFright as been introduced in the last version of Éclair
Should replace OpenCore into FroYo
Stage
Fright
Android Architecture

OpenCore external library


Based on PacketVideo OpenCORE platform


OpenCORE is based on the commercial CORE of ParcketVideo


for Android. This framework is Open source.
http://www.pv.com/products/android/index.html


Large documentation into
opencore/doc

directory


Supports standard video, audio, still
-
frame formats


Non optimized reference codec available in opencore/codecs_v2


Support for hardware / software codec plug
-
ins using OMX


Openmax headers available in external_libs_v2

Android stack overview

Hardware Accelerator
(
IVA
2
/
DSP
)
UI Framework
Frameworks
\
base
\
media
\
PV Open Core
External
\
opencore
\
TI’s or
3
d party OpenMax
Hardware
\
ti
\
omx
\
Binder IPC

MediaPlayerInterface
OMX Core library wrapper
DSP bridge
Application framework as part of the Android framework

Delivered by Google Android package


(mainly in C++)




PV OpenCore is an opensource multimedia framework handle
multimedia content except Vorbis and MIDI

External delivery from PacketVideo




TI OMX implementation to support the IVA2/C64x DSP.

Part of the hardware specific implementation from TI



Android

Multimedia

Android AV Playback layers

UI Framwork
Hardware Acceleratos
(
IVA
2
/
DSP
,
Display
)
Java UI Application
(
mediaPlayer
)
PV Player Driver
Java API
JNI Media Class
PV Player Engine
PV Player Nodes
/
MIO Nodes
/
OMX
PV OpenCore
TI OMX
Third
-
party Video Player develop under JAVA


On Android there is two classes (MediaPlayer and VideoView) provide
support to the video player (VideoView based on MediaPlayer)

framework
\
media
\
java
\
android
\
media
\
MediaPlayer.java


Native Java Interface Layer. Allow JAVA APIs access to the native
libraries (C++)

JNI:framework
\
media
\
java
\
android
\
media
\
JNI
\
android_media_MediaPlayer.cpp

Implementation:framework
\
media
\
java
\
android
\
media
\
libmedia
\
MediaPlayer.cpp


Load all necessary libraries for pvplayer and creates the proper MIO
Nodes that will interact with the platform specific HW

external
\
opencorejava
\
android
\
playerdriver.cpp


PVPlayer creates and configure the required nodes involved on the AV
Playback pipeline so they can interact with each other then it will
oversee the operation of the pipeline.


PV node wrap software or hardware codecs and they can sink or
source of data. The MIO nodes contain glue code between the Video
Pipeline and the HW (e.g. methods to send buffers to the Display)

Android

Multimedia

Media Player Service


Responsibility
:


Provide mediaplayer services to the application framework (MediaPlayer) through the
IPC binder.


Create a proper
playerdriver

to handle the input stream.


The MediaPlayerService supports 4 players:









Create the connection between the
MediaPlayer

(framework process) and the new
playerdriver
(service process)


Manage the audio ( output to the audio framework /Audio cache)

MediaPlayer C++ class interface with the IMediaPlayerService service through binder IPC

PVPlayer
external
\
opencore
\
android
\
playerdriver
.
cpp
Compilation flag NO
_
OPENCORE to remove it
Mediaplayer
VorbisPlayer
Framework
\
base
\
media
\
libmediaplayerservice
\
Vorbisplayer
.
cpp
MidiFile
Framework
\
base
\
media
\
libmediaplayerservice
\
midifile
.
cpp
binder
binder
binder
StagefrightPlayer
external
\
opencore
\
android
\
playerdriver
.
cpp
Compilation flag BUILD
_
WITH
_
FULL
_
STAGEFRIGHT
to include it
binder
Android

Multimedia

Player Driver Creation

Process B
Process A
Media Player
ServiceManager
Binder Driver
MediaPlayer
Service
Get
service
(
“media
.
player”
)
MediaPlayerService
marshal proxy
this
(
mediaplaye
)
Datasource
(
url
)
Call Create
(
getpid
()
,
this
,
url
)
Transact
(
CREA
TE
_
URL
,
data
)
Replay an instance of playerdriver
PVPlayer
/
vorbisPlayer
/
MidiFile
Android

Multimedia

Player Driver

process/service name
:
media.player (dumpsys media.player)

Interface
:
android.hardware.IMediaPlayer

Definition
:
iMediaPlayer.h


Binder proxy
:
BpMediaPlayer

(i
MediaPlayer.cpp
)


perform the interface calls through the IPC (
application process side
)







Binder native
:
BnMediaPlayer

Stub of BnMediaPlayerService
(
MediaPlayerService.cpp
)


Implement the interface (service process side)


Stub as from BnMediaPlayerService s running into the same process





status_t start() {

Parcel data, reply;

data.writeInterfaceToken(IMediaPlayer::getInterfaceDescriptor());

emote()
-
>transact(START, data, &reply);


return reply.readInt32(); }


onTransact(

reply
-
>writeInt32(start());

Android

Multimedia

Media Player Driver

Process B
Process A
Media Player
Binder Driver
PVPlayer
driver
prepareAsync
()
Transact
(
PREPARE
_
SYNC
)
Interface with
OpenCore through
PVPlayerInterface
Start
()
Transact
(
START
)
Replay iMediaPlayer getInterfaceDescriptor
Transact
(
PREPARE
_
SYNC
)
Interface with
OpenCore through
PVPlayerInterface
Android

Multimedia

Media Player Service UML

1

2

3

1

2

3

Binder : Retrieve appropriate Player


Binder: Abstract the player Control

OpenCore interface: Abstract PV Engine

Android

Multimedia

Android OpenCore Player

OpenCore Common library
(
libopencorecommon
)
OSCL
(
Operating System Compatibility Library
)
PVPlayer
Player Driver
Video Output
Audio Output
Android Player
Player Engine
PVMI
(
PacketVideo Multimedia
Interface
)
Libopencoreplayer
.
so
Android

Multimedia

OpenCore PVPlayer

Wrapper between Android and OpenCore
:


Implement binder
android.hardware.IMediaPlayer

interface for
UI framework connection


Support a playerdriver object (
mPlayerDriver)

to interface with
OpenCore (PV Player Engine)


header:
frameworks
\
base
\
include
\
media
\
PVPlayer.h

Implementation
:
external
\
openCore
\
android
\
playerdriver.cpp





Android

Multimedia

OpenCore PlayerDriver

Interface with the PVPlayer Engine
:


Support PVPlayerInterface
:
Asynchronous and Synchronous
commands

send to

PVPlayer Engine
(PlayerCommand)


Implement callback for
event handeling called by PVPlayer Engine

(PVCommandStatusObserver,PVInformationalEventObserver,
PVErrorEventObserver OpenCore interfaces)



Header/Implementation:
external
\
openCore
\
android
\
playerdriver.cpp

Commands definition/implementation:
external
\
openCore
\
android
\
playerdriver.h

Android

Multimedia

OpenCore PVPlayer Engine


Heart of PVPlayer
:


receives and processes requests from playerDriver


manages the PVMF components required for multimedia playback and related
operations.



Define the PVPlayerInterface to allow PlayerDrivers to interface with


Process PlayerDriver commands initiated by API calls asynchronously.


Notify the PlayerDriver from events coming from PVMF


Maintains a state machine modified based on PVPlayerInterface APIs called and
events from PVMF components below


Header:
external
\
openCore
\
engines
\
player
\
include
\
pv_player_interface.h

Implementation:

external
\
openCore
\
engines
\
player
\
src
\
pv_player_interface.cpp


Android

Multimedia

PVPlayer Engine Class Diagram

Android

Multimedia

PVPlayer Engine State Transition Diagram

Android

Multimedia

OSCL (Operating System Compatibility Library)



OSCL abstracting the HLOS dependencies to
manage:


Thread (osclproc directory)


memory (osclmemory directory)


file/socket (osclio directory),


etc…



OSCL directory:
external/opencore/oscl/


Documentation
: external/opencore/doc/oscl_html


Android

Multimedia

PVMI(PacketVideo Multimedia Interface)

1) PVMF is the OpenCore multimedia framework:


Define the multimedia architecture upon which the PVPlayer engine is based


Define Nodes

(based on OSCL) to manage:


files Nodes

Responsible to parse media file (mp3, avi)


Nodes available in: in
external/opencore/nodes


Parser available in: external/opencore/fileformats


OMX Nodes

to communicate with OMX layers


Encoder: external/opencore/nodes/pvomxaudiodecnode/


Decoder: external/opencore/nodes/pvomxvideodecnode/


PVMF directory:
external/opencore/pvmi


2) Provide
MIO

(Media I/O)
interfaces


Files MIO: external/opencore/pvmi/media_io


Android Specific for Surface, Camera, Audio MIO:external/opencore/android/



No specific documentation


Android

Multimedia

Media Output to Node and Media I/O

2 methods for PVPlayer Engine handles interfacing to platform
specific output media devices :

1.
Encapsulate the media device in a PVMF node which PVPlayer
Engine can use directly.


This method minimizes the code between PVPlayerEngine and the
media device interface.


Interface the media device to PV’s Media I/O interface.


PVPlayer Engine can use the PVMF node that interfaces PV Media IO to
output

Android

Multimedia

PVMI Component Diagram (Decode path)

PVPlayer Engine

MIO Output PVMF Node

Master Core

Parser Node

Opencore

PVMI


Speaker

Display

surfaceflinger

audioflinger

AudioHardwareInterface

omx audio/video dec node

OMX Core

MM decoder

TI OMX

media input node

DisplayHardwareInterface

TI SW

Google

Packet Video

Opensource

Hardware

Android

Multimedia

PVMI Component Diagram (Record path)

Record Applications

media Input node

Master Core

Muxer Node

opencore

Microphone

Camera

surfaceflinger

audioflinger

AudioHardwareInterface

omx audio/video enc node

OMX Core

MM Encoder

TI OMX

media output node

CameraHardwareInterface

Camera HAL

Display

Display

Hardware

Interface

Camera Service

TI SW

Google

Packet Video

Opensource

Hardware

Android

Multimedia

OpenMax Definition

Definition
:


OpenMAX is a royalty
-
free, cross
-
platform API that provides


abstractions for routines especially useful for computer
graphics,video, and audio.


Website:
http://www.khronos.org/openmax/

OpenMAX AL

(Application Layer)


Audio/Video/Image Playback/Recording

OpenMAX IL

(Integration Layer)


Blocks of functionality: sources, sinks, filters, etc.

OpenMAX DL

(Development Layer)


Low
-
level building blocks: DCT, LoopFilter, Quantization, etc..

Android

Multimedia

OpenMax IL Overview


OpenMax IL API is a component
-
based media API


Consists of two main segments:



Core API (unique component):


Dynamic loading/unloading components management


Facilitating component communication (Tunneling)


Component API (several components):


Individual blocks of functionality


Can be source, sink, codecs, filters, splitters, mixers, etc..

Android

Multimedia

OpenMax IL into OpenCore

Master Core
: OpenCore support several OMX vendors solutions:


method of dynamic loading/linking of multiple OMX cores


OMX core methods called through a thin wrapper layer
(mastercore):
external/openCore/codecs_v2/omx/omx_common/src/pv_omxmasterco
re.cpp


Avoid co
-
existence of multiple OMX cores issues


Maintain a cross
-
vendors OMX component registry


Interface


PV OpenCore provides an implementation of PV OMX core and PV
OMX components in
external/openCore/codecs_v2/omx/


OpenCore framework behaves as OpenMax IL client when it
communicates with Vendors OMX core.


OpenCore
PVMF OMX Nodes
:


Wrapper between the PVPlayer Engine and the OMX components


Directory:external/openCore/nodes/pvomxencnode


Directory:external/openCore/nodes/pvomxvideodecnode


Directory:external/openCore/nodes/pvomxdecnode


Directory:external/openCore/nodes/pvomxbasedecnode






Media Framework
OpenCore
TI DSP
/
OMX
OpenMax IL
Ffmpeg
/
OMX
ARM Optimized Codec
DSP Accelerated OpenMax codec
Android

Multimedia

OpenMax IL Master Core

Avoid co
-
existence of multiple OMX cores issues:


Maintain a cross
-
vendors OMX component registry


Method of dynamic loading/linking of multiple OMX
cores

Define “OMXInterface” Interface
to be
implemented by OMX Core vendors


Implementation
:external/openCore/codecs_v2/omx/omx_c
ommon/src/pv_omxmastercore.cpp


Interface definition:

external/openCore/codecs_v2/omx/omx_common/include/om
x_interface.h


Documentation:

external/openCore/doc/
omx_core_integration_guide.pdf






Media Framework
OpenCore
TI DSP
/
OMX
OpenMax IL
Ffmpeg
/
OMX
ARM Optimized Codec
DSP Accelerated OpenMax codec
OMX Core

master core

TI core wrapper

master core

component
registry

Component query by role from
Client

OMX Core

PV core wrapper

Other vendor OMX?

OMX.TI.AAC.decode

OMX.TI.XYZ.decode

OMX.TI.AAC.decode

First match?

OMX.PV.aacdecode


opencore player


decode


opencore author


encode


opencore common


PV master core


OMX master registry


Vendor OMX Core wrappers


Android

Multimedia

OpenCore OMX Master Core

Android

Multimedia

TI OpenMax IL Solution


TI OpenMax IL is a DSP based solution
.


TI OMX Core
:
hardware/ti/omx/core_plugin/omx_core_pluging/src/ti_omx
-
interface.cpp

TI OMX Components
:



hardware/ti/omx/[audio,video,imaging]


Interface with the DSP SocketNode through the Bridge

DSP Components
:


DSP baseimage including the algorithm/SocketNodes/bridge/bios
are loaded by android init process (init.rc)


The ARM Bridge is loaded as part of the Linux kernel







Android

Multimedia

TI OpenMax IL Solution

ARM

DSP

OMX Core

AAC Decoder

OpenMAX Component

MPEG Video Decoder

OpenMAX Component

AAC Decoder

Socket Node

MPEG Video Dec

Socket Node

LCML

MPU Bridge

DSP Bridge

AAC Decoder

Algorithm

MPEG Video Dec

Algorithm

Master OMX Core

PV OpenCore

PVMF OMX audio/video
decoder nodes

PVMF media
output node

AudioFlinger

Audio Hardware
Interface

TI OMX

SurfaceFlinger

Video Hardware
Interface

USN

TI MM Component

External Component

PVMF Parser
node

PVMF media
input node

Android

Multimedia

TI OpenMax IL Solution

Algorithm


XDM/XDAIS compliant. Runs on DSP/BIOS.

Socket Node


Algorithm
-
specific preparation around the algo’s process() call


TCONF file specifies memory requirement for various codec profiles

USN


Generic data & control processing, including DSP cache coherency operations

Bridge


Loads DSP baseimage and dlls. Manage DSP nodes. DSP Power Management.


Map/unmap buffers. Implements ARM cache coherency APIs.

LCML


Performs common data & control processing for OMX components.


Groups together several Bridge API calls into one. Serves as Bridge abstraction.

OMX


Implements standard OpenMAX interfaces for integration with HLOS MM Framework


Android/PV Master OMX framework is in C++, whereas TI’s code is in C

Android

Multimedia

Create of AV playback pipeline

Android

Multimedia

Rendering through Overlay Interface

Android

Multimedia

Overlay Interface


1.

Implement the stub funtions

in
hardware
\
libhardware
\
modules
\
overlay

\
overlay.cpp
, this will generate a share library liboverlay.trout.so.


2. In SurfaceFlinger, it will
create a DisplayHardware instance
, and

call
DisplayHardware::init()

which will create an overlay engine from

liboverlay.so.



3. When we want a OverlayRef from ISurface, we call

ISurface::createOverlay()

which will take use of the overlay engine


create in step 2.


4.
Create a Overlay from OverlayRef

as it in
frameworks
\
base
\
libs

\
surfaceflinger
\
tests
\
overlays
\
overlays.cpp
.



TI implementation to support overlay for the openCore

video is available
in:
Hardwared/ti/omap3/libopencorehw/


http://git.omapzoom.org/?p=platform/hardware/ti/omap3.git;a=tree;f=libopencorehw;h=4
b843e240897931c150f1a0cb25b1785c9e8dab2;hb=HEAD

Android

Multimedia

TI
Audio Decode Architecture

LCML
TI OMX Core
PV OMX Core
TI OMX
Decoder
TI OMX
AAC Dec
/
Enc
TI OMX
AMR Dec
/
Enc
TI SN
Decoder
TI SN
AAC Decoder
/
Encoder
TI SN
AMR Decoder
/
Encoder
DSP Bridge
PV OMX Node
PV OMX Node
PV OMX Node
PV
Parser Node
PV AAC
Parser Node
PV AMR
Parser Node
PVPlayer
/
Recorder Engine
PVPlayer
/
PVRecorder Driver
PV Audio
Input
/
output
MIO
PV Input
/
Output
Node
(
MIO
)
Inpu
/
Output File
Audio Flinger
ALSA driver
Audio Driver
DSP
ARM
Kernel
User
M
c
B
S
P

D
R
V
I
2
C

D
R
V
I
2
C
I
2
S
P
C
M
T
r
i
t
o
n
2
Earphone
Stereo
/
mono
Headphone
2
x
Speakers
Stereo
/
mono
Microphones
JNI Media Class
/
PVPlayer
MediaPlayer Class
Audio HAL
Android Music Record UI
Android Music Player UI
MediaRecorder Class
TI s
/
w
PV s
/
w
Google s
/
w
Open source
TI H
/
w
OMX PCM Dec
DASF
CDN
Not
-

Supported
AudioFlinger provides Audio Routing & Mixing Features.

Android

Multimedia


TI
Video Decode Architecture

LCML
TI OMX Core
PV OMX Core
TI OMX
MPEG
4
Dec
TI OMX
H
264
Dec
TI Socket
Node
MPEG
4
Dec
TI Socket
Node
H
264
Dec
DSP Bridge
PV OMX Node
PV OMX Node
PV MPEG
4
Parser Node
PVPlayer
/
PVRecorder Engine
PVPlayer
/
PVRecorder Driver
PV Video
output MIO
PV Input
/
Output
Node
(
MIO
)
Inpu
/
Output File
Surfaceflinger
V
4
L
2

display
user lib
V
4
L
2
display
DSP
ARM
Kernel
User
Video overlay
obj
Android Video Player UI
TI s
/
w
PV s
/
w
Google s
/
w
Open source
TI H
/
w
DSS
vid
1
vid
1
gfx
LCD
PV H
264
Parser Node
Android

Multimedia


AV Sync

Android

Multimedia


720 Split Codec Architecture

TI Proprietary Information


Strictly Private

Android

Multimedia


720 Split Codec Architecture

Choice of Video Codec


Config parser helps determine the video codec based on the video resolution


Load Ittiam (split) codec if resolution is 720p, TI (DSP
-
only) codec if
resolution is D1 or lower.


Video codec OMX component registers it’s DSP MHz requirement with the
OMX Resource Manager

Choice of Audio Codec


OMX Core tries loading TI’s (DSP) audio codec first. If this fails, then it loads
Ittiam’s (ARM) audio codec.


Audio codec OMX component registers it’s DSP MHz requirement with the
OMX Resource Manager


If 720p video codec was loaded earlier, then DSP MHz are not available for
TI audio codec. RM returns error, failing the loading of TI’s audio codec.

Requires Video Codec to be loaded before Audio Codec


PacketVideo is evaluating this change to their Framework.