Trends and Standards for 3D Graphics for Handsets

trexpeeverSoftware and s/w Development

Dec 13, 2013 (3 years and 7 months ago)

56 views

Trends and Standards for 3D
Graphics for Handsets

Christophe Quarre’

Advanced System Technology, Shanghai Lab

STMicroelectronics

Christophe.quarre@st.com

State Key Lab of CAD&CG, Zhejiang University

1
-
3 March 2005

2

Outline


Handhelds VS Desktops.... new industry


Challenges for ISVs


Standards for graphics mobile.... and 3D acceleration


The key players... ISVs & IHVs


Conclusion


3

Mobile 3D industry


Mobile phones are about to be transformed in personal compute
devices


Graphics capabilities will be a key ingredient


just as on the PC


Before we had 2D, but now we want 3D !


The Mobile 3D industry is embryonic
-

but moving fast!


We are where PC graphics were in 1996


But evolving about three times faster!

4

Game market in Korea

0
50
100
150
200
250
300
350
400
450
500
Online game
Mobile
Video(PS2,
XBOX)
Arcade game
PC game
2001
2002
2003
<From : 2003 Korea Game White Paper>

[
USD million
]

Surging Video and mobile game market

5

Desktop vs. Handheld Systems

Desktop

Handheld

CPU

INTEL

1GHz up to >3GHz

ARM as de
-
facto standard

100MHz to 500+MHz

FPU

YES

Today

ARM9 (Nomadik 1 @ ST)


NO

Tomorrow

ARM11(Nomadik 2 @ ST)


YES

CACH, MEMORY,
BANDWIDTH

Very big,

power hungry

~10s of KB of eSRAM

~1 MB of eDRAM is now feasible

~100s of MB DDR
-
SDRAM

Ultra low power design library

RESOLUTION

SVGA

>1024 * 768 pixels

QVGA (320*240)

VGA (640*480)

GPU (e.g.)


Nvidia: GeForce 6800,

ATI: Radeon X800

IMG, Falanx, ATI, nVidia, Bitboys and
others

3D API

OpenGL1.5 / 2.0

DirectX

Java3D

OpenGL ES 1.0/1.1and future 2.0

D3DM

JSR184 M3G

6

Challenges on Handheld Platforms


Small battery, small CPU



Management of Host CPU



unload max work on Host CPU



and

hardware resources



can save 90% of battery life !


Limited system memory size and bandwidth



Memory architecture will make the difference !!



Cache memory; buffers compression


A wide range of Mobiles and gfx architectures, from Low
-
End


High
-
End



Means
scalable and flexible

solutions !!!


7

Challenges for Software Vendors

1/2



If we consider 3 likely classes of handheld device



Class A : High
-
performance CPU, GPU with vertex processing +
FPU



Class B : High
-
performance CPU, GPU with vertex processing



Class C : CPU, rasterizer



Classes B and C, low cost will likely ship in higher volume,



therefore

may offer more revenue opportunities for software vendors



but

yet platforms do not have floating
-
point support (FPU)



Class A device may win out for better capacity and performance



Software vendors must cover all the bases to guarantee success


Observation

8

Challenges for Software Vendors

2/2


How to develop or port rich and attractive applications with such a
spread of computational capabilities?


System Software must be
scalable

with
Unified driver architecture

to
rapidly be adapted to different system architectures


Middleware is a key component



E.g. RenderWare (Criterion) optimized for Nokia N
-
gage


Action

9

Standardization


Different class of mobiles and Handsets, different capabilities


Many Graphics accelerator Suppliers


Graphics
low power

for mobile is a new industry




Need for standardizations


10

An Important and leading Group


Member
-
funded
industry consortium


Lead specification of
open standard
,
multi
-
platform
,
royalty
-
free

low level

APIs for HW acceleration


Leverage graphics technology by working closely with ISVs IHVs

2002

Embedded 3D

Graphics Acceleration

2004

Embedded 2D Vector
Graphics
Acceleration

2000

Streaming Media
Control and Video
Acceleration

2004

Embedded Media
Primitive
Acceleration

11

Khronos Members

Promoting Members

Contributing Members

12

OpenML


Standard for Dynamic media authoring


royalty
-
free, cross
-
platform programming environment


Abstraction layer for capturing, transporting, processing,
displaying, and synchronizing digital media :




audio/video streams


2D/3D graphics streams


OpenGL extensions for accelerated video processing


Lowest level of unified functionality, thinnest layer on top of HW

Application

Windows
system

OpenGL


OpenGL
Extension

MLdc

ML

Synchronisation

13

OpenVG

1/2


Standard for
Vector Graphics

acceleration


Why OpenVG ?


2D vector graphics is necessary for
high quality

GUIs or Font display on small
display


Need
hardware acceleration

for useful performance


Existing vector graphics APIs
are proprietary

and developers must rewrite
code for each platform


Existing APIs usually designed for software rendering



Vector Graphics



Exemple

Original image SVG zoom Bitmap zoom

14

OpenVG

2/2


Standard for
Vector Graphics

acceleration


OpenVG
: Royalty
-
free, cross
-
platform,
low
-
level APIs for hardware
acceleration for high quality
2D vector graphics
.


Interface for vector graphics libraries such as Flash and SVG


Same framework as OpenGL|ES


combine with 3D APIs

(Blending, lighting, texturing…)


OpenVG1.0 specs is currently under review by selected ISVs.


Official release expected in
March 2005

!

15

OpenMAX


Standard for
Media acceleration primitive


OpenMAX
: royalty
-
free, cross
-
platform,
common APIs

for multimedia
application


standardizes access to media
processing primitives

used extensively in
graphics, audio and image libraries and video codecs (eg. MPEG
-
4)


Benefit:
Accelerate porting

of multimedia SW


Currently in development, expected release in End 2005

Video

MPEG
-
4

H.264

Etc.

Voice

G.7xx

Etc


Audio

MP3

AAC

etc.

Image

JPEG

JPEG2k
etc

Multimedia

API’s

RISC


DSP


SMP


H/W

accel

Multimedia

Accelerators



The problem

PORTING PROBLEM!



slow down
multimedia sw
development



POORLY optimized

16

OpenMAX


Standard for
Media acceleration primitive


Accelerating a broad range of media types

3D

Small footprint 3D for
embedded systems

Video

Dynamic Media
Authoring

Vector 2D

Low
-
level hardware
acceleration API

Media Engines


CPUs, DSP, Hardware Accelerators etc.

Accelerated media primitives for

cross
-
platform acceleration

of media libraries

Image libraries

Video Codecs

Sound Libraries

Applications

Three media libraries called by
applications. Can be ported
directly to silicon.

17

OpenGL ES


Standard for Embedded 3D graphics acceleration


Khronos has created a
small
-
footprint

subset of OpenGL


Created with the blessing and cooperation of the OpenGL ARB


Full functionality for 3D games


On a wide variety of platforms


including fixed point processors



Eliminate
Redundancy

Eliminate
Workstation
Functionality

ARB

Feedback and Ratification

Embedded

Focus

Workstation

Focus

Small footprint

e.g. 50KB software
engines

18

OpenGL ES Roadmap


Update OpenGL ES every year by default


To expose rapidly developing handheld platforms capabilities


BUT ONLY introduce features with proven demand from ISVs or IHVs


Guarding against unnecessary pieces of HW


Track and adapt developments in desktop OpenGL


OpenGL ES 2.0

OpenGL 2.0

OpenGL ES 1.1

OpenGL ES 1.0

OpenGL 1.3

OpenGL 1.5

Enabling software AND hardware implementations


including small
-
footprint, low
-
end fixed point platforms

Shader programmability
for embedded devices

Widely available
cross
-
platform 3D
graphics API

Full high
-
level shading language capability to
harness the power of programmable hardware

Mid
-
03

Mid
-
04

Mid
-
05

Increasing emphasis on enabling
emerging hardware with video
support and enhanced 3D pipeline

19

Java


The multiplatform language


Game developers prefer Java, better for creating and
managing tools and high level gfx libraries


But still need
low level

access to the 3D HW


Binding to OpenGL and OpenGL|ES


JSR 231



OpenGL



(Expert Group created in Oct 2003)


JSR 239



OpenGL ES



(Expert Group created in March 2004)


Wrappers around native OpenGL ES implementation


Higher level, Scene Graph API


JSR 184



Mobile 3D Graphics API for J2ME
TM

(Released in Nov 03)


can be built on a native OpenGL ES implementation


can be layered on JSR 239


Reduce application size for OTA downloading


20

SW Standards layers for 3D Mobile graphics

C/C++
Applications

Scenegraph APIs

M3G (JSR 184)

Middleware

Libraries

Games

Engines

Java

Applications

Hardware

OpenGL ES


Engines

Software

OpenGL ES
Engines

Operating
Systems

Low
-
level

3D Graphics

API

High
-
level

Graphics

Libraries

Applications

J2ME

Hardware

Hardware

21

Graphics SW (Engine, Middleware…) Providers


Games developers



Game Engine, Middleware,
Dev. tools



OpenGL|ES
implementation and driver,
Authoring tools...



JSR184/239

Mascot Capsule

X
-
FORGE

22

Graphics HW acceleration Providers

Acceleon

G30/G40™

PowerVR MBX / MBX Lite

GoForce 4000/3000

IMAGEON 2300

And



of course !!

Mali50/100

23

Mobile Multimedia chips Providers

ARM9 (200MHz), VFP, MBX

ARM11 (330MHz), TMS310 (220MHz), VFP,
MBX

ARM11 (500MHz), MMDSP(ST), VFP,
3D Acc (ST)

Hitachi semiconductor
Mitsubishi semiconductor

SH
-
CPU (600MHz), SH
-
DSP, VFP, MBX

24

Conclusion


3D graphics for mobile is a new industry,



for a new market...


To be competitive, SW systems need to be scalable, with



unified low level standards,



unified drivers...


Middleware is the key for success,



to provide efficient data conditioning...