Spaceport PerfMarks Report

uglyveinInternet and Web Development

Jun 24, 2012 (5 years and 3 months ago)

363 views

Spaceport PerfMarks Report:
Testing HTML5 Performance
Across Mobile Browsers

- March 2012 -
www.spaceport.io
Want to run the tests yourself or review our code?

PerfMarks are an open-source project maintained by spaceport.io.
Fork us on github
.
The Data:
The March 2012 PerfMarks report seeks to answer the following question:
How many images can be moved around a screen at one time, while maintaining a 30
frames per second (FPS) frame-rate*?
Game and app performance relies heavily on animated objects smoothly moving around the
screen. As such, this test – when conducted across various operating systems, browsers
and devices – serves as a proxy for evaluating performance of HTML5 games and apps.
For an explanation of the various animation techniques used
click here
. The test was
conducted across the following browsers and devices:
Operating Systems**

:


iOS 5

iOS 4

Android 4.0

Android 3.0

Android 2.0

Android - Kindle Fire

Blackberry Playbook OS
*30 FPS is the minimum frame rate
for maintaining visual smoothness on
mobile games
**Android 4.0 for tablets not
included in this report.
Android Devices

:


Samsung Galaxy Nexus
(Android 4.0.2)

Samsung Droid Charge
(Android 2.3.6)

Motorola Droid 2
(Android 2.3.3)

Sony Ericsson Xperia
PLAY
(Android 2.3.4)

Amazon Kindle Fire
(System Version 6.2.2)

Samsung Galaxy Tab
10.1
(Android 3.2)

Asus Eee Pad
Transformer TF101
(Android 3.2.1)

Blackberry Playbook
(Playbook 1.0.8.6067)

Motorola XOOM
(Android 3.2.6)
iOS Devices

:



iPhone 4S (iOS 5.0.1)

iPhone 4 (iOS 5.0.1)

iPod Touch 4G (iOS
4.3.5)

iPhone 3GS (iOS 4.3.3)

iPad 2 (iOS 4.3.3)

iPad 2 (iOS 5.0.1)

iPad (iOS 4.2)
High Level Summary of Findings:
iOS is strongly in the lead here - both in terms of phones and tablets.

Developers can
expect to have more than 200 moving objects on the best iPhones, and more than 300
moving objects on the best iPads, but only around 100 moving objects on the best Android
phones.
There is an even starker contrast when looking at the low-end devices in each category. The
worst iPhone tested (3GS) was able to achieve 50 moving objects at 30 FPS - using CSS
3D transforms on an image element. The worst Android device tested (Motorola Droid 2)
was not even able to maintain 30 FPS with a
single
moving object on the screen, across all
possible animation techniques.
Platform and Device-Specific Results:
Now, we dive into platform-specific results with more in-depth data:
Android Phone Performance
Android does not look terrible in this summary, however, the high average values across the
two top animation techniques are skewed due to a single outlier: the Samsung Galaxy
Nexus. This outlier pulled up the average for all other Android phones.
Only the Samsung Galaxy Nexus can process more than 1 moving object, across all
available technique.

That said, even the Samsung Galaxy Nexus only manages to make 30
FPS using two techniques – CSS 3D background and css3d image.

Unfortunately, neither
of these two techniques are supported on any of the other Android devices tested here.

With the Samsung Galaxy Nexus being an Android 4.0 phone,

we can conclude that there
have been significant improvements implemented in the Android Browser with this upgrade.
For this reason, we do not recommend playing HTML5 games on an Android device with
V3.0 or below. Even then, application developers will have to be very careful to detect that
the game is running on the Android Browser and make sure to use CSS 3D transforms to
do all animation.

Using any other technique simply will not work.

This is unfortunate
because not only does this technique not work on other Android devices, but because it is
not the fastest technique on all HTML5 devices.

For now, HTML5 game developers will
simply be forced to do platform detection and use a particular animation technique on each
platform.
iOS iPhone/iPod Touch Performance
The first point to recognize here is that mobile Safari on the iPhone and iPod supports all
the animation techniques tested, all the way back to the iPhone 3GS. Right off the bat this
is a major step above Android.

Secondly, unlike the Android Browser, there is more than one acceptable option; both
canvas and CSS 3D transforms will perform adequately.
Before the iPhone 4S came along, the best technique was clearly using CSS 3D transforms
on image elements.

However, since the introduction of iOS5 this is no longer the case.


With iOS5 we found the best technique to be canvas.

This is a shock as canvas has, for a
long time, been one of the slowest techniques (further benchmarks to come in the future
will fully vet this by testing rotation as well as scaling).

Through the combination of iOS5 and the iPhone 4S, an incredible 250 objects can be
moved around while maintaining 30 FPS.

This is some serious performance, and makes
complex games a possibility (while keeping in mind a real game does a LOT more than
just move images around).
In comparing the bottom line, even the oldest device here (the iPhone 3GS) is able to reach
53 moving objects at 30 FPS.

This is another stark contrast with the one moving object on
the low-end Android devices.
Finally, note the exponentially increasing speed curve.

For each release, Apple has greatly
improved their HTML5 frame rates. This looks like a strong trend that will continue in the
future, which is great for the HTML5 development community.
Android Tablet Performance
The Android tablet results are kind of suspect.

There were significant visual issues with
the Android 3.0 browser on the CSS 3D transform techniques due to bugs in the Android
browser.

As such, the maximum achieved frame-rates should be taken with a grain of salt
as not all objects were rendering.
Note that the best technique here (CSS 3D background image) is not the best technique
anywhere else and should be another special case for developers to consider.
That said the results here are well below 100 moving objects, regardless of the device or
technique.

That should be surprising given how powerful this hardware is.

Tablets are like
mini-computers and have significant resources compared to mobile phones. Some of these
tablets have Tegra graphics cards but these are clearly not being well utilized by the
Android Browser software stack.
The Kindle Fire is also included in this group.

At best the Kindle Fire can handle 25
objects.

This is half of the iPhone 3GS, which is very

disappointing.

Also note that the
Kindle Fire does not support CSS 3D transforms.

This gives it the dubious distinction of
being the only Android tablet tested that does not support all of these techniques. As such,
we would not recommend this tablet for HTML5 gaming.
Although not an Android device, the Blackberry Playbook is also included in this group for
the sake of comparison.

It is able to support 85 moving objects using CSS 3D transforms.
Therefore the Playbook beats out all of the Android tablets for HTML5 (at least in terms of
number of translating images).
iOS Tablet Performance
As one can see in the graphs, there are two data series for the iPad 2.

That's because the
iOS 5.0 update dramatically changed the results:
The iPad 2 on iOS 4.3.3 was at best able to achieve 198 moving objects while maintaining
30 FPS.

That was using the css3d transforms on img elements technique. However, since
updating to iOS 5.0 the iPad 2 can achieve
326 moving objects
.

The technique of choice is
now using canvas, which is very interesting. We speculate that Apple put a special case in
the mobile Safari renderer for the canvas drawImage() to use hardware

accelerated

blitting.


This same technique on the same hardware prior to the iOS upgrade used to only be able
to handle 18 objects.

This showcases the difference between hardware and software
rendering.
As with the iPhone, all techniques are supported on both iPads.

The findings also show that HTML5 is generally getting faster over time, with the notable
exception of CSS 3D transforms.
To conclude, the iPad 2 manages over 300 moving images at 30 FPS and cannot be beat
for speed on any browser/device/OS combination.