Work with augmented reality

hihatcloverSoftware and s/w Development

Jul 4, 2012 (5 years and 4 months ago)

242 views

84 Project four New technologies
Adobe Flash CS4
ActionScript 3.0
Work with
augmented
reality
Sam Rivello reveals how
to create a fully immersive
user experience that blends
3D models with the real world
Augmented reality (AR) combines elements of the
physical world with an interactive 3D virtual world. While the
technology isn’t new – think EyeToy on the PlayStation – it’s now
much more accessible to individual designers, and has taken off in
a big way recently thanks to high-profi le AR campaigns from the
likes of Nike, Ford and Coca Cola.
Unlike virtual reality, AR doesn’t seek to replace our
physical world, but add to – or augment – our reality. There are two
main applications of this. Firstly, you can project an interactive digital
interface onto a pair of translucent glasses, turning you into a
real-world version of Schwarzenegger’s Terminator. Clearly this
approach has various military applications, not to mention fully
immersive gaming experiences.
Alternatively, you can blend a virtual environment with
webcam footage in real time, so the augmented reality plays out on
your computer screen, rather than over your fi eld of vision. It’s this
second approach that we take you through in this project, and it can
really add spice to your interactive design work.
Sam Rivello
With a decade of
experience creating
applications and
games, Rivello works
across consulting,
software architecture,
development and
training. www.
rivellomultimedia
consulting.com
Skills
Display webcam
video in Flash
Add interactivity
with Papervision3D
Gain a better
understanding of
ActionScript 3.0
Computer Arts Projects _July 2009 www.computerarts.co.uk
01
Adobe’s Flash Player handles the heavy lifting
for this project, but as this diagram shows, there are
three other major components that make augmented
reality possible. Video input from your webcam, the
FLARToolkit plugin, and Papervision3D. You’ll need your
own webcam, but you’ll fi nd FLARToolkit, Papervision3D
and all the relevant libraries you need on your cover disc.
03
Print your marker. Open up the Marker Generator AIR application on your cover disc, then point your camera at the
marker. Make sure it’s placed in a clutter-free location, such as on a blank wall, as complex backgrounds can confuse the
application. The Marker Generator will indicate a readable marker with a red outline. Click Save Pattern to create the marker
data fi le – this will be used by the fi nal application to recognise your marker when you show it to your webcam.
02
Create a marker graphic, like the one above.
FLAR will detect this, then tell Papervision3D to position
your model over it. Your marker must contain a simple
black-and-white shape, centred within a thick black
square. Note that a slice of the circle has been removed:
this asymmetry will help FLAR to detect its angle and
rotation properties, so your model can rotate with it.
On your disc
You’ll fi nd the
resources you need
to work along with
this project on your
cover disc, in the
Resources section.
There are several
Collada models to
experiment with
CAP125.tut_augreal 84
29/5/09 10:50:28
Work with augmented reality 85
www.computerarts.co.uk July 2009_ Computer Arts Projects
06
Create two new sub-folders in your Augmented Reality folder, called ‘libs’ and ‘assets’. Copy all the
SWC fi les from your cover disc into the libs folder, and copy your Collada model, camera data, and the pattern
generated from your marker graphic into the assets folder. Back in Flash, create a new project in the Project
Panel (Window>Other Panels>Project Panel), using your Augmented Reality folder as the root directory.
04
As well as the Hummer we’ve used here, you’ll fi nd a selection of Collada-format 3D models on your cover
disc – a fi re truck, police car, taxi and Porsche – all created by Antics 3D. Alternatively, you can download a new
royalty-free model from the web, or create your own in a Collada-compatible 3D program. Blender is free, and you just
need to install the Collada plugin (www.colladablender.illusoft.com). Remember: if you do use your own, be sure to
change the COLLADA_3D_MODEL variable in step 9 to point to the new URL for your model.
05
Create a new Flash CS4 fi le called AugmentedReality_v1.fl a, and save it in a
new folder called Augmented Reality. All the code you need for this project is contained
within AugmentedReality.as – copy this fi le from your cover disc into the folder. Using
the Properties panel (Window>Properties), set the player to Flash 10 with ActionScript
3.0, and set the document class to AugmentedReality.as.
07
Open up AugmentedReality.as. This is the full working code that you need to
make the augmented reality process work – you don’t need to understand all of it, but
there are certain editable elements that you can tweak to customise your project.
Steps 08 to 10 will explore the main four sections of the document class, as shown
above: Imports, Class Properties, Constructor, and Methods.
Google SketchUp
A free modelling program from Google, SketchUp is part of a huge-scale project to populate Google
Earth with 3D models. You can also fi nd a growing, fully-searchable repository of Collada models (of varying
quality) at http://sketchup.google.com/3dwarehouse, which is where we found the vehicles for this project.
CAP125.tut_augreal 85
29/5/09 10:50:29
86 Project four New technologies
Computer Arts Projects _July 2009 www.computerarts.co.uk
08
First, fi nd the Imports section. This pulls in existing libraries from three
major packages, to handle most of the work. Classes from Flash (lines 6-11) handle
low-level operations like capturing webcam data and displaying visual elements
(sprites) on-screen. Libspark (lines 13-19) handles the augmented reality marker
detection, and fi nally Papervision3D (lines 21-28) imports and displays your 3D model.
09
Scroll down to the Class Properties section. These modify the libraries you imported in step 8, and
are defi ned in three major areas – webcam properties (lines 42-43), which mirror the user’s physical space
on-screen; FLARToolkit properties (lines 46-55), which get your marker detection running smoothly; and
Papervision3D properties (lines 65-68), which load and render your 3D model. Remember to check the
fi lepath for your marker graphic (line 58) and model (line 81) if you’ve created your own.
10
The third section is the Constructor, which gets your project ready to run. Lines
96-98 prepare the webcam, the marker detection function, and Papervision3D in turn.
Finally, line 102 loops the marker detection continuously while the application is running, so
your model moves in tandem with its marker graphic. We touch on this more in step 13.
Editable properties
The last block of Class Properties in Step 09 (lines 71-78) are editable. Try altering them and
republishing the project. Dramatically reducing WEB_CAMERA_WIDTH and WEB_CAMERA_HEIGHT will
pixelate your webcam video like an 80s arcade game, for instance. Use values within our suggested
guidelines for predictable results, or experiment with something off the scale and see what happens.
The easiest thing to tweak, of course, is the model – the four to the right are included on the disc.
CAP125.tut_augreal 86
29/5/09 10:50:30
Work with augmented reality 87
www.computerarts.co.uk July 2009_ Computer Arts Projects
11
Scroll to the fourth section: Methods. Part A prepares the webcam to capture video. Recent
versions of Flash Player feature great webcam support – if a webcam is detected, it will automatically prompt
the user to enable it as these lines of code execute. The ‘video’ object is the window onscreen that displays
your video, while the ‘camera’ object records and feeds live footage into that window.
Anatomy of an ActionScript 3.0 class
As AugmentedReality.as demonstrates, AS3 class files are structured into several sections – each with a
specific role in the operation. We’ve broken the 200+ lines of code in this project into bite-sized chunks, and inserted
explanatory comments within the code wherever possible. To recap, the main four sections are:
1) Imports: There’s no sense in reinventing the wheel, and classes will typically pull in functions from existing libraries – which
you may not have coded yourself. As you’ll see from Step 08, each class or package is imported to the class with a simple line of
code. In this project, we use libraries from Flash, FLARToolkit, and Papervision3D.
2) Class Properties: This is where you declare any static variables, which can be private or public – available to be shared with
other classes. Arguments and any local variables are declared later, within the Constructor and Method sections. Only private
variables are needed in this project (see step 09), as there are no outside classes that need to be granted access.
3) Constructor: This function only runs once, so it’s an ideal place to initialise your class. Its name must match the class name,
in this case ‘AugmentedReality’ (step 10, line 93). In this project, we place code here to set up the camera, FLARToolkit and
Papervision3D, and to prepare a repeating loop to handle marker detection.
4) Methods: These tend to define particular functions, often including code that can be reused throughout the application. One
function in this project runs continually – to detect the marker and update the 3D model – and just four methods are used in
total, outlined in steps 11, 13, 15 and 17.
12
With just those five lines of ActionScript in Step 11, Flash will relay the live
footage it sees onto your screen.
14
Much like Schwarzenegger’s Terminator, your project can now read its
surrounding environment, visually scan and detect relevant objects, and make
dynamic calculations based on that information. As we mentioned in step 03, it’s
important to keep the backdrop as simple as possible when showing your marker to
your webcam, as very complex scenes may confuse it.
13
The cornerstone of this project is marker detection, and this is covered in Part B of the Methods
section. As your marker graphic is detected, its position, rotation, and scale in the user’s physical space must
be calculated on the fly so your 3D model can move with it. Within each individual still-frame of webcam
footage, FLARToolkit will search for the predefined marker you created in step 03 (lines 146-149). This
detection process uses the ‘bitmapData’ object to store only the latest still-frame of video at any given time
– which helps to conserve processing power.
CAP125.tut_augreal 87
29/5/09 10:50:30
88 Project four New technologies
Computer Arts Projects _July 2009 www.computerarts.co.uk
15
Part C of the Methods section covers the 3D setup – typical Papervision3D,
if you’re familiar with it. The ‘basicRenderEngine’ object handles the bulk of the vector
maths, converting your model’s 3D geometry into 2D to display it onscreen (lines
162-168). Your Collada 3D model is then loaded and inserted into the 3D scene as the
‘viewport’ object (lines 173-179), which is then added to the stage on top of your
webcam ‘video’ object from step 11. The result is a seamless blend of 2D and 3D.
18
And here’s the end result. After just over 200 lines of code, your application can now superimpose a
3D model on live webcam footage, and scale and rotate it in real time. For some exciting applications of this
technology, check out our interview with Poke London.
16
Here’s our Collada model Hummer, displayed inside Flash.
17
Finally, scroll down to Part D of the Methods section. This completes the
project by defining the repeating detect-and-update routine that we started in step 10.
To keep the effect as smooth as possible, the function is called 30 times per second.
Each time this happens, the ‘bitmapData’ object from step 13 inserts the latest
still-frame from the webcam footage (lines 281-287). FLARToolkit then scans the
still-frame, and if it detects the marker, feeds its current position, rotation, and scale
into the ‘TransformMatrix’ object (line 210). Finally ‘flarBaseNode’ – the object holding
the 3D model – is updated to match that position (lines 211-212), so the model appears
to be at exactly the same position in physical space as the marker you’re holding.
Embedding with Adobe Flex
Flex utilises a special type of code called the Embed metatag, which you may have
spotted in step 09 (lines 58-62). Uniquely, these tags allow you to load a particular property
from an external file at compile-time (as the project is published) rather than at run-time
(while the user interacts with the application) – in other words, the necessary code comes
baked-in to your SWF file. Flash CS4 now supports this great feature too, with a bit of help
from the flex.swc file that’s included with this project.
CAP125.tut_augreal 88
29/5/09 10:50:31