[转载]Augmented reality using a webcam and Flash | Adobe Developer Connection

Augmented reality (AR) is a field of computer science that involves combining the physical world and an interactive, three-dimensional virtual world. This article takes a look at augmented reality, its current uses, and its future potential. Then you’ll dig in and see how to apply this exciting technology using Adobe Flash CS4 Professional, Adobe Flash Player 10, and a webcam.

While mainstream audiences are now becoming aware of AR, it is not new. Its background is intertwined with decades of computer science development. Virtual reality (VR), AR’s more familiar counterpart, is the replacement of the user’s physical reality (particularly that which is experienced through sight and hearing) with a computer-generated reality. The idea of a virtual experience is exciting—creating entertaining and educational sensory encounters that do not exist in our everyday lives.

From a consumer standpoint, it seems that AR advances have come out of nowhere to surpass VR advances. The acceleration in AR technology is due to two major factors: First, users are still experiencing reality, so believability is easier to achieve. Adding simple graphics (such as text or simple shapes) and color effects (such as night vision or thermal vision) to reality creates a better user experience. The user is still seeing a mostly familiar world. Second, this more subtle use of computer graphics is less expensive with today’s technology, making it more feasible than VR.

Applications of augmented reality

The video game industry has released major augmented reality products for more than a decade. The EyeToy for Sony PlayStation 2 and 3 takes input from an inexpensive video camera and composites the live video feed with CG onto the TV screen. This toy detects the user’s body position in front of the camera as an alternative input method to the typical joystick or gamepad, deepening the user’s immersion into the game world. Detection schemes continue to advance, allowing for more engaging interaction.

There are AR applications outside of console games, including military and consumer products, too. Night-vision goggles and targeting-assistance technology help marksmen in battle, and children’s stories come to life with AR-enhanced books. The uses are vast.

With AR-enhanced books, each page of a real-world book is outfitted with a marker—a pattern that is detectable by AR. As the child turns each page, a computer is able to place a virtual 3D animation on top of a physical image printed on paper. In this article, you will learn to create this type of AR experience. Let’s dig in!

Augmented reality with Flash CS4

Let’s create a project in which a webcam will capture the user, holding a marker image in an arbitrary position, and the application will augment the webcam feed with a 3D model overlaid onto the marker’s position in a believable way. You can get this project up and running in less than an hour.

With Flash Player 10, developers have the most robust toolset yet for rich application development. Flash Player handles the heavy lifting such as video input, pixel-level image manipulation, and the heavy number crunching afforded with ActionScript 3. All you need is a webcam, a few open source libraries, and Flash CS4 to get the job done.

New project setup

Before coding a Flash CS4 project, you need to prepare the libraries and assets. Once the project is set up, most changes will happen within the document class (see Figure 1). To get started:

1. Create a new project in Flash CS4 by choosing File > New > Flash File (ActionScript 3.0), and save the project as AugmentedReality.fla inside an AugmentedReality folder on your desktop.
2. Create a document class by choosing File > New > ActionScript File, and save it as AugmentedReality.as within the AugmentedReality folder.
3. Set this file as the project’s document class to the AugmentedReality class file by choosing Window > Property > Class. The class file will be empty at this point.
4. Create a libs folder inside the AugmentedReality folder, and add it to the project’s library path by choosing File > Publish Settings > Flash > ActionScript 3.0 > Library Path > [Folder Icon].
5. Download the three libraries discussed in the next section, and add each SWC to the libs folder.
6. Create an Assets folder and add the 3D Collada model file and two AR data files.

External library setup

A great benefit of ActionScript3’s object-oriented programming model is that developers can importexisting libraries of code to save ramp-up time and vastly expand thedevelopment power of Flash. This implementation of AR takes advantageof three such libraries, each with a specific role in the project:

• Flex SDK code library: Processes the Embed metatag needed to import FLAR data files.
• Spark Project’s Flash Augmented Reality code library (FLARToolkit): Handles marker graphic detection.
• Papervision3D code library (PV3D): Handles the importing, positioning, and rendering of the 3D model.

Asset preparation

TheFlash Augmented Reality Toolkit (FLARToolkit) library holds much of themagic for this application. It will introspect the webcam image for aparticular graphic image and judge where in real space to map the 3Dmodel. The FLARCameraParams.dat file, which comes with the toolkit,will be used as-is. It corrects webcam distortion and helps the toolkitwith its general detection routine. The graphic marker imageinformation is held within the FLARPattern.pat file. The file is madeto work with a particular marker graphic image (see Figure 2). If youwant to use a different graphic, create a JPEG image with your favoritedrawing program, and use the FLAR Marker Generator (AIR, 322K) to generate a new pattern file.

The marker image is a graphic drawn, printed, and shown to the endapplication as it runs. FLAR, with help from the marker data file andparameters file, will detect this shape via webcam. To design agraphic, fit the desired shape within a white square that is centeredwithin a larger black square. Keep your shape simple for best results.

Notethe sun shape in the graphic. This helps the application detect whichway is up for the marker and assists its angle and rotation detection.Your design doesn’t need to have this shape per se, but it should beasymmetrical in some way for best results.

Also, you’ll needa 3D model in the Collada format that Papervision3D requires. Colladamodels can be created and exported using many popular 3D modelingtools, including the open source Blender3D.

The document class

TheAugmentedReality.as document class is where all of the project’s codelives. The following code shows the completed class. As you can see,much of the complexity of FLAR and Papervision3D for the project ishidden within the imported libraries and is therefore out of the way:

Stepping through the document class

Imports and properties

Startingat the top of the AugmentedReality.as document class, we see importstatements for the needed libraries. Within the Properties section,variables are declared. Most of this section is pretty standard, exceptthe Pattern and Params property declarations from lines 58–62. Thissection showing the Flex Embed metatag within Flash CS4 appears here:

Adobe Flex uses a special type of code called the Embed metatag.Embed tags, as they are commonly called, enable a property to beinitialized with a value loaded from an external file. A unique aspectis that the byte loading happens at compile time (as the project ispublished) rather than at runtime (while the user interacts with theapplication). The bytes for the loaded file are baked into the SWFfile. Flash CS4 now supports this feature as well with help from theflex.SWC, which is included in this project’s libs folder.

Class constructor

Sincethe constructor of a class is called only once, it is an ideal locationfor one-time-only setup code. Starting on line 93, the preparationmethods are called for each of the following: the webcam, the markerdetection, and Papervision3D. In addition to the setup methods, the loopToDetectMarkerAndUpdate3Dfunction is set up to run again and again. By listening to theENTER_FRAME event built into Flash, this loop will be called repeatedlyas long as the application is running.

Webcam setup

Flashfeatures great webcam support. As the application runs and a camera isdetected, Flash Player will automatically prompt the user to enable itas line 96 of the code executes. The Video object is the windowonscreen where the video will be shown, and the Camera object recordsand feeds its live footage to that Video object.

FLAR setup

Acornerstone of this project is marker detection. As the marker graphicis detected via webcam, its position, rotation, and scale in the user’sphysical space are calculated. This calculation will be used in thenext steps. Starting with line 97, FLAR will search each still-frame ofwebcam footage for the predefined marker graphic (Pattern object). Thedetection scheme uses an ActionScript BitmapData object, which containsonly the latest still-frame of video at any given time (the BitmapDataDraw function).

Papervision3D setup

For the mostpart, the 3D setup at line 98 is typical of any Papervision3D project.A BasicRenderEngine object will handle the bulk of the vector math,converting 3D geometry to 2D graphics for display on the screen. TheCollada 3D model is loaded and inserted into the 3D scene, and the viewinto that scene (ViewPort object) is added to the Stage (see Figure 3).Luckily this is all done for you. In the final project, this view islayered on top of the 2D webcam Video object shown earlier so thecomposite of 2D and 3D will look believable.

Repeated detect-and-update routine

As mentioned earlier, by using the ENTER_FRAME you ensure this loopToDetectMarkerAndUpdate3Dfunction on lines 102 will be called 30 times per second. Each time,the BitmapData object will copy in the latest still-frame from thewebcam footage. FLAR runs detection on the still-frame. If the markeris detected, FLAR returns updated position, rotation, and scale data(TransformMatrix object). The container (FLARBaseNode object) holdingthe 3D model is updated to that position so the model appears to be atthe same position in the user’s physical space as the marker. Theresulting effect is nothing short of amazing for first-time viewers.

Running the completed project

Printout the graphic marker. Show the marker to the running application viathe webcam (see Figure 4). The red line is for illustration purposes todenote successful initial marker detection. Much like ArnoldSchwarzenegger’s Terminator character, the project can now read in thesurrounding environment, visually scan and detect relevant objects, andmake calculations based on that information. It works best to have asimple backdrop of a solid wall in the application because complexscenes may confuse the detection scheme.

Next, seeing the marker graphic through the webcam, the applicationsuperimposes an animated 3D model onto the user’s physical space (seeFigure 5). The model now rotates and scales believably as the usermoves the marker.

Where to go from here

There you have it. Augmentedreality is within the grasp of all Flash developers. The FLARToolkitcan detect multiple markers to bring even more 3D into your world. Astechnology matures, you will be able to detect more complex graphicmarkers within more complex scenes. The choices are endless. What willyou choose to do with augmented reality: print and show a marker to anonline application as we’ve done here, buy and play with a real-worldAR-enabled toy, or detect and destroy like the Terminator?

To see another example of augmented reality, check out the MAX 2009 example with links to AR sessions at the conference.

Note: This article appeared originally in the Edge newsletter.