Augmented Reality with Unity 3D & Vuforia

Patricia Mu noz Menc a - Carlos III University, Madrid December 2013 patricia.mmencia@gmail.com

Abstract The aim of this paper is to describe the possibilities of Augmented Reality and its applications in different elds. The main purpose of this research is to bring forts an in-depth description of how tools like Unity 3D and Vuforia libraries can provide AR of numerous possibilities. To nish this paper, a mini game created with the integration of Unity 3D and Vuforia will be presented to evaluate this whole technology.

I. INTRODUCTION Augmented reality is a live, direct or indirect view of a physical, real-world environment whose elements are supplemented by computer-generated sensory input such as sound, video or graphics data. As a result, the technology works by enhancing ones current perception of reality[1]. In contrast, virtual reality replaces entirely the real world with a simulated one[2][3]. Normally, the augmentation occurs in real-time and in semantic context with environmental elements. With the advanced AR technology, the information about the surrounding real world of the user becomes interactive and digitally manipulable. Articial information about the environment and its objects can be overlaid on the real world[4][5][6][7]. Augmented reality techniques have been used in teleoperation in order to search for new solutions to roundtrip time-delayed. In the presence of time-delay, bilateral force reection frequently becomes unstable. Solutions based on predictive graphic displays break the force feedback loop, by enhancing visual feedback and superimposing a virtual representation of the slave robot, which is used in a sequential programming procedure. When a virtual robot is operated, subsequent motion is displayed in real-time to the operator on top of the videofeedback. This virtual robot acts as a visual predictor, and has no direct effect on the real one. Once the motion is done, replayed as much as needed and conrmed by the operator, the virtual robot controls the real one in a replay scenario where the operator cannot act. On this way, direct manipulation of the virtual robot and the real one are exclusive. Predictive graphic display development was pioneered by Noyes and Sheridan[8]. In gure 1 we have an example of predictive display. The AR-predictive display method has been recently democratized in many Internet-based teleoperation technologies. Current applications concern: multimedia, tele-surveillance, tele-diagnosis, tele-exploration, tele1

Fig. 1. Predictive display through augmented reality techniques for space teleoperation (The Jet Propulsion Laboratory, NASA)

supervision, etc. In these cases, the principle allows human operators to control a remote system which achieves desired programmed tasks, which may imply a modication of the environment (physical teleworking) or not (soft teleworking). In all cases, operator needs to be provided with feedback information while sending control commands. In addition, AR allows further functionalities. For instance, sophisticated mixed rendering can be generated by adequately associating computer synthetic pictures and live-video images and allows enrich feedback[9]. II. APPLICATION FIELDS Applications for augmented reality are broad. The military uses augmented reality to assist men and women makings repairs in the eld. The gaming industry is moving games outside like the old days, equipped with wearable head gear of course. A. Navigation Navigation applications are possibly the most natural t of augmented reality with our everyday lives. Enhances GPS systems are using augmented reality to make it easier to get from point A to point B. Information can be displayed on an automobile windshield indicating destination directions and meter, weather, terrain, road conditions and trafc information as well as alerts to potential hazards in their path. Aboard maritime vessels, AR can allow bridge watch-standers to continuously monitor important information such as a ships heading and speed while moving throughout the bridge or performing other tasks. For example, Wikitude Drive[10] for the Android operating system which is currently in beta brings the GPS into the 21st century. Using the phones camera in combination

Fig. 2. Wikitude Navigation is the rst pedestrian and car navigation system that integrates an augmented reality display and eliminates the need for a map. It has won multiple awards, and been hailed as a revolutionary step forward in the navigation and guidance eld.

with the GPS, the users see the selected route over the live view of what is in front of the car. We can see this display in gure 2. B. Sightseeing Augmented reality applications can enhance users experience when traveling by providing real time informational displays regarding a location and its features, including comments made by previous visitors of the site. AR applications allow tourists to experience simulations of historical events, places and objects by rendering them into their current view of a landscape. AR applications can also present location information by audio, announcing features of interest at a particular site as they become visible to the user. The augmentation of a live view of displays in a museum with facts and gures is a natural use of the technology. Total Immersion, a global leader in augmented reality, develops systems designed to enhance the experience of the museum attendee. Their interactive kiosk solution allows guests to interact with the display in 3D. They choose a model card such as a human heart and hold it under the camera. On screen, a perfect 3D representation of the heart appears allowing the guest to interact as if its a real heart. Out in the real world, sightseeing has been enhanced with augmented reality. Using a smartphone equipped with a camera, tourists can walk through historic sites and see facts and gures presented as an overlay on their live screen. These applications use GPS and image recognition technology to look up data from an online database. In addition to information about a historic site, applications exist that look back in history and show how the location looked 10, 50 or even 100 years ago. C. Medical There have been really interesting advances in medical application of augmented reality. For example, medical students use the technology to practice surgery in a controlled 2

environment instead of a real patient. AR can reduce the risk of an operation by giving the surgeon improved sensory perception. This technology can bring everything into a single view by being combined with MRI or X-ray systems, conforming a Mixed Reality[11]. See gure 3 of a project named Liverplanner, property of ICG. Mixed Reality, sometimes referred to as Hybrid reality (encompassing both Augmented Reality and Augmented Virtuality) refers to the merging of real and virtual world to produce new environments and visualizations where physical and digital objects co-exist and interact in real time. Neurosurgery is at the forefront when it comes to surgical AR applications. the ability to image the brain in 3D on top of the patients actual anatomy is very powerful for the surgeon. Since the brain is somewhat xed compared to other parts of the body, the registration of exact coordinates can be achieved. Concern exists surrounding the movement of tissue during surgery, which can affect the exact positioning required for Augmented Reality to work.

Fig. 3. Augmented Reality based liver surgery planning in action. The images were captured by a tracked camera and overlaid with the information about the virtual objects placed in space to visualize the virtual planning process.

D. Military In combat, AR can serve as a networked communication system that renders useful battleeld data onto a soldiers goggles in real time. People and various objects can be marked from the soldiers viewpoint with special indicators to warn of potential dangers. Virtual maps and 360 degree view camera imaging can also be rendered to aid soldiers navigation and battleeld perspective, and this can be transmitted to a remote command center. A typical example of Augmented Reality when it comes to military application is the Heads Up Display (HUD). It consists of a transparent display positioned directly in the ghters pilots view. Data typically displayed includes altitude, airspeed and the horizon line in addition to other critical data. The term heads-up comes from the fact that the pilots

dont have to look down at the aircrafts instrumentation to get the data they need. There is an example of this display in gure 4.

In the last 10 years there has been a lot of improvements of technology, resulting in better movement detection (for example, applicable to Wii, but also direct detection of the players movements[12]. One of the classication for augmented reality games is based on the physical space demanded by the game, separating games that require large areas from the ones that need limited and prepared areas, and others that use the advantage of the natural mobility of devices such as phones and PDAs and are independent of the area in which they are played.

Fig. 4.

The view through a Rockwell Collins head-up display.

There is a variant named Head-Mounted Display (HMD), that is used by ground troops. Critical data such as enemy location can be presented to the soldiers within their line of sight. This technology can also be used for simulations with training purposes. E. Sports and entertainment Augmented Reality has become common in sports telecasting. Sports and entertaining venues are provided with see-through and overlay augmentation through tracked camera feeds for enhanced viewing by the audience. Examples include the yellow rst down line seen in television broadcast of American football games showing the line the offensive team must cross to receive a rst down. AR is also used in association with football and other sporting events to show commercial advertisements overlaid onto the view of the playing area. Other examples include hockey puck tracking, annotations of racing car performance and snooker ball trajectories. Electronic games are a kind of computer application that has avid demand for innovation and for more engaging user experiences. So far the main focus of improvements in electronics games has been on graphics; more recently articial intelligence has been drawing more attention from researches and developers. Back in 1997, entertainment was already mentioned as one of the main applications of AR. At that time, however, it was mainly used in a new lm productions by the movie industry, to allow the actors see and interact with virtual elements in real time, which would later be inserted in the movie (with much better quality) through the chroma key technique, for instance. The gaming industry has beneted a lot from the development of this technology. Augmented Reality allows gamers to experience digital game play in a real world environment. 3

Fig. 5.

One of the Qualcomm demos working with Augmented Reality.

III. UNITY 3D Unity is a cross-platform game engine with a built-in IDE developed by Unity Technologies. It is used to develop video games for web plugins, desktop platforms, consoles and mobile devices. It grew from an OS X supported game development tool in 2005 to a multi-platform game engine.

Fig. 6. Unity is an integrated development environment for game creation used by over 1.2 million game developers all over the world.

The latest update, Unity 4.3.1, was released December 2013. It currently supports development for iOS, Android, Windows, BlackBerry 10, OS X, Linux, web browsers, Flash, PlayStation 3, Xbox 360, Windows Phone 8, and Wii U. Two versions of the game engine are available for download, Unity and Unity Pro. Unity is a game development ecosystem: a powerful rendering engine fully integrated with a complete set of intuitive tools and rapid workows to create interactive 3D and 2D content. It has thousands of quality, ready-made assets in the Asset Store and a knowledge-sharing community[13]. For independent developers and studios, Unitys democratizing ecosystem smashes the time and cost barriers to creating uniquely beautiful games. They are using Unity to

build a livelihood doing what they love: creating games that hook and delight players on any platform. A. Scripting The game engine scripting is built on Mono. Mono is a free and open source project led by Xamarin (formerly by Novell and originally by Ximian) to create an Ecma standard-compliant, .NET Framework-compatible set of tools including, among others, a C compiler and a Common Language Runtime. The stated purpose of Mono is not only to be able to run Microsoft .NET applications cross-platform, but also to bring better development tools to Linux developers[14]. Mono can be run on many software systems including Android, most Linux distributions, BSD, OS X, Windows, Solaris, and even some game consoles such as PlayStation 3, Wii, and Xbox 360. Programmers can use UnityScript, which is a custom language with ECMAScript-inspired syntax, referred to as JavaScript by the software; C or Boo (with a Phytoninspired syntax). In the 3.0 release of Unity, it ships with a customized version of MonoDevelop for debugging scripts[15]. B. Rendering The graphics engine uses Direct3D (Windows, Xbox 360, etc), propietary APIs (as Wii), OpenGL (Mac, Linux, Windows, PS3) and OpenGL ES (Android and iOS). There is support for bump mapping, reection mapping, parallaz mapping, screen space ambient occlusion, dynamic shadows using shadows maps, render-to-texture and full-screen postprocessing effects. With the release of 4.2 in 3rd August of 2013, Unity allows Indie developers to use Realtime shadows only for Directional lights, also the support of DirectX11 is added, which gives shadows a more pixel perfect resolution, textures to create 3d objects from grayscale, more facial graphics, smoother animations and a boost to the FPS.

production of appropriate levels of light and color within the image, Unity uses the ShaderLab language (see gure 7). The ShaderLab language support both declarative programming of the xed function pipeline and shader programs written in GLSL or Cg. A shader can include multiple variants and a declarative fallback specication, allowing Unity to detect the best variant for the current video card. If none are compatible, Unity fallback to an alternative shader that may sacrice features for performance. C. Platforms Unity supports deployment to multiple platforms, including Xbox One, Blackberry 10, Windows, Mac, Linux, Android, iOS, Adobe Flash, PS3, Xbox 360, Wii and Unity Web Player. Within a project, developers have control over delivery to mobile devices, web browsers, desktops, and consoles. It also allows specication of texture compression and resolution settings for each platform the game supports. D. Asset tracking Unity includes the Unity Asset Server, a version control solution for the developers game assets and scripts. It uses PostgreSQL as a backend, an audio system built on the FMOD library (with ability to playback Ogg Vorbis compressed audio), video playback using the Theora codec, a terrain and vegetation engine (which supports tree billboarding, Occlusion Culling with Umbra), built-in lightmapping and global illumination with Beast, multiplayer networking using RakNet, and built-in pathnding navigation meshes[16]. In addition, Unity has a Asset Store launched in November 2010. This store consists of a collection of over 4400 assets packages, including textures, materials, particles systems, music and sound effects, 3D models, tutorials and projects, scripting packages, online services and editor extensions. E. Graphics Unity also has built-in support for Nvidias PhysX physics engine (as of Unity 3.0) with added support for real-time cloth simulation on arbitrary and skinned meshes, thick ray casts, and collision layers. To have convincing physical behavior, an object in a game must accelerate correctly and be affected by collisions, gravity and other forces. Unitys built-in physics engines provide components that handle the physical simulation. With just a few parameter settings, the program lets create objects that behave passively in a realistic way (ie, they will be moved by collisions and falls but will not start moving by themselves). By controlling the physics from scripts, its easy to give an object the dynamics of a vehicle, a machine or even a moving piece of cloth[17]. 4

Fig. 7. Shaders are most commonly used to produce lighting and shadow in 3D modeling. This image illustrates Phong shading, one of the rst computer shading models ever developed.

Unity supports art assets and le formats from software like Maya, Blender, Adobe Photoshop and ZBrush. These assets can be added to the game project, and managed through Unitys graphical user interface[13]. To regulate the

IV. VUFORIA The Vuforia platform is a product of Qualcomm Technologies, and it enables augmented reality app experiences that are best in class and creative beyond denition. These experiences reach across most real world environments, giving mobile apps the power to see[18].

wants to track. The target resources can then be accessed by the mobile app in two ways:

Accessed from a cloud target database using web services Downloaded in a device target database to be bundled with the mobile app

For text recognition, the developer can specify a set of words that Vuforia can recognize, using the following text data sets:

Fig. 8. Vuforia is the software platform that enables the best and most creative branded augmented reality (AR) app experiences across the most real world environments, giving mobile apps the power to see.

Word lists in the VWL binary format (Vuforia Word List) Additional word lists, which can be specied via simple text les Optional word list lters (black or white lists) to explicitly include/exclude the recognition of specic words

Vuforia is a software platform which uses top-notch computer vision-based image recognition and offers a wide set of features and capabilities. It gives developers the freedom to extend their visions without technical limitations. With support for iOs, Android and, of course, for Unity 3D, the Vuforia platform allows to write a single native app that can reach the most users across a wide range of smartphones and tablets. An application developed for Vuforia will give the users a more compelling experience[18]:

The word lists and lter les are bundled with the mobile app and loaded at runtime using the Vuforia API. The Vuforia Engine provides a library (shared object libQCAR.so on Android, static library - libQCAR.a on iOS) that should be linked to the app. A Vuforia SDK-based AR application is composed of the following core components, that composes its architecture (see gure 10)[19].

Faster local detection of targets. User dened targets for run-time target generation. Better results in real world conditions: low light, partially covered target. Recognition and tracking of printed text. Optimizations thay ensure better and more realistic graphics rendered on the target.

This diagram provides and overview of the application development process with the Vuforia platform. See gure 9.

Fig. 10. Data ow diagram of the Vuforia SDK in an application environment.

A. Camera The camera component ensures that every preview frame is captured and passed efciently to the tracker. The developer only has to initialize the camera to start and stop capturing. Automatically, the camera frame is delivered in a device-dependent image format and size.
Fig. 9. The platform consists of the Vuforia Engine, inside the SDK, the Target Management System hosted on the developer portal and optionally, the Cloud Target Database.

B. Tracker The tracker component contains the computer vision algorithms that detect and track real-world objects in camera 5

A developer uploads the input image for the target that he

video frames. Based on the camera image, different algorithms take care of detecting new targets or markers and evaluating virtual buttons. The results are stored in a state object that is used by the video background renderer and can be accessed from application code. The tracker can load multiple datasets at the same time and activate them. C. Image Converter The pixel format converter converts from the camera format (e.g., YUV12) to a format suitable for OpenGL ES rendering (e.g., RGB565) and for tracking (e.g., luminance) internally. This conversion also includes downsampling to have the camera image in different resolutions available in the converted frame stack. D. Video Background Renderer The video background renderer module renders the camera image stored in the state object. The performance of the background video rendering is optimized for specic devices. E. Application Code Application developers must initialize all the above components and perform three key steps in the application code. For each processed frame, the state object is updated and the applications render method is called. The application developer must:

I. Word Targets The Vuforia SDK can recognize words and track them similarly to other types of targets, with two available recognition modes, Words and Characters. When using the Words reco mode, a word is recognized (and then tracked) if it belongs to a given Word List, which can be stored on the device and loaded by the application at runtime. Word lists can also be extended with additional words, and specic word lters can be loaded and applied via the SDK API. In addition, when using the Characters mode, any word resulting from an arbitrary character sequence (including numbers) is also detectable. V. M INIGAME APPLICATION During the course of this research, a mini game was developed with the goal of learning more about Unity 3D and its integration with Vuforias libraries and to be aware of all possibilities of this technology. Unity 3D offers a choice between three different programming language: JavaScript, C# and Boo. Each of them has its advantages, but I chose C#, which is fully object oriented and its syntax is similar to Java. The game consist of shooting the ghosts that appear around the marker, which is located below a cloud of dust. If the ghost disappears the player earns points; if instead the ghost goes away the player looses a life. Its a game with very simple dynamics, designed for beginners using Unity 3D; see gure 11.

Query the state object for newly detected targets, markers or updated states of these elements Update the application logic with the new input data Render the augmented graphics overlay

F. Device Databases Device databases are created using the online Target Manager. The downloaded device target database assets contain an XML conguration le that allows the developer to congure certain trackable features and a binary le that contains the trackable database. These assets are compiled by the application developer into the app installer package and used at runtime by the Vuforia SDK. G. Cloud Databases Cloud databases can be created using the Target Manager or using the Vuforia Web Services API. Targets are queried at runtime of the application using the cloud recognition feature that performs a visual search in the cloud using sent camera images. In addition to the target data, the provisioned targets can contain metadata which is returned upon query. H. User-Dened Targets A fundamentally different supported approach is the userdened targets feature. Rather than preparing targets outside of the developer, this feature allows for creating targets onthe-y from the current camera image. A builder component is called to trigger the creation of a new user-target. The returned target is cached, but retained only for a given AR session. 6 B. Game objects Developing an application in Unity 3D is very objectoriented, where the object is here named GameObject. GameObjects are containers for all other Components, and all object in our game are inherently GameObjects[20]. Aside from being a container for the components, GameObjects have a tag, a layer, a name and different scripts to control
Fig. 11. The principal aspect of this minigame application.

A. Importing Vuforia Its necessary to import the package downloaded from the webpage of Vuforia, including its libraries to the project in the menu Asset - Import custom package.

the behavior of these components. In this mini game, we have the following components (see gure 12).

Fig. 14. The scripts created during the development of the ghost mini-game

E. Prefabs
Fig. 12. For the development of this game we need these GameObjecs. Both the ghost and the shot only appear like clone of prefabs when must appear in the screen.

ARCamera, backgroundCamera and Directional light are specic to the game display. The shot and the ghost only appear on the list when they are on the screen, if its time to create new ghosts or the player has shot. C. Animation Animation Clips are the smallest building blocks of animation in Unity. They represent an isolated piece of motion, such as RunLeft, Jump, or Crawl, and can be manipulated and combined in various ways to produce lively end results. In this mini-game, animation clips have been used just in the cases of very simple motion, like the seesaw movement of the ghost around the marker (see gure 13).

A Prefab is a type of asset a reusable GameObject stored in Project View. Prefabs can be inserted into any number of scenes, multiple times per scene. When you add a Prefab to a scene, you create an instance of it. All Prefab instances are linked to the original Prefab and are essentially clones of it. No matter how many instances exist in your project, when you make any changes to the Prefab you will see the change applied to all instances. We can see the ghost design in Blender on gure 15.

Fig. 15.

The ghost model designed in Blender.

Its important build as prefab those objects that the program will create and delete instances of during the game. In this case, we create the ghost, the shot and the explosion that appear when the shot reach a ghost as prefabs. F. Colliders
Fig. 13. The simple movements can be done with animation clips.

D. Scripts To implement the rest of the actions in the game, different scripts in C# are used attached to the objects. We can see these scripts on gure 14. 7

A bf collider, in basic terms, is a wrapper that makes a particular object solid and, therefore, it can collide with other objects that also have a collider. It consists, rst, of a certain shape (which doesnt have to be identical to the objects shape, although its preferable to be as similar as possible to cause credible collisions) and, secondly, a specic physical material. We refer to the material as the rebounding ability

or friction of the object, and not to the physical material of the object. According to their shape, we can divide them into two subgroups: mesh colliders and primitive colliders:

We create a mesh collider when we import a mesh from a 3D application like Blender. Selecting the mesh in the Project view, before importing it, there is a check box of generate colliders. When we mark this box and import the mesh,Unity creates a collider that will have the same shape as the mesh. This type of collider has an important limitation: if two mesh collider crash too fast the collision wont be detected. The reason is that it takes a lot of resources to calculate this type of collision (with a complex mesh). The primitive collider requires to assign a prefabricated collider to a mesh, choosing between the following primitive shapes: sphere, box, capsule or wheel. Although this second option is more laborious, it is used in the minigame because both the ghost and the shot have a very basic geometry.

Fig. 17.

Minigame in execution.

The future of augmented reality is clearly bright, even as it already has found its way into our cell phones and video game systems. R EFERENCES

VI. CONCLUSION The game can be exported to different platforms, provided we have the appropriate SDK. The build settings of Unity are represented on gure 16. In this case, the game is built for Android, because its the most immediate application of use due to the position of the AR Camera in the game. On the gure 16 we see the Unity build settings.

Fig. 16.

Build setting for Android.

The nal aspect of this project is reected on gure 17. Once analyzed deeply augmented reality, imagine the possibilities: you may learn things about the city youve lived in for years just by pointing your AR-enabled phone at a nearby park or building. If you work in construction, you can save on materials by using virtual markers to designate where a beam should go or which structural support to inspect. Paleontologists working in shifts to assemble a dinosaur skeleton could leave virtual notes to team members on the bones themselves, artists could produce virtual grafti and doctors could overlay a digital image of a patients X-rays onto a mannequin for added realism. 8

[1] Graham, M., Zook, M., and Boulton, A.,Augmented reality in urban places: contested content and the duplicity of code in Transactions of the Institute of British Geographers, DOI: 10.1111/j.14755661.2012.00539.x 2012. [2] Steuer, Jonathan. Dening Virtual Reality: Dimensions Determining Telepresence, Department of Communication, Stanford University. 15 October 1993. [3] Introducing Virtual Environments National Center for Supercomputing Applications, University of Illinois. [4] Chen, Brian X. If Youre Not Seeing Data, Youre Not Seeing, Wired, 25 August 2009. [5] Maxwell, Kerry. Augmented Reality, Macmillan Dictionary Buzzword. [6] Augmented reality-Everything about AR, Augmented Reality On. [7] Azuma, Ronald. A Survey of Augmented RealityPresence: Teleoperators and Virtual Environments, pp. 355-385, August 1997. [8] M.V. NOYES and T.B. Sheridan. A novel predictor for telemanipulation through a time delay. In Anual Conference Manual Control, Moffet Field, CA, 1984. [9] Manuel Ferre, Martin Buss, Rafael Aracil, Claudio Melchiorri and Carlos Balaguer. Advances in telerobotics.Springer Tracts in Advanced Robotics, Volume 31, 2007. [10] http://www.wikitude.com/ Last access on 4/12/2013. [11] http://www.icg.tugraz.at/research/interdisciplinary/liverplanner/summary Last entry on 04/12/2013 [12] Leino Olli, Wirman Hanna, Fernandez Amyris. Extending Experiences. Structure, analysis and design of computer game player experience, Lapland University Press, 2008. [13] http://unity3d.com/unity, last entry on 04/12/2013. [14] http://docs.unity3d.com/Documentation/Manual/ Last entry on 04/12/2013. [15] Getting started with Mono Develop. Unity Technologies. Last entry on 04/12/2013. [16] Asset Server (Pro Only). Unity Technologies. Last entry on 05/12/2013. [17] Physics. Unity Technologies. Last entry on 05/12/2013. [18] Getting started, Developer vuforia. Last entry on 05/12/2013. [19] Development guide, Developer vuforia. Last entry on 05/12/2013. [20] Components, docs.unity3d.com. Last entry on 07/12/2013.