These massive photo sharing systems provide an ever growing imagery dataset that could be used to create reconstruct real world scenes in 3D.  In recent years, researchers have been processing images from these sites using computer vision techniques to reconstruct real world scenes in 3D.

Photosynth

Recent progress in this area can be traced to when Microsoft, in cooperation with the University of Washington, publicly released Photosynth.  This application processes images and creates a 3D model and point cloud from a user’s uploaded images.   The user experience is akin to viewing a slideshow of photos placed in their proper spatial context.

Photosynth introductory video:

Microsoft researcher Blaise Aguera y Arcas’ Photosynth demo at TED 2007:

Reconstruction From Web Photo Collections

With Photosynth, individual users upload their own photos and are encouraged to follow shooting guidelines to create the best results.  The researchers behind Photosynth later extended their previous work by developing a system that used unstructured sets of photos uploaded by thousands of different users.  A series of computer vision techniques were used to analyze the millions of Flickr photos of major tourist destinations such as Rome and Venice.   Ultimately, they were able to reconstruct 3D point clouds of the Coliseum, the Trevi fountain and other frequently photographed European landmarks.

For more info on this project, go here: Building Rome in a Day.

In 2010, researchers at the University of North Carolina at Chapel Hill released a paper describing a technique for extracting 3D models from community photo databases which was significantly faster than other methods.  Their approach can analyze 3 million Flickr images of Rome and reconstruct the major landmarks in less than 24 hours on a single PC.  They also applied the technique to photos taken of Berlin.  If video footage of a location is available, their technique can reconstruct 3D models even faster

Project demo video:

What’s Next?

Work in this area will accelerate in coming years as smartphone ownership increases along with improvements in the imaging and geolocation capabilities of these devices.  For example, phones are now being equipped with gyroscopes and compasses in addition to GPS, providing richer data for 3D reconstruction algorithms.  Bubbli is a stealth start up that is apparently seeking to work in this area by coupling smart phone image and location data with computer vision algorithms. They will be interesting company to watch as more of their product plans are revealed.

These projects are laying the groundwork for the emergence of 3D online maps which can be continually updated by millions of users uploading geo-located photos and videos from around the world. We are approaching a point where the combination of digital imaging, computer vision, and location sensor technology enables us to capture entire 3D scenes rather than just photographs.  Ultimately, users will be able to experience a form of virtual time travel where they can see a location from various viewpoints at different points time.

Reference

Jan-Michael Frahm, Pierre Georgel, David Gallup, Tim Johnson, Rahul Raguram, Changchang Wu, Yi-Hung Jen, Enrique Dunn, Brian Clipp, Svetlana Lazebnik, Marc Pollefeys, “Building Rome on a Cloudless Day,” ECCV 2010. (view)

Michael Goesele, Jens Ackermann, Simon Fuhrmann, Ronny Klowsky, Fabian Langguth, Patrick Mücke, Martin Ritz, “Scene Reconstruction from Community Photo Collections,” Computer, vol. 43, no. 6, pp. 48-53, June 2010. (IEEE link)

Sameer Agarwal, Noah Snavely, Ian Simon, Steven M. Seitz and Richard Szeliski, “Building Rome in a Day” International Conference on Computer Vision, 2009, Kyoto, Japan. (view)

• Apps in the Living Room: An increasing number of televisions are connected to the internet.  Many of these have pre-installed applications to access YouTube, Netflix, Pandora, and other popular services.  Internet connected television platforms from Boxee, Google, Yahoo,  LG, and others make it possible for viewers to install additional applications by browsing app directories or bookmarking TV optimized websites.   It is an open question of which “lean back” apps will prove popular with consumers.  In the next twelve months we will have a better idea of the types of apps that are right for living room versus those that are better suited for single users on a PC or mobile device.

• Further Definition of the New Music Industry: Overall, the music business will continue its ongoing process of consolidation.  The notion of owning music in the form of a CD or a digital download will continue to erode, particularly for Generation Y, being replaced by cloud driven subscription models.  Internet data driven charts such as those powered by BigChampagne and the Echonest will rise in influence as sales and airplay based charts lose relevance.  Many of my thoughts are echoed in this Hypebot interview with Music 3.0 author Bobby Owsinski.

• Interactive Retail: In 2010, shoppers used smartphones to compare prices, earn coupons, and even make purchases.  This trend was encouraged by services like Foursquare, SCVNGR, and the use of QR Codes, Microsoft Tag, and other visual barcode systems.  As smartphone ownership continues to increase in 2011, marketers and advertisers will further use these technologies to bring new levels of interactivity to the retail experience.  We can expect  more companies to offer custom applications similar to  this one from the Carl’s Jr. and Hardee’s restaurant chain.

• More Tablets: By all accounts, the iPad was an enormous success.  This year more Android based tablets are expected to be released along with ones  from Blackberry and Palm.  With a new iPad being released this year, competing tablets must be aggressively priced and provide compelling app offerings to gain significant market share.

• 3D Backlash: 3D was heavily pushed by the film and consumer electronics industries this past year.  There was an over abundance of 3D film releases in 2010, and a number exhibited poorly implemented stereoscopic effects.  Consequently, movie goers will be more reluctant to pay extra to see a film in 3D.   Though 3D capable televisions were touted to consumers, there is a lack of 3D content for these devices.  Unless the content availability issue is resolved, 3D television will be of interest to only the very early adopters.

• Kinect Hacks: For years, researchers have utilized computer vision to build gesture based interfaces.  An excellent example is seen in the M.I.T. Media Lab’s Smart Rooms projects in the 1990′s.  Depth sensing cameras could cost in the $30,000 range, a price which prevented many developers from exploring this area of interface design.  The Microsoft Kinect is an amazing piece of hardware in that it provides a highly functional depth sensing camera system for only $150.00.  Realizing its potential, hobbyists and researchers have quickly built up a thriving Kinect hacking community in similar fashion to what occurred with the release of the Wii.  With open source efforts now springing up to support the Kinect, this community will continue to thrive and innovate in 2011.  Their work will hopefully inspire and guide the design of innovative gesture driven games and applications for the mass market.

Conference highlights include keynotes by Bruce Sterling and Will Wright among others.  On Thursday I will be giving a talk titled “Augmented Reality in Music Entertainment: Then and Now.”

You can receive the reduced $195 registration price – $200 off the $395 normal price – by using Discount Code: E195 during registration. This fee includes both conference days, lunch, reception and more.

To get a sense of the richness and depth of content, take a look at the ARE2010 Schedule – it will be THE must-attend AR event of the year: 2 days, 3 tracks, 40 AR companies; with keynotes:

- Bruce Sterling – (The AR Prophet!)
- Will Wright – (Spore and the Sims)
- Jesse Schell – (The Gamepocalypse)
- Blaise Aguera y Arcas – (Microsoft, BIng Maps & AR)
….and 80 other speakers! AR fans from around the world are registering and will be there, will you?

This event will be a unique opportunity to see the “state of play” of the emerging augmented reality industry.  Special events include:

- A press conference for new product announcements by leading AR companies
- Launch pad for new Augmented Reality start ups – a judging panel with VCs and entrepreneurs
- Over 80+ AR speakers over two intense days
- “The Auggies” – competition for the best augmented reality demo
- Art and magic live performances by AR artists

All details, speakers, and registration at www.augmentedrealityevent.com.

Using mobile apps like those from Ustream and Qik, a few are even live streaming the show. Taking note of this behavior, a compelling opportunity exists to use these devices to create opportunities for fans to interact with the on stage artist and become participants in the performance.

One of the most ambitious efforts in this area is being led by techno DJ and producer Richie Hawtin. An established music technology innovator, Hawtin gained attention in recent years for tweeting his DJ sets. As part of his current tour performing as his alter ego, Plastikman, he and his collaborators have released the “SYNK” application for the iPhone and the iPod Touch.

During performances audience members can use the app to see real time data from percussion gear, live video from an on stage perspective, and the application will synchronize phone visuals with activity on the stage LED wall.  More details on the app can be found here.

Here’s video of the “Synkotik” mode of the app.

In this LA Weekly Coachella interview, Richie Hawtin describes the motivation for integrating SYNK into his performance.

“I think a great concert experience is when unexpected things happen and when people come together to experience something that goes beyond music. The idea was to throw something into the mix that would heighten that experience. With the app we’re able to send people messages, we’re able to actually synchronize things that are happening on stage with the iPhone apps, we’re even actually allowing people to actually have access to my set-up, you know, triggering sounds. So everyone talks about technology democratizing everything. With our app we will actually blur the lines between the performer and the audience, actually forcing questions of, who is the performer, who is being in control, It is an experiment.”

In addition to the work of Hawtin and his collaborators, it should be noted that there are other efforts in this area, such as Eyebeam’s Open Video Sync project.  It aims to create an open source codebase which will allow multiple iPhones to synchronously playback video using Bluetooth or Wifi transmitted timecode.

Using smartphones apps for interactive concerts is a concept that is still in its infancy.  It is clearly a powerful avenue for tech savvy artists to push the boundaries of live performance.  Furthermore, by creating a way for audiences to participate in the performance, fans are given an additional incentive to buy tickets, an opportunity which cannot be overlooked in today’s music industry where artists are increasingly dependent upon live show revenues.

In 1997, I attended a brilliant presentation by wearable computing pioneer Dr. Thad Starner that made me aware of how this vision would be realized.  At the time, Thad wore a PC/104 based computer equipped with a “Private Eye” head worn display, a twiddler chorded keyboard, and a CDPD wireless internet connection. With a series of demonstrations, he illustrated the concept of contextually aware computing in which knowledge of location, time, and past user behaviors can be leveraged to better assist a person in completing their tasks. The idea is that through contextual information and a growing body of knowledge of a user’s habits, a computer interface can evolve to fit the user as opposed to the user having to adapt to a static interface. Over time, he described how such an interface could learn enough about an individual to become a “digital doppelganger” which could independently handle a number of one’s routine responsibilities. As an example, he described a scenario in which the time of year is December, and your wearable computer uses its knowledge of your gift buying habits to act on your behalf to complete all of your Christmas shopping for you.

Millions of us now carry contextually aware computing devices in the form of smartphones. Consequently, these devices are a mass market platform for advancing the notion of the digital doppelganger, now commonly referred to as a virtual personal assistant (VPA). Last fall, Clark Dodsworth, pointed me to a new mobile application called Siri which I see as the next step in VPA development. Siri is notable in that it uses a conversational interface that accepts either typed or spoken word input. It goes beyond traditional speech to text systems in that it can utilize natural language processing techniques to derive a person’s intent. Context from location, time, and user history is used to select from a variety of web based services capable of carrying out the requested task. For instance, if I say “I am hungry,” the system first determines that I am looking for restaurants. Next, using the phone’s geo-location capabilities, it determines that I am in Santa Monica, California, and then lists restaurants within walking distance along with details provided from a variety of web services.

This short video of demonstrates Siri’s current features.

Two decades ago, Apple produced a now famous video called “Knowledge Navigator” which depicted a future virtual personal assistant system. In this keynote from the Semantic Technology Conference, Tom Gruber introduces Siri and compares its features to the ideas presented in the Knowledge Navigator video.

It should be noted that beyond making existing information services easier to use, virtual personal assistants like Siri will also play an important role in winning over late adopters who have not been fully utilizing online search and recommendation tools.

Siri is without a doubt a major step in the fulfillment of a vision for computing set forth decades ago, and I look forward to seeing additional innovative VPA systems in the coming years.

In an effort to help manage expectations regarding AR technology, I will briefly describe what works today while clarifying what we can expect in the future.

Augmented Reality Glasses are not Viable in the Near Term

When covering AR, a number of technology pundits have assumed that within the next few years, we can expect head mounted displays or augmented reality “glasses” to become the best display for AR applications. Without a doubt, it would be groundbreaking if a high quality AR display could be built into the form factor of sunglasses. Unfortunately, a lightweight, wide field of view, daylight readable, head mounted display (HMD) at mass market prices is not something we can expect to see in the next five years. I have either bought or used most head mounted displays sold commercially since 1992, and I have seen great strides over the years in HMD resolution, brightness, and power usage.  To illustrate where the technology now stands, here are the best that I have used.

-Fakespace Labs “Wide 5” : Easily, the most immersive head mounted display I have tried, it can be modified for video see through augmented reality applications.

-The “Advanced Helmet Mounted Display”:  Optical Research Associates, a firm that helped fix the Hubble telescope, designed the optics. It has an amazing field of view, it’s optical see through, and has great resolution. It is also very large and is intended for helmet mounting.

-eMagin Z800 3Dvisor: It has a relatively small form factor, it’s very affordable, uses OLED’s, and it can be adapted for video see through augmented reality applications. The field of view is limited.

No one has yet delivered a wide field of view display in a small package that approaches the footprint of sunglasses as the public expects. With significant engineering investments, innovative approaches from companies such as Lumus, ORA, and Digilens may show promise in solving this problem. In addition to optics issues, eye strain and other head mounted display ergonomics problems must be dealt with.  For a detailed overview of the current and future state of head mounted display technology, I suggest the following:

-The Coming Generation of Head-worn Displays (HWDs): Will the Future come to us through new Eyes? Kevin and Jannick Rolland-Thompson, James P. McGuire, and Ozan Cakmakci, Optical Society of America 2009 Annual Meeting, San Jose, CA Download PDF

-The Past, Present, and Future of Head Mounted Display Designs” by Jannick Rolland and Ozan Cakmakci (College of Optics and Photonics: CREOL & FPCE, University of Central Florida) Download PDF

Ultimately, there will be no single “ultimate” AR display. Instead, the type of display used for AR will be determined by the needs of the application. Current and future developers will have a range of options including smart phone screens, handheld tablets, desktop monitors, micro projectors, and AR glasses.

GPS and Compass is not Enough

There are a number of mobile augmented reality applications, namely Layar, Wikitude, and the Yelp Monocle, which use GPS and compass data to overlay graphic information on a live video view of the real world. Though these applications are novel and interesting, the data provided by a mobile device’s GPS and compass is simply not precise enough to deliver a quality AR user experience. Information overlays usually appear to wobble or bounce around the video view. Consumers will rarely use these applications after the novelty factor wears off. At best, these applications provide an alternative viewing mode for data that should be first presented on a 2D map or in a list. Here’s a well written piece from New Scientist commenting on the current state of mobile augmented reality applications: Augmented Reality Gets off to a Wobbly Start

Sensor Fusion is the Answer

The best approach to AR tracking and registration involves hybrid tracking and sensor fusion techniques which use computer vision technology in conjunction with GPS and compass data. This paper provides deeper insight.

-Azuma, Ronald T., Bruce R. Hoff, Howard E. Neely III, Ronald Sarfaty, Michael J. Daily, Gary Bishop, Vern Chi, Greg Welch, Ulrich Neumann, Suya You, Rich Nichols, and Jim Cannon. Making Augmented Reality Work Outdoors Requires Hybrid Tracking. Proceedings of the First International Workshop on Augmented Reality, (San Francisco, CA, 1 November 1998), 219-224. Download PDF

Mobile augmented reality applications using a variety of sensor fusion techniques have been prototyped over the years, and we will hopefully see applications leveraging them on the market soon.

AR Hardware Platforms for Today

Here are five AR hardware platforms that developers can utilize today for creating augmented reality applications.

-Webcams

In the past year, there has been an explosion of ad campaigns which deliver webcam based AR experiences through an internet site or a downloadable application. Most of these utilized the FLARToolkit. I’ve used it to develop a number of AR web demos, available via the links below.

-Augmented Reality Interactive Album Cover

-Augmented Reality Concert

-Augmented Reality Fashion Ad

-Augmented Reality Business Card

-Kiosks, Digital Signage, Window Displays

Webcam AR applications can be packaged as interactive kiosks, digital signs, or window displays installed in retail locations as seen in this excellent example from Lego and this one promoting the movie Coraline.

-3G Smartphones

3G smartphones have enormous potential for AR applications. Phones from manufacturers including Apple, HTC, and Nokia are equipped with a camera, GPS, compass, accelerometers, and 3D graphics capabilities. These devices have enormous potential for AR applications. Unfortunately developers are currently more limited by the restrictions of manufacturer API’s than by the hardware itself. For example, developers cannot currently release iPhone applications which directly perform vision based tracking on the camera’s videostream. Computer vision based AR applications that are currently in the app store rely on analyzing still frames which limits the AR experience. In 2010, hopefully these frustrating API barriers will be overcome, and we can see vision based AR applications proliferate on mobile devices.

-Game Consoles

Augmented reality first came to game consoles when Dr. Richard Marks invented the EyeToy for the Playstation 2. More recently, we have seen the release of AR applications for the Playstation 3 (PS3) with the Eye of Judgment and the Eyepet.  In 2010, Sony is expected to debut their motion controller which will further extend the PS3′s AR capabilites.  Here’s an article where Richard Marks answers questions about the motion controller along with a video showing a precursor running on the Playstation 2.

Microsoft’s heavily publicized Project Natal should provide a set of tools for authoring AR experiences for the Xbox 360.  Hopefully developers will be able to embrace AR and ingeniously incorporate it into their game designs.

-Tablets

Researchers have extensively used tablets to prototype AR applications proving the viability of the tablet as an AR platform.  Most of these demos were created using the last generation of tablets from HP and Toshiba.  Those these devices provided great functionality, they were not widely accepted by consumers.  Here’s a 2004 AR paper from George Klein et. al. which utilized the HP TC1000 tablet.  With its sleek design and form factor, Apple’s forthcoming tablet may have greater success. Apple’s tablet will reportedly be equipped with a camera which could make it another excellent AR platform. The process of porting an iPhone application to the Apple tablet is expected to be a straightforward process which could encourage the release of AR applications with versions for both devices.

Focus on Utility and Fun, not Novelty

After the “wow factor” wears off, there’s a risk that consumers may begin to dismiss AR as a gimmick rather than a technology that provides real value. Developers can prevent this situation from happening by using AR to design applications that focus on utility and fun rather than the novelty of seeing an object pop out of a marker.

A great example of a useful AR tool is the US Postal Service’s box measurement application.

Try it here.

This application turns a cereal box into an interactive game.  It’s a fun gaming experience which simultaneously promotes a new film.

Both of these applications provide unique value helping consumers understand that AR is a technology that is useful beyond just being an attention grabbing feature.

The Road Ahead

For those who have been a part of the AR community since its infancy and those who are just entering the field, these are very exciting times. After decades of research, mass market AR applications are finally viable and can be delivered on a variety of platforms. If developers, investors, analysts, and consumers can develop a real understanding of what AR can and cannot do, the future of AR technology is bright, and I look forward to its evolution as an innovative and inspiring medium for creativity, communication, and commerce.

Further background reading suggestions:

-Thoughts on AR, an excellent November 2009 post on the state of AR from  Dr. Blair MacIntyre

-The Ultimate Display: The seminal 1965 essay envisioning AR from Ivan Sutherland, the creator of Sketchpad, and  the father of computer graphics.

-Sutherland, I. E. 1968. A head-mounted three dimensional display. In Proceedings of the December 9-11, 1968, Fall Joint Computer Conference, Part I (San Francisco, California, December 09 – 11, 1968): Sutherland’s 1968 paper describing the first AR display.    Download PDF

Researchers at the University of California San Diego Computer Audition Lab have developed a system that could address this problem by allowing people to find music using descriptive words rather than artist names and song titles.  For instance, a person could enter the words “high energy guitars” or “romantic vocals” and then receive a list of tracks that match that description.

The USCD system is capable of ingesting songs and automatically tagging them with annotation data without human intervention. To provide accurate results, the system must first be taught to hear music and describe it using natural language.  The training process uses digital signal processing and machine learning algorithms to expose the system to a broad array of music along with the words people use to describe it.  For example, to be able to accurately identify music that is referred to as “driving rock”, the system must analyze a large number of driving rock songs and then identify signal patterns that make that particular style of song unique.

The researchers have been gathering training data through crowdsourcing using an innovative Facebook game called “Herd-It“.  In this game, users are played a song snippet and asked to associate descriptive words and phrases with it.  Users earn points based on how well their answers match those of previous players.  Here’s a video describing the game.

The research group’s latest work in improving automatic music analysis was recently presented at the 2009 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) in the paper “Dynamic Texture Models of Music,” by Luke Barrington, Antoni Chan, and Gert Lanckriet.

With the continuing decline of the radio DJ as taste maker, web based music search and discovery tools will become increasingly important. With further development, machine learning driven music search systems such as this one could provide an intuitive and compelling method for listeners to find music they will enjoy.

Here are video captures of the performances.

This promotion is an excellent example of using innovative technology and engaging content to capture audience attention. Simultaneously, it provides a unique avenue for artists to promote their music and live shows.

Augmented reality (AR) is a technology that has been around for quite a while, primarily in the academic and research domains. Recently, its mainstream presence has increased due to the development of the Flash version of the Augmented Reality Toolkit (FLARToolkit). The ARToolkit was developed by Dr. Hirokazu Kato with Dr. Mark Billinghurst at the University of Washington’s Human Interface Technology Lab (HITLAB) over ten years ago.

In 2000-2001, I led a team which modified the original C based ARToolkit to work on 3D accelerated desktop Windows PC’s. We used the toolkit to develop interactive augmented reality projections for the band Duran Duran’s live concert tour. Here’s a video of ARToolkit effects that we used in the live shows.

The project was documented in this presentation at the 2002 Augmented Reality Toolkit Workshop:

Jarrell Pair, Jeff Wilson, Jeff Chastine, Maribeth Gandy. “The Duran Duran Project: The Augmented Reality Toolkit in Live Performance”. The First IEEE International Augmented Reality Toolkit Workshop, 2002. Download PDF

Now that it’s possible, what will be the breakthrough applications for mobile live broadcasting?   I think the answer may lie in looking at the trend of celebrities using Twitter.  Twitter is popular with stars because it is simple and easy to maintain.  It can be almost spontaneously updated unlike a traditional blog or personal website.  Celebrities can easily enhance this fan communication channel using Ustream.tv, Qik, Kyte, or Flixwagon.  In particular, mobile broadcasting could be appealing to touring musicians who rarely have an opportunity to sit at a computer and send a well composed personal blog post. Rapper Soulja Boy has been an early adopter in this area by using Kyte’s mobile platform to keep his fans in the loop.  In a similar fashion, Lil Wayne is set to begin using Ustream.tv’s mobile application.  Mobile broadcasting is clearly a concept to keep an eye on over the next year.

Here’s Ustream.tv’s mobile demo video.