These massive photo sharing systems provide an ever growing imagery dataset that could be used to create reconstruct real world scenes in 3D.  In recent years, researchers have been processing images from these sites using computer vision techniques to reconstruct real world scenes in 3D.

Photosynth

Recent progress in this area can be traced to when Microsoft, in cooperation with the University of Washington, publicly released Photosynth.  This application processes images and creates a 3D model and point cloud from a user’s uploaded images.   The user experience is akin to viewing a slideshow of photos placed in their proper spatial context.

Photosynth introductory video:

Microsoft researcher Blaise Aguera y Arcas’ Photosynth demo at TED 2007:

Reconstruction From Web Photo Collections

With Photosynth, individual users upload their own photos and are encouraged to follow shooting guidelines to create the best results.  The researchers behind Photosynth later extended their previous work by developing a system that used unstructured sets of photos uploaded by thousands of different users.  A series of computer vision techniques were used to analyze the millions of Flickr photos of major tourist destinations such as Rome and Venice.   Ultimately, they were able to reconstruct 3D point clouds of the Coliseum, the Trevi fountain and other frequently photographed European landmarks.

For more info on this project, go here: Building Rome in a Day.

In 2010, researchers at the University of North Carolina at Chapel Hill released a paper describing a technique for extracting 3D models from community photo databases which was significantly faster than other methods.  Their approach can analyze 3 million Flickr images of Rome and reconstruct the major landmarks in less than 24 hours on a single PC.  They also applied the technique to photos taken of Berlin.  If video footage of a location is available, their technique can reconstruct 3D models even faster

Project demo video:

What’s Next?

Work in this area will accelerate in coming years as smartphone ownership increases along with improvements in the imaging and geolocation capabilities of these devices.  For example, phones are now being equipped with gyroscopes and compasses in addition to GPS, providing richer data for 3D reconstruction algorithms.  Bubbli is a stealth start up that is apparently seeking to work in this area by coupling smart phone image and location data with computer vision algorithms. They will be interesting company to watch as more of their product plans are revealed.

These projects are laying the groundwork for the emergence of 3D online maps which can be continually updated by millions of users uploading geo-located photos and videos from around the world. We are approaching a point where the combination of digital imaging, computer vision, and location sensor technology enables us to capture entire 3D scenes rather than just photographs.  Ultimately, users will be able to experience a form of virtual time travel where they can see a location from various viewpoints at different points time.

Reference

Jan-Michael Frahm, Pierre Georgel, David Gallup, Tim Johnson, Rahul Raguram, Changchang Wu, Yi-Hung Jen, Enrique Dunn, Brian Clipp, Svetlana Lazebnik, Marc Pollefeys, “Building Rome on a Cloudless Day,” ECCV 2010. (view)

Michael Goesele, Jens Ackermann, Simon Fuhrmann, Ronny Klowsky, Fabian Langguth, Patrick Mücke, Martin Ritz, “Scene Reconstruction from Community Photo Collections,” Computer, vol. 43, no. 6, pp. 48-53, June 2010. (IEEE link)

Sameer Agarwal, Noah Snavely, Ian Simon, Steven M. Seitz and Richard Szeliski, “Building Rome in a Day” International Conference on Computer Vision, 2009, Kyoto, Japan. (view)

Using mobile apps like those from Ustream and Qik, a few are even live streaming the show. Taking note of this behavior, a compelling opportunity exists to use these devices to create opportunities for fans to interact with the on stage artist and become participants in the performance.

One of the most ambitious efforts in this area is being led by techno DJ and producer Richie Hawtin. An established music technology innovator, Hawtin gained attention in recent years for tweeting his DJ sets. As part of his current tour performing as his alter ego, Plastikman, he and his collaborators have released the “SYNK” application for the iPhone and the iPod Touch.

During performances audience members can use the app to see real time data from percussion gear, live video from an on stage perspective, and the application will synchronize phone visuals with activity on the stage LED wall.  More details on the app can be found here.

Here’s video of the “Synkotik” mode of the app.

In this LA Weekly Coachella interview, Richie Hawtin describes the motivation for integrating SYNK into his performance.

“I think a great concert experience is when unexpected things happen and when people come together to experience something that goes beyond music. The idea was to throw something into the mix that would heighten that experience. With the app we’re able to send people messages, we’re able to actually synchronize things that are happening on stage with the iPhone apps, we’re even actually allowing people to actually have access to my set-up, you know, triggering sounds. So everyone talks about technology democratizing everything. With our app we will actually blur the lines between the performer and the audience, actually forcing questions of, who is the performer, who is being in control, It is an experiment.”

In addition to the work of Hawtin and his collaborators, it should be noted that there are other efforts in this area, such as Eyebeam’s Open Video Sync project.  It aims to create an open source codebase which will allow multiple iPhones to synchronously playback video using Bluetooth or Wifi transmitted timecode.

Using smartphones apps for interactive concerts is a concept that is still in its infancy.  It is clearly a powerful avenue for tech savvy artists to push the boundaries of live performance.  Furthermore, by creating a way for audiences to participate in the performance, fans are given an additional incentive to buy tickets, an opportunity which cannot be overlooked in today’s music industry where artists are increasingly dependent upon live show revenues.

In 1997, I attended a brilliant presentation by wearable computing pioneer Dr. Thad Starner that made me aware of how this vision would be realized.  At the time, Thad wore a PC/104 based computer equipped with a “Private Eye” head worn display, a twiddler chorded keyboard, and a CDPD wireless internet connection. With a series of demonstrations, he illustrated the concept of contextually aware computing in which knowledge of location, time, and past user behaviors can be leveraged to better assist a person in completing their tasks. The idea is that through contextual information and a growing body of knowledge of a user’s habits, a computer interface can evolve to fit the user as opposed to the user having to adapt to a static interface. Over time, he described how such an interface could learn enough about an individual to become a “digital doppelganger” which could independently handle a number of one’s routine responsibilities. As an example, he described a scenario in which the time of year is December, and your wearable computer uses its knowledge of your gift buying habits to act on your behalf to complete all of your Christmas shopping for you.

Millions of us now carry contextually aware computing devices in the form of smartphones. Consequently, these devices are a mass market platform for advancing the notion of the digital doppelganger, now commonly referred to as a virtual personal assistant (VPA). Last fall, Clark Dodsworth, pointed me to a new mobile application called Siri which I see as the next step in VPA development. Siri is notable in that it uses a conversational interface that accepts either typed or spoken word input. It goes beyond traditional speech to text systems in that it can utilize natural language processing techniques to derive a person’s intent. Context from location, time, and user history is used to select from a variety of web based services capable of carrying out the requested task. For instance, if I say “I am hungry,” the system first determines that I am looking for restaurants. Next, using the phone’s geo-location capabilities, it determines that I am in Santa Monica, California, and then lists restaurants within walking distance along with details provided from a variety of web services.

This short video of demonstrates Siri’s current features.

Two decades ago, Apple produced a now famous video called “Knowledge Navigator” which depicted a future virtual personal assistant system. In this keynote from the Semantic Technology Conference, Tom Gruber introduces Siri and compares its features to the ideas presented in the Knowledge Navigator video.

It should be noted that beyond making existing information services easier to use, virtual personal assistants like Siri will also play an important role in winning over late adopters who have not been fully utilizing online search and recommendation tools.

Siri is without a doubt a major step in the fulfillment of a vision for computing set forth decades ago, and I look forward to seeing additional innovative VPA systems in the coming years.