Giving Computers a Human-Scale Understanding of Space

Computer vision and Human-Computer Interaction are just about to hit their stride. Within the past 4 years, the real-time/robotics computer vision research community has made leaps and bounds - much of it out of the Active Vision Group at Oxford and the Robot Vision Group at the Imperial College of London. One of the first pieces of work that really started to impress was PTAM (Parallel Tracking and Mapping) by Georg Klein:



Full up markerless augmented reality had been a long time dream of many. But, few people knew actually how to do it. PTAM was the first system that showed promise that it could handle the rough conditions of real-time motion of a handheld camera.

Also from Oxford, Gabe Sibley and Christopher Mei started demonstrating RSLAM (relative simultaneous localization and mapping) which provides fairly robust real-time tracking over large spaces. The following video uses a head-mounted stereo camera rig:



Just in the past couple weeks, some new projects done with the help of Richard Newcomb show what happens when you combine this tracking ability with either a depth camera like Kinect, or try to do traditional reconstruction from the RGB. These projects are called KinectFusion (a Microsoft Research Cambridge project) and DTAM (Dense tracking and mapping) respectively.



The following video uses a normal RGB camera (not a Kinect camera):


It's important to remember that no additional external tracking system is used, only the information coming from the camera. Also, it's worth pointing out that the 6DOF position of the camera is recovered precisely. So, what you can do with this data reaches well beyond AR games. It gives computers a human-scale understanding of space.

This is pretty exciting stuff. It'll take a little while before these algorithms become robust enough to graduate from a lab demo to a major commercial product. I usually like to say that "people will beat the crap out of whatever you make, and quickly gravitate to the failure cases". But as this work evolves and people begin build useful applications/software on top, it'll be an exciting next few years.

UIST 2011 Student Innovation Contest

UIST 2011 is just a couple months away, and Microsoft Hardware is generously providing the toys again this year. This time it's a touch mouse that provides a full capacitive touch image (which is fairly unique). If you are a student, try to enter, win some prizes and get to meet a bunch of other people interested in interface technology.

Official Contest Page

Myth of the Dying Mouse

It's definitely not the most polished delivery I've made (ignite talks don't let you control your slides, which is very unsettling for me). But, here's a 5 minute ignite talk I recently gave entitled "The mouse and keyboard are NOT going away, and there's NO SUCH THING as convergence".

Kinect Projects - The First 5 Months

Since it's release in November of 2010, there have been thousands of projects use the Kinect camera from independent developers, artists, and researchers. This is just a short montage of a few that I have enjoyed seeing.

It Gets Better

Why your arms don't suck.

Oooooooo..... me likey:

It's rare I see a product demo video and say, "Man, I wish my life would be longer so I can see the amazing future we will have." While I am fairly certain the yet un-purchasable robot above will be the cost of a small house, it is hard to contain my techno-lust. Having worked with a 6DOF robot before, they can be deceptively hard to program and without running a manufacturing line - the immediate utility of owning such a device is debatable. However, if these do end up being adopted by some manufactures, it does potentially reduce both the time to design/produce and the cost to manufacture consumer products. While this means the already blinding rate which new products are released will continue to accelerate, it also means that the bar for producing mass manufactured devices will also come down. As companies adopt re-programmable manufacturing/assembly tools, creating a new product may eventually be a matter of loading new files into all the machines on the floor. I think that's an exciting future and perhaps one day the "Print" button on your computer may take on a much more powerful meaning.

A small educational comment about the arms of this robot. They appear to be 7 degree-of-freedom arms... which is actually the same number of degrees of freedom that your arms have. If you grab a pole, or place your hand on the wall... without moving your shoulder (or your hand), you still have some freedom over the position of your elbow. But why do we need 7 when objects in the world only have 6 degrees of freedom (x,y,z, yaw, pitch, and roll)? The 1 extra degree of freedom is what allows us to reach around obstacles. If we only had 6 degrees of freedom, there would be only 1 way to reach out to pick up an object. So any obstruction along that path would prevent us from getting our food or some tool we needed to survive. Arms that contain 7 degrees-of-freedom have a dramatically larger operating range increasing their utility in uncooperative environments like the real world. For some reason, I find it quite satisfying that there is a mathematical basis for the evolution of our arms.

The re-emergence of DIY vs Big Organizations

Wow! Limor (of Adafruit Industries) is on the cover of Wired. Wicked. Congrats, I am filled with envy. =o) Even if she does look a little bit Photoshop'd. =o/ But, nice homage!
It's great to see someone from the DIY community get such recognition. I haven't read the issue yet, but extrapolating from some of the phrases on the cover, I imagine it is a celebration of the re-emergence of independent tinkering.

I have actually put quite a bit of thought toward this topic having recently jumped back and forth between the DIY hobby culture, serious academic research, and massively funded commercial product development. I've had the fortune to observe people trying to make new and interesting things at extremely different scales...from $100 budgets to $100,000,000 budgets.

One thing that I find very consistent: good ideas come from anywhere. The biggest factor in predicting where good work will come from is "how much does this person actually care about what they are working on?" In fact, big budgets and a sense of entitlement can actually hinder the emergence of interesting ideas. Having the *expectation* to do really great work can lead people or organizations to develop tunnel vision on "big" ideas, and miss out on smaller ideas that end up having a lot of impact or dismiss seemingly silly approaches that actually end up working.

There's a really great TEDx talk by Simon Sinek that touches on this. He actually brings up a number of great points in his talk, but the one I want to highlight here is his anecdote about Samuel Pierpont Langley vs. the Wright Brothers in pursuit of powered flight. Langley represented the exceptionally well funded professional research organization, and the Wright Brothers were the scrappy passionate pair of DIY'ers. Today, we now know the Wright brothers as the ones who created the first airplane and most have never heard of Langley. Big investment is not a very strong predictor of valuable output. But, an individual's willingness to continue working on the same problem with very little to no pay... is a good predictor.

The great thing about the hacker community is that, generally, most of them fall into the later category. Independent developers and hobbists care enough to spend their own money to work on the projects they believe in. As a result, I'm finding that the delta in the quality of ideas from a well funded research group, and the independent community (in aggregate) is getting smaller and smaller by the month. Increasingly, the best "hobby projects" surpass the quality level of "true research" work in the same area. This startling lack of contrast (or sometimes inversion) becomes laughably evident when I am reviewing academic/scientific work submitted for publication on a project that uses Kinect, and then the newest Kinect hack pops up on Engadget that simply beats it hands down.

Now, I could simply make a kurmudgenly claim that the quality of professional/research/academic work has gone down. But, I actually don't think that's true. In my opinion, what is happening is that the quality of independent projects are getting better.... fast. Which, I think resonates with this observation of a "DIY Revolution".

But, why is this re-emergence happening now? Wasn't is just a few years ago people were lamenting about how "black boxed" consumer products had gotten, and that the good old days where you could open up a product and futz with the innards in a meaningful way were gone? What's changed to cause this apparent re-birth?

I have a theory.

My Theory about the Re-Emergence of the DIY community:
In the 90's and early 2000's, Moore's law was absolute king. The primary deciding factor in purchasing an electronic product was simply how fast it was. This meant an intense focus on tighter and tighter integration of components and all the functionality was disappearing into tiny little black chips that could not be accessed nor modified by mere mortals. But now, people barely talk about raw "megahertz" or "megabytes" anymore. General purpose computers have gotten "fast enough". We now want specialized kinds of computers: one that fits in our pocket, plays games in 3D, one shaped like a tablet, one that goes in our car, one that can go under water, or get strapped your snowboard and not break. We have reached a surplus in computing power that makes it affordable to build (and buy) devices for smaller and smaller needs. Our imagination for what to do with computing has simply not kept up with Moore's Law. So, we find more uses for more modest amounts of computing power. But, what does this have to do with the DIY community?

A byproduct of having such an immense surplus in computing, is that the tools you can buy within a hobbist budget have also gotten exponentially better in just the past 3-4 years, while the improvement in professional tools have been more modest. The difference in capability between the electronics workbench of a professional engineer and a hobby engineer is getting really really small. Kinect is an overwhelming example of this. The cost of a high quality depth camera dropped nearly 2 orders of magnitude overnight. As a result, hobbists are out pacing many professionals in the same domain simply due to sheer parallelism. Perhaps not as dramatically, but this is happening with nearly all genres of electronic and scientific equipment. One day, maybe we'll see backyard DIY electron beam drilling for nano-machining.

When it is no longer about who has the most resources, it's about who has the best ideas. Then, it becomes a pure numbers game:

Take 10,000 professional engineers vs. 1 million hobbists with roughly equivalent tools. Which group will make progress faster? Now, consider that you have to pay the 10,000 engineers $100K/year to motivate them work, and the 1 million hobbists are working for the love of it. Does that change your answer? Even if it doesn't, you have to concede that there does exist a ratio which will make the output of these two groups equal. It's merely a matter of time.

If you follow me through this argument, which I won't claim to be bullet proof but it explains the trends we are observing quite nicely, then this has an interesting implication on organizations that are currently funding big research groups. When it's simply a matter of who has the best ideas, it's tough to try to employ enough people to get good coverage. You could try to spend a lot of energy on trying to find the "best" people, but that's about as challenging as predicting the stock market. Some inventors simply go "dry" of good ideas and end up not providing a good lifetime return on investment (I fully expect this to happen to me someday. I just hope it happens later rather than sooner.)

So to me, this suggest 3 options for big exploratory organizations:

1. Start tackling more resource intensive problems - things that fundamentally cannot be done today for a few thousand dollars, but at some basic level requires materials, tools, energy, computation, space, manpower that is impossible to obtain at a hobby level. The LHC and space programs are good examples of this. Even if the end goal may be of debatable near term economic value, there is a high probability that unexpected derivative technologies/projects will bring commercial/educational benefits elsewhere.

2. Empower everyone within your organization to do exploratory work. The tools are cheap and "research groups" have no monopoly on good ideas. It's hard to know where lightning will strike, so make sure you encourage it anywhere and hope you haven't missed a spot.

3. Partner with the outside developer community. There is plenty of precedence where using the resources you have to channel the creative power of the masses through the platforms you control can bring a tremendous amount of value if done in an organized manner. It is the rocket fuel that powers companies like Facebook, Twitter, and Groupon to go from non-existence to dominating entities in less than 3 years. The same can absolutely happen with traditional physical electronics and other consumer goods. It simply requires treating your customers as potential partners, rather than assuming they are all potential predators.