Working with the PixArt camera directly

This has been a pretty whirlwind past few months. Lots of things have happened, almost none of which procrastineering related which is why I haven't posted anything here. But, one of the things that I have poked at in the past few weeks was creating a PixArt to USB-HID device which allows the camera from the Wiimote to appear as a relatively easy to access USB device. This addresses several problems with using the Wiimote such as running off batteries for extended periods and flakey platform specific Bluetooth drivers. It's also possible to read from the Pixart cam at over 100Hz if you read directly via I2C as well as track visible dots once you remove the IR filter. Of course, none of this was discovered by me. All credit belongs to the numerous individuals who have contributed thier knowledge to the various Wiimote hacking websites. Normally, this project wouldn't be worth a post, but all the information on how to do this is pretty scattered and difficult to follow. So, I figured I would contribute by trying to making this all a bit clearer.

This project is fairly advanced. You must be comfortable with working with microcontrollers. Several simpler devices such as the Arduino or the Basic Stamp may work, but I used the 18F4550 PIC Microcontroller which provides built-in full-speed USB capabilites. But first, let talk about the PixArt camera:

Here's the pinout thanks to kako and a PCB picture. The Reset pin is active low, so use a pullup resistor to Vcc. The Wiimote runs the camera with a 25Mhz clock, but it also works with a 20Mhz clock so you might get away with fudging this a bit. The I2C communication is fast 400Khz and the slave device address is 0xB0. Most microcontroller development platforms should include I2C communication capabilities. If yours doesn't, get a better dev kit =o). Desoldering the camera can be hard with so many pins. But, careful use of a hot air gun will do the trick. The first part is to initialize the camera over I2C. Here's the pseudo code for initializing to maximum sensitivity (actual CCS C code in comments):

1. write(hex): B0 30 01
2. wait 100ms
3. write(hex): B0 00 00 00 00 00 00 00 90 //sensitivity part 1
4. wait 100ms
5. write (hex): B0 07 00 41 //sensitivity part 2
6. wait 100ms
7. write(hex): B0 1A 40 00 //sensitivity part 3
8. wait 100ms
9. write(hex): B0 33 03 //sets the mode
10. wait 100ms
11. write(hex): B0 30 08
12. wait 100ms

It's still somewhat mysterious to me what all these mean, but in this mess is the sensitivity and mode settings described at Wiibrew. The above code uses the sensitivity setting suggested by inio "00 00 00 00 00 00 90 00 41, 40 00" experssed in the 2nd, 3rd, and 4th message. The wait times are conservatively long. After you initialize, you can now read samples from it:

1. write(hex): B0 37 //prepare for reading
2. wait 25us
3. write(hex): B1 //read request
4. read 8 bytes
5. wait 380us
6. write(hex): B1 //read request
7. read 4 bytes

This yeilds one sample from the camera containing 12 bytes, 3 for each of the 4 potential points. The format of the data will be the Extended Mode (X,Y, Y 2-msb, X 2-msb, Size 4-bits). The wait timings approximate what the Wiimote does. I've called this routine 1000 times per second without ill effect. Though, I doubt this is actually scanning the sensor and instead is just reporting the contents of an interal buffer. But, people claim 200Hz updates are possible. So, you can use that as a suggestion.

Hooking this up to your microcontroller is pretty straight forward. Give the camera 3.3v power using a voltage regulator, ground, a 20-25Mhz clock, and connect the SDA and SCL lines (don't forget your pull up resistors), and pull up the reset pin.

The CCS C Compiler for the PIC18F4550 includes USB-HID sample code. It's simply a matter of stuffing the data you got from the PixArt camera into the input report buffers for the USB. With this, you could actually create a USB mouse profile and make it control the cursor without any software or drivers at all. If set it up as a full speed device, it's possible to get 1ms reports providing extremely low latency updates. CCS provides relatively affordable PIC programmers as well. Explaining how to set all this up is not within the scope of this post, but it should be plenty to get you started. If you want to make a PCB, you can try ExpressPCB which can get you boards in-hand for as low as $60.

Update 9/6/08: Just a note about the clock. Since my PIC was using a 20Mhz resonator, I just piggy backed the Pixart clock pin off the OSC2/CLKO pin of the PIC which seemed to work fine. Also, Kako has more details (in Japanese) on doing this with an Arduino

More Wiimote Projects - A Brain Dump

It’s been a while since I’ve posted anything. That’s largely because I’ve been traveling a lot, giving talks, and most recently relocating to a new city. It became clear to me a while ago that I wasn’t going to get around to making more videos anytime soon. So, I figured I would make a post about the projects that I would probably make videos of if I had more free time. The content of this post has been in the talks that I’ve been giving, but I’m just sitting down to write it out now for my trusty blog readers.

1. Throwable Displays using the Wii remote

This I actually built and demoed in my lab at CMU. But, it only existed for about two days before I had to break it down to move and I didn’t get a chance to document it. Several months ago, a patent filed by Philips made some of the tech new sites about throwable displays in games. But it was a concept patent pretty far from a working demo. However, it turns out it’s pretty easy to implement using a projector, a wiimote, an IR emitter, and some of our trusty retro-reflective tape. It essentially combines the techniques from the finger tracking and the wiimote whiteboard projects. You put a little bit of reflective tape on each corner of a square piece of foam core, turn on the IR emitter so the Wiimote can see the four corners, align the camera tracking data with a projector using the 4-point calibration, and then the projector can display images perfectly aligned to the edges of a moving piece of foam core. The process of using a projector to augment the appearance of objects is called “Spatially Augmented Reality”.

Research colleagues of mine made a really fun demo where they tracked an air hockey puck from above and projected down on the air hockey table to display all sorts of visual effects that responded to the location/motion of the puck. They were demonstrating a fancy new type of high-speed tracking system. But, the Wiimote works quite well at 100Hz. I wish I had documented the throwable display on video, because it worked quite well. You really could pick it up and throw it around and the video image stays fairly locked onto the surface. There's a small latency primarily due to the 60Hz refresh of the projector. I even made a rough demo of the air hockey table, but it was VERY rough - just drew a line tail behind the puck. Again, a little patch of reflective tape on the puck and IR ring illuminated Wiimote above. However, the throwable display concept is actually a simpler implementation of a project I did earlier on “Foldable Displays” (tracked using a Wii remote) which I did make a video of, but not in tutorial format like my other Wii videos:

2. 3D tracking using two (or more) Wii remotes

Since the tracking in the Wiimote is done with a camera, if you have two cameras you can do a simple stereo vision triangulation to do full 3D motion capture for about $100. This was actually already done by some people at the University of Cambridge:

This is text book computer vision algorithm, but I haven’t gotten around to making a C# implementation. Obviously, you can use more than 2 wii remotes to increase tracking stability as well as increase occlusion tolerance. This would be a VERY useful and popular utility if anyone out there wants to make a nice software tool to transform multiple wiimotes into a cheap mocap system.

3. Universal Pointer using the Wii remote

The nice thing about the camera is that it can detect multiple points in different configurations. The four dots could be used to create a set of barcode-like or glyph-like identifiers above each screen in a multi-display environment. This would not only provide pointing functionality on each screen, but also provide screen ID which means you could interact with any cooperating computer simply by pointing at its screen. No fumbling for the mouse and keyboard, just walk around the room, or office building, or campus, and point at a screen. If all the computers were networked, you could carry files with your Wiimote virtually (using the controller ID) letting you copy/paste or otherwise manipulate documents across arbitrary screens regardless of what computer is driving the display or what input device is attached to the computer. You just carry your universal pointer that works on any screen, anywhere automatically. This makes a big infrastructure assumption, but it really alters the way one could interact with computational environments. The computers disappear and it becomes just a bunch of screens and your universal pointer.

Similarly, arbitrary objects could have unique IR identifiers. For example, if each lamp in your house had a uniquely shaped Wii sensor bar on it (and they were computer controlled lamps, of course), you could turn on a specific lamp simply by pointing at it and pressing a button or dim it by rotating the wiimote. If was an RGB led lamp, you could specify brightness, hue, and saturation with a quick gesture..

4. Laser Tag using Wii remotes

If you put IR leds on each of the Wii remotes, they can see each other. So, you can have a laser-tag like interaction just using Wii remotes – no display, except perhaps if you wanted a big score board. You’d have to validate which Wii remote you were shooting at, which you would do using some kind of IR LED blink sequence for confirmation. Just wire up the IR leds to the LEDs built into the Wii remote, so you can computer control their illumination.

5. IR tracking with ID using the Wii remote

This is more technical (and related to the above idea), but it addresses an important issue that I have yet to see done in either commercial or research systems. The problem with IR blob tracking using cameras is that you can’t which blob is which. You could blink the LEDs to broadcast their ID. But, this 1) would be slow because the ID data rate is limited by the frame rate of the camera 2) really hurts your tracking rate/reliability because you don’t know where the dot is when the LED is off. Now, the Wii remote’s camera chip gives 100Hz update, which might be tolerable for a small number of IDs. But, this approach doesn’t really work well when you want fast tracking with lots of unique IDs. One solution is to attach a high speed IR receiver to the side of the Wii remote for data transmission and simply use the camera for location tracking. IR receivers used in your TV probably support data rates of around 4000 bps - much higher than the 50 bps sampling limit you could squeeze out of the Wii remote. So, as the LEDs furiously blink their IDs at 4Kbps, they look like they are constantly on to the camera. This yields good tracking as well as many IDs. Now when you have multiple LEDs transmitting simultaneously, you’ll get packet collisions. So, some type of collision avoidance scheme would be needed of which there are many to choose from. It will also be necessary to re-associate the data packet with a visible dot. So, not all the LEDs can be visible all the time. But, you only have to sacrifice a small number of camera frames to support a large number of IDs. You can also probably boost performance if you are willing to accept short term probabilistic ID association.

WiimoteWhiteboard v0.2 - slightely updated/fixed

Hi, all. I got a few moments yesterday to make a couple of small improvements to the Wiimote Whiteboard software. Most notably, I improved the mouse emulation code. There were problems where it wouldn't work with some programs like PowerPoint, Alias Sketchbook, etc. So, those work fine now.

I also added a "Tracking Utilization" feedback to the GUI which tells you how much of the camera you are utilizing for tracking. This gets updated after calibration. This provides a way to evaluate how good your Wiimote placement is which directly impacts tracking quality. Getting this number to 100% is virtually impossible in any usable configuration, but dropping below 50% is a sign of not-so-great Wiimote placement.

I've also updated the code to use Brian Peeks's WiimoteLib v1.2. I think this may help a little with greater Bluetooth/Vista compatibility. On a technical note, the WiimoteLib in this download is slightly modified to increase the Wiimote camera sensitivity.

You can use this link to download the new version: WiimoteWhiteboard v0.2

Inspiring Students

One of the great, unexpected, and perhaps most influential aspects of creating these videos has been how many people they have inspired and sparked an innovative spirit in. I've gotten hundreds of emails from young students that express this enthusiasm. But, perhaps one of the best testimonials is this news article about kids in the Clara Byrd Baker Elementary School's Lego Club in Williamsburg, VA. The students there, lead by Kofi Merritt, are getting excited about innovating in technology by creating their own electronic white boards.

Tracking multiple laser pointers @ 200Hz using the Wii remote

A couple of videos (vid1 and vid2) from sha433 demonstrate that if you are willing to crack open your Wii remote (you'll need a tri-wing screw driver) and take out the IR filter, the camera will track visible points of light, including laser pointers. That's pretty neat. Additionally, it has also been recently discovered by a few people that if you read the data directly off the camera chip using I2C, you can get 200 Hz tracking (vs 100Hz that you get over Bluetooth). It looks like there might be a latency hit, though, depending on how you get the data into your computer.

So, if you are willing to get a little down and dirty with the hardware, you can pull out quite a bit more capability! Good job Sha!

YouTube Awards: Nominated for Best Instructional Video of 2007

Hey all, (shameless plug) it looks like my video on head-tracking was nominated for the Youtube Award - Best Instructional Video of 2007! Very neat. =o) Please, go to the site and cast your vote! Thanks!

EA's Boom Blox to include Wiimote Headtracking

Louis Castle announced yesterday at GDC2008 that EA's upcoming title "Boom Blox" will ship with an easter egg that allows head-tracking using the Wii remote! Very cool. Here's the Digg article someone created for it - which you should all digg to help drum up some attention and demand for this feature. Reward the developers who decided this was worth including and send a signal to EA and the greater game development community that this is a desired step forward in the evolution of game play technology. The expected release will be in May.

I'm proud. If this pans out, it'll be only 5 months between the initial research prototype to integration into a major product release. Sweet! Happy to see my stuff being used. Humorously, there were 3 other demos on the GDC expo floor showing variations of my head tracking demo. =o)

Just in case you are wondering: No, I don't get any royalties or benefits for the use of this technique in games. Personally, I'm much happier impacting the state of technology on such a large scale in such a short period of time rather than struggling to transform it into personal financial gain. In terms of my original intent behind creating the head-tracking demo, it has already been a wild success beyond my highest expectations.

Joystiq article

Digg article