Fakenect - OpenKinect driver simulator, experiment with the Kinect without the hardware
The Kinect is quickly becoming a transformative device for Human/Computer Interaction, Computer Vision, and Robotics. I have been fairly acive in the OpenKinect community and one thing I commonly hear is that "I want to experiment with the demos out there but I don't have a kinect". From that, I decided to make a library that identically replicates the libfreenect library so that anything that dynamically links it can function without requiring a Kinect. Instead of generating synthetic data, I made a "record" program that dumps the RGB, Depth, and accelerometer data into a directory that can be loaded by fakenect.
Examples
I will use my demo program demo_cv_async.py in libfreenect/wrappers/python/ for this example.
./demo_cv_async.py
Here is a frame from the output
The kinect sensor data can be dumped by running 'record' in libfreenect/build/utils
./record
Records the Kinect sensor data to a directory
Result can be used as input to Fakenect
Usage: ./record <out_dir>
./record legos0
...
legos0/a-1290762884.085605-2032009319-30.dump
legos0/d-1290762884.101368-2042233973-614400.pgm
legos0/a-1290762884.151033-2042233973-30.dump
legos0/r-1290762884.174879-2044011044-921600.ppm
legos0/a-1290762884.249213-2044011044-30.dump
legos0/a-1290762884.251880-2044011044-30.dump
legos0/a-1290762884.260512-2044011044-30.dump
...
The dump output is available here. During the example I moved the Kinect to get different viewpoints. I can now load this into the demo shown previously by running this on Linux (the following is one big command, not multiple commands)
LD_PRELOAD="/usr/local/lib/fakenect/libfreenect.so" FAKENECT_PATH="../../build/utils/legos0" ./demo_cv_async.py
or on OS X
DYLD_LIBRARY_PATH="/usr/local/lib/fakenect/" FAKENECT_PATH="../../build/utils/legos0" ./demo_cv_async.py
Here is a frame from the output
Note that the demo was not created with fakenect in mind, I just overrode the library it uses with fakenect and passed fakenect the sensor dump path with an environmental variable.
Another example on glview in build/bin
LD_PRELOAD="/usr/local/lib/fakenect/libfreenect.so" FAKENECT_PATH="../utils/legos0" ./glview
Here is a frame from the output
Again, nothing was recompiled and glview has been around since the beginning (so clearly it works on 'legacy' applications).
Applications
There are a variety of use cases for this.
I work in a research lab and it rarely makes sense to do experiments on a live sensor as it makes reproducing results difficult and it constraints you to real-time performance (first make it work, then make it work fast). This allows you to record Kinect data and easily switch between live and recorded modes.
If you don't have a kinect but want to play with one, this is your opportunity. Get a sensor dump (you can use the one I posted above), build the libfreenect driver, and run demos like I showed previously. This can 1.) show your friends how cool you are, 2.) test your hardware out, 3.) help you understand what the kinect does, 4.) start writing code even before you get a kinect, and 5.) save yourself a few hundred bucks.
This can be used to mock the Kinect out for testing purposes where you can have a few test shots and run automated unit testing, etc.
Where to get it
This is currently pending acceptance in the main repo but you can get it now from
and the pull request
It is in the mainline tree now
https://github.com/OpenKinect/libfreenect
https://github.com/OpenKinect/libfreenect
Implementation
If you've hung around this far you probably want to know more, well here it is.
The program takes one argument (the output directory) and saves the acceleration, depth, and rgb data as individual files with names in the form "TYPE-CURRENTIME-TIMESTAMP" where TYPE is either (a)ccel, (d)epth, or (r)gb, TIMESTAMP corresponds to the timestamp associated with the observation (or in the case of accel, the last timestamp seen), and CURRENTTIME corresponds to a floating point version of the time in seconds. The purpose of storing the current time is so that delays can be recreated exactly as they occurred. For RGB and DEPTH the dump is just the entirety of the data provided in PPM and PGM formats respectively (just a 1 line header above the raw dump). For ACCEL, the dump is the 'freenect_raw_device_state'. Only the front part of the file name is used, with the rest left undefined (extension, extra info, etc).
A file called INDEX.txt is also output with all of the filenames local to that directory to simplify the format (e.g., no need to read the directory structure).
And it will keep running, when you want to stop it, hit Ctrl-C and the signal will be caught, runloop stopped, and everything will be stored cleanly.
We read 1 update from the index per call, so this needs to be called in a loop like usual. If the index line is a Depth/RGB image the provided callback is called. If the index line is accelerometer data, then it is used to update our internal state. If you query for the accelerometer data you get the last sensor reading that we have. The time delays are compensated as best as we can to match those from the original data and current run conditions (e.g., if it takes longer to run this code then we wait less).
Sensor Dumps
