Motion Recognition Work

Now that I have my base c# framework ported over to unity, I have began work building more advanced motion recognition to power my beta review object editor/navigator application.

There are multiple motion recognition engines that "look for" specific motions. They all take in the same atomic action data from my motion segmenter. The motion recognizers are also turned on/off based on the current state/context and from other motion recognizers. For example, I implemented a motion recognizer that checks to see if you have moved your hands into the "up" position (up from you side and pointed towards the general direction of the screen). Once this state is reached, then my rotation recognizer is activated and will look to is if you rotate you two hands in synchronization like you would if you were rotating a real life object.

Here is a snapshot of my work in progress. I am rotating a 3d cube in 3 dimensions (x,y,z axis) based from the motion controls. If you look on the bottom, your will see the 2 unity icons and a black bar. The unity icon (its a placeholder image for now) shows up if you right/left hand is the "up" position. The black bar represents the distance between your two hands which will affect the sensitivity of your rotation (more about that on the next post). The basic idea is to give on-screen feedback at every step of the interaction.

Look for a video demo once I get the rotation kinks worked out.

Post-Alpha Review Next Steps

The alpha-review was great in terms in getting feedback on my current progress. There are two main issues/areas of focus that have been repeatedly emphasized by my reviewers:

1. Gesture Recognition is not trivial and can be challenge.

2. You should define an application/use-case early on and adjust your motion recognition accordingly.

These are two really good points and I definitely agree with them.

I'm currently bringing my existing C# test framework into Unity, which will my final production code. This has taken sometime, but I have finally resolved many issues regarding wrappers, dlls and Unity's inability to use the Microsoft .net 4.0 framework (the framework that the Kinect SDK utilizes).
I found this trick online http://www.codeproject.com/KB/dotnet/DllExport.aspx and basically had to play around with lots of compile settings both in C# and C++ for this to work.

Now getting back to addressing my main two challenges, I have decided that my initial use case for the beta-review will be a spatial interaction environment for 3D objects. Thus in simple terms, the user should be able to move and rotate an object in 3D space while also being able to change his/her view point. Think of a the cis 277 object editor but in unity and with voice and gesture controls.

Probably one of the most important aspects to gesture recognition doesn't actually have to pertain directly to the recognition of motion, it has to do with user feedback. How does a user know if his gesture has been registered? How does the user even know if the interface is listening for input in the first place?

This user feedback has been implemented religiously in most traditional gui interfaces (good ones that is). Hover your mouse over a button--That button now will change color/opacity/shape letting the user know that the button is listening. Move your mouse out and the button will return its previous state, letting the user know that the button is no longer listening to your mouse click. Finally if you click the button, the button will change its visual state once more to signify that your action has been successfully recorded.

In order for successfully gesture recognition, this same interaction flow must be replicated in our NUI. If our user does not know that his action is being recorded, then they will madly wave back and forth which will leader to further misinterpretation by our recognition engine. Furthermore, the user must be told if his gesture/motion has been correctly recognized if it has been ignored because our engine is enable to parse it.

Since it does not make sense to have button feedback (the whole point of a NUI is to remove the mouse-pointer paradigm) and pop up dialog boxes are intrusive, I've decided to utilize a border highlight that displays the feedback. The response is coded in the color of the border. The color will then fade away after the feedback is shown.

Initial "recording feedback". The user has stepped into the interaction space.

User action has been, recorded and saved. However, it might be of a longer gesture sequence so the entire gesture is not complete yet and we are waiting for more motions.

User action or sequence of actions has recognized and we have updated the state of the interface to correspond with this.

Current action or current sequence of actions cannot be recognized as a specific command. The sequence/action has been deleted from queue. Start the action again from the beginning.

My rationale for this:
1. A color border is non-obtrusive but yet has enough global visual scale to reach the attention of the suer.
2. The differentiation between recorded gestures and actually completed gestures allows the use of gestures that formed from the build up of many atomic gestures. Thus as we build up to these complex gestures, it is still good to know that our sequence of actions is still being recorded.
3. The initial "on-phase" provides feedback to the user that he is in the interaction space and all his motions are currently being watched.

This mode of feedback is inspired from Alan Cooper's concept of modeless feedback from his book on interaction design, About Face.

Motion Segmentation

The past few days, I have been playing around with a good motion segmentation scheme so that I can break down user motions into discrete, atomic "actions". The plan for these actions is that they will form the "alphabet" of my motion gesture language. Combination of actions or individual actions in the correct context and mode (remember from my previous posts?) will lead to interactions with our onscreen interface.


I used the concept of zero crossing to segment motion, thus motion is broken into actions by a combination of velocity and acceleration. The basic principle is that a abrupt change in velocity (high acceleration) signals the end of a previous action and the start of a new action.

Thus I used a finite state machine to represent the breakdown of these motions.

Then after each motion has been completed, I save the many attributes of that specific motion and put in my queue of previous motions that will then be interpreted.

Here is what I store (so far).

Finally, here is an trial of my motion segmentation. On the bottom left I keep track of the various states that the motions go through, and everytime a motion is completed it is pushed onto the queue and the total number of motions increase. As you see, its fairly accurate but fails to pick up very rapid minute motions, but in the grander scheme of things I think it is good enough for our purposes.

Smoothing vs No Smoothing

Here is a video comparison of smoothing vs no smoothing in my framework. I utilized the Kinect's built Holt smoothing for the joint positions. I also applied my own holt smoothing to my calculated velocity and acceleration values.

http://www.youtube.com/watch?v=TUgUnrWrjRw

Working with the low-level motion data.

Please excuse me for my late posting. I was only at Penn the last week for less than 48 hrs (Fall Break + a trek to EA's Redwood Shores HQ for my Wharton EA field application project). Expect a few post in the next few days to make up for it.

As previously mentioned, the Xbox Kinect SDK provides joint location positions (in the form of Vector3's).
Until now, I have been simply mapping relative velocity/position of joints to onscreen actions such as the zoom and pan of 2D objects. However, there are a few problems with this naive approach. First, the raw position points from the Kinect sensor are not perfectly stable. For example, if you hold your hand out straight without any movement, the onscreen object mapped to your hand's position will jitter based on sensor inaccuracies.
Additionally, we are not able to separate gestures: if an individual accelerates his hand to zoom and then stops, we should the onscreen zoom to reflect this behavior; however, this does not occur if we simply map zoom to hand position since the mapping does not stop, even after our action.

Thus for the next few days, I will add on filtering and motion segmentation features on top of my existing Kinect framework.

For filtering, my main objective is to reduce data jitter and smooth the raw position data such that motion mappings produce smooth onscreen actions. There are two aspects in filtering.
(1)The first is to reduce drift which is low frequency noise. Drift is often caused when our subject ever so slightly changes his overall position. The simplest way to reduce drift is to make the subject root position (which can be the shoulder position in the case of hand/arm motion gestures) the origin of the coordinate frame.
(2)The second component of our jitter comes from high frequency noise. This is a result of both minute motions of the subject and the slight inaccuracies of the Kinect camera sensors. The best way to fix this problem will be to pass the data through a smoothing filter. Luckily the Kinect SDK comes its built in smoothing function that is based off the Holt double exponential smoothing method.

I will also apply the Holt smoothing method to my velocity and acceleration calculation in the framework.
Finding the right the constants for optimal smoothing and finding a balance between accuracy/responsiveness and minimal jitter will be an ongoing process throughout my project.

My motion segmentation implementation will mainly revolve around the zero-crossings of specific joint velocities. The basic idea is that every abrupt change in our velocity will represent a new motion. More to come on this as I begin the implementation....

Elements of Interaction

While I was playing around with moving a shape across the screen using my Kinect Framework, I realized that a gesture does not make an interaction alone. When we swipe our hands left to move an object left, our interaction comprises of more than just a mapping between the hand and object position.

    At the root of any interaction, we have the gesture. The gesture is a explicit action with a purpose of achieving a specific goal or result. In my interface, the gesture is usually revolves around the position, velocity or acceleration of specific skeletal joints.

However, this gesture information is useless if we do not know what our gesture maps to. This is where the mode of interaction comes into play. The mode of interaction is a explicit mapping between our gestures and state of the interface. So for our simple example, panning is the mode of interaction.

Finally the last element of our interaction is the context. The context comprises our relationship with our input environment. Are we sitting or standing? Are we actually facing the screen? What actions/gestures did we perform leading up to this current interaction? Context envelops our spatial relationship in the environment and the history of the sequence of our interactions.

In conclusion:  Gesture + Mode + Context = Interaction.

I am currently on working on implementing this interaction scheme in my Kinect framework.

Setting up the for Kinect Development

This past week, I have been focusing on becoming familiar with development using the Kinect SDK. I decided to develop the recognition engine in C# rather than C++. Two main reasons:

1.) C# makes it easier to perform threading, and this will come in useful when having to integrate both audio and motion input.

2.) C# has some basic UI elements built in, so this allows me to test and prototype faster as I can easier code simple demos to test the recognition engine before I render the final interface in unity/

After looking through some demo code and documentation I found out that the Kinect SDK allows for two methods of motion tracking. The first is a polling-based approach, where the Kinect sensors return information at stated intervals. The other option is a event-based approach, where Kinect sensors return data if there is actually an event (motion, change in depth, etc....). The event-based approach seems more efficient for my purposes, and I decided to build on top of this approach.

The first step in linking up any code to the Kinect is to turn on the sensors that you need.

Here we activate the depth sensors, skeletal tracking and the raw color camera (for testing purposes so we can see how our real world actions correlate to on screen actions)

Next, we hook assign event handlers to the sensor events.

Finally, we create an entire new thread for our audio input and speech recognition.

Most of my code right now is under the nui_SkeletonFrameReady. This is called every time the Kinect senses a chance in any of the skeletal joint position, and my interface will react accordingly.

Joint Data 

The Kinect Skeleton only returns the positions of the joints in as a Vector3. Thus I created a wrapper class over these joint positions that calculates both velocity and acceleration data. The velocity and acceleration are updated every time the SkeletonFrame event handler is called. I am looking at how to better smooth and interpolate this data so that gesture motion will not lead to "spikey".

Voice Data

Through the Kinect, we can leverage Microsoft's Speech API that has voice recognition features. We open the Kinect's microphone input as an audio source stream, and this stream is passed to the Microsoft's speech recognizer.

Before anything else, it is necessary to generate a grammar, a list of words or phrases that the speech recognizer should be looking for.

Much like the SkeletonTracking, the speech recognition system uses an eventhandler system. The 3 events that are thrown are SpeechRecognized, SpeechHypothesized and SpeechRecognitionRejected events. I still need to do more reading about the exact details of these three events, but I am only responding for SpeechRecognized events for now.

Testing Environment

I made a simple testing interface as I continue to implement gesture recognition features and modes. 

All my work right now involves basic manipulation (zoom, pan) of a circle in two-dimensional state.

I also output some key information such as the current mode that my interface is in and the sensitivity modifier(discussed in the previous post). 

I will go more in-depth about this test environment in my next post and explain some of the new gesture recognition models that I have come up with. Videos too! Look for it tomorrow once I get camtasia (need Joe's admin password) installed on my dev computer.

Test interface: