Frames and Data

Frame Extraction 

 
On the client side, we have this set of code that sends a frame of data over the socket connection to the server. The actual Leap Motion SDK uses the 'Leap.loop' to control the frames that come in from the Leap Motion, but since we have to send it over a socket connection, we can't use the functions that come with the SDK to process the data on the server-side (this would have many advantages, as the Frame object that is created from the SDK gives us more information than the raw data). Instead, we have to dump the raw JSON data from the websocket the Leap Motion creates itself, and process it ourselves.

But what's actually in the 'frame.dump()' function?

Here's an example of the mass of data that comes from the frame dump. Yes, it's as crazy as it looks, but we can actually extract what we want from this dump of data.

The start of the dump gives a summary of certain data, namely a summary of data from the Fingers, Hands and Gestures. However, there is also a portion of the data that starts as 'Raw JSON', which is what we are more interested in.

Using the above snippet of code, we can extract the raw JSON data from the Leap Motion, which is what we need to control the arm and hand. JavaScript has powerful string control functions to search and extract certain bits of data. As this data is in JSON format, but has been sent as a string, we 'JSONify' it by using 'JSON.parse(frame)'. This allows it to be a JSON object that we can then reference if we require certain parts of the data.

I think it's important to note that since the start of this project, the Leap Motion SDK and API have been updated twice. It is currently at V3, which incorporates the Virtual Reality side of technology, including the Oculus Rift. Our project has been based on V1, and we are in the process of rewriting for V2. The reason this is important is that each update incorporates more data into each dump of data. Furthermore, the API is updated with newer and better functions.

Thus, instead of having a static reference to the data as to where the JSON data is, we use the relative reference from the words 'Raw JSON', which is more adaptable to changes in the API.

Processing Frame Data

Here is an example of how we have adapted the data to be able to process what we want from it. The Frame data is processed and returns pitch, roll and yaw of the hand.
However, the raw JSON data does not return these values, as it is dependent on the functions that are called in the API.

Thus, we simply extracted the functions that we required from the API:

As I am still learning the intricacies of JavaScript and interfacing with the Leap Motion, I was not certain as how to send a custom 'Frame' object over the socket. I will continue searching to see whether this can be done in a better way!

Next up - vibration motor problems and IR sensor data!