Frames and Data

Frame Extraction 

 

	Leap.loop(controllerOptions, function(frame) {
	socket.emit('frame', frame.dump());
	})

view raw gistfile1.txt hosted with ❤ by GitHub

On the client side, we have this set of code that sends a frame of data over the socket connection to the server. The actual Leap Motion SDK uses the 'Leap.loop' to control the frames that come in from the Leap Motion, but since we have to send it over a socket connection, we can't use the functions that come with the SDK to process the data on the server-side (this would have many advantages, as the Frame object that is created from the SDK gives us more information than the raw data). Instead, we have to dump the raw JSON data from the websocket the Leap Motion creates itself, and process it ourselves.

But what's actually in the 'frame.dump()' function?

Here's an example of the mass of data that comes from the frame dump. Yes, it's as crazy as it looks, but we can actually extract what we want from this dump of data.

The start of the dump gives a summary of certain data, namely a summary of data from the Fingers, Hands and Gestures. However, there is also a portion of the data that starts as 'Raw JSON', which is what we are more interested in.

	if (data.length > 100) {
	var index = data.search('Raw JSON');
	var frame = data.slice(index+14);
	json = JSON.parse(frame);
	}

view raw gistfile1.js hosted with ❤ by GitHub

Using the above snippet of code, we can extract the raw JSON data from the Leap Motion, which is what we need to control the arm and hand. JavaScript has powerful string control functions to search and extract certain bits of data. As this data is in JSON format, but has been sent as a string, we 'JSONify' it by using 'JSON.parse(frame)'. This allows it to be a JSON object that we can then reference if we require certain parts of the data.

I think it's important to note that since the start of this project, the Leap Motion SDK and API have been updated twice. It is currently at V3, which incorporates the Virtual Reality side of technology, including the Oculus Rift. Our project has been based on V1, and we are in the process of rewriting for V2. The reason this is important is that each update incorporates more data into each dump of data. Furthermore, the API is updated with newer and better functions.

Thus, instead of having a static reference to the data as to where the JSON data is, we use the relative reference from the words 'Raw JSON', which is more adaptable to changes in the API.

Processing Frame Data

Here is an example of how we have adapted the data to be able to process what we want from it. The Frame data is processed and returns pitch, roll and yaw of the hand.

	/**
	* The pitch angle in radians.
	*
	* Pitch is the angle between the negative z-axis and the projection of
	* the vector onto the y-z plane. In other words, pitch represents rotation
	* around the x-axis.
	* If the vector points upward, the returned angle is between 0 and pi radians
	* (180 degrees); if it points downward, the angle is between 0 and -pi radians.
	*
	* @method pitch
	* @memberof Leap.Hand.prototype
	* @returns {number} The angle of this vector above or below the horizon (x-z plane).
	*
	*/
	Hand.prototype.pitch = function() {
	return Math.atan2(this.direction[1], -this.direction[2]);
	}
	/**
	* The yaw angle in radians.
	*
	* Yaw is the angle between the negative z-axis and the projection of
	* the vector onto the x-z plane. In other words, yaw represents rotation
	* around the y-axis. If the vector points to the right of the negative z-axis,
	* then the returned angle is between 0 and pi radians (180 degrees);
	* if it points to the left, the angle is between 0 and -pi radians.
	*
	* @method yaw
	* @memberof Leap.Hand.prototype
	* @returns {number} The angle of this vector to the right or left of the y-axis.
	*
	*/
	Hand.prototype.yaw = function() {
	return Math.atan2(this.direction[0], -this.direction[2]);
	}
	/**
	* The roll angle in radians.
	*
	* Roll is the angle between the y-axis and the projection of
	* the vector onto the x-y plane. In other words, roll represents rotation
	* around the z-axis. If the vector points to the left of the y-axis,
	* then the returned angle is between 0 and pi radians (180 degrees);
	* if it points to the right, the angle is between 0 and -pi radians.
	*
	* @method roll
	* @memberof Leap.Hand.prototype
	* @returns {number} The angle of this vector to the right or left of the y-axis.
	*
	*/
	Hand.prototype.roll = function() {
	return Math.atan2(this.palmNormal[0], -this.palmNormal[1]);
	}

view raw gistfile1.js hosted with ❤ by GitHub

However, the raw JSON data does not return these values, as it is dependent on the functions that are called in the API.

Thus, we simply extracted the functions that we required from the API:

	var pitch = Math.atan2(hand.direction[1], -hand.direction[2]);
	var roll = Math.atan2(hand.palmNormal[0], -hand.palmNormal[1]);

view raw gistfile1.js hosted with ❤ by GitHub

As I am still learning the intricacies of JavaScript and interfacing with the Leap Motion, I was not certain as how to send a custom 'Frame' object over the socket. I will continue searching to see whether this can be done in a better way!

Next up - vibration motor problems and IR sensor data!