Integration of Subsystems on the Software level

This post we will be talking about integrating the all subsystems on software level. As a whole, we are using 3 arduino boards.

                                Board 1 - Robot Arm and Hand

                                Board 2 - IR Sensors on the Robot Hand

                                Board 3 - Glove (Vibrational Motors)

Shown is the overview of how the user control the robot remotely. Setting up the server is discussed in detail here.

Figure 1 - Overview

Calibrating the Arm

In the previous posts we talked about how we applied Inverse Kinematics and kinematic analysis of the mentor arm. As mentioned the arm consists with Potentiometers and DC motors and with the POTs we get a feedback on the angle and movement direction. In this post we will be talking on calibrating the arm. 

Indeed the reading from the POTs are consistent. And we don't have a control circuit to receive accurate readings. We are passing each POT readings through a digital low pass filter so it filters out the high frequency noise. Also there is a threshold value for pot readings (+/- 10) to avoid the overshoot. 

In the mentor arm there are physical limits to avoid gears getting damaged physically. In the code we will limiting the movement of each arm respect to the limits of the leap motion and including these physical limits. Also in the using the IK formulae calculated angles, will go to infinity if the distance of the palm and the leap origin is greater than the total length of Top arm and bot arm which is 31.5cm. So if the function "jointangles"outputs "NaN" (Not a Number - infinity) we ignore them.

Bending angle calculation - FINAL

In the previous post we mentioned the limitations on calculating the bending angle. As shown in the below figure when bending the finger from B to C we get the same angle as from B to A.

Figure 1- Bending angle

As discussed in the new leap SDK post, skeletal tracking enables us to receive data through out till LM can detect our palm. In the previous one we can identify fingers by its ids as long as the fingers are visible. We had to manually identify fingers earlier because when ever the fingers disappear finger id randomise.

Inverse Kinematics on MENTOR Arm - Part 2

As mentioned in the previous post we are using IK only for 2dof (on x and y plane). By that we calculated the joint angles on the Top and Bottom arm for the given x,y,z coordinates. 

Using the conversion from cartesian coordinate system (x,y,z) to cylindrical coordinates (ρ,theta2,r) the joint angle for base arm (theta2) was calculated.

Angle - Base arm

To calculate the joint angles the function "jointAngles" is used. The x,y,z coordinates of the palm that we get from the leap motion data are passed as an array "v" into the function and theta2 - base angle, angle2 - Bottom angle and angle3 - Top angle are returned as the output. 

A Wrist-y exercise

In this post I'll be detailing the problems that we had with the wrist and integrating it with both the hardware and software.

Without a doubt, this has been one of the most time-consuming problems we've had. Let's get into some of the gory details.

In essential terms, the way that we get the arm moving is to read the potentiometer values from each joint, calculate where the arm should be based on the Leap Motion data, and continue towards that value until we get there. The potentiometers, as we found out, were continuous rotation potentiometers - this is opposed to more typical potentiometers which have maximum and minimum values that you can go. Continuous rotation potentiometers are ones where, as the name states, you can continually rotate them. These potentiometers also have a maximum and minimum point, but instead of stopping you from moving past it, it lets you!

So instead of reading from, say, 0 to 1023 and have the potentiometer physically stop it from continuing on, it just continues on and wraps around back to 0. This is problematic, because if we're at a value of 900, and need to go to 1000, there is a chance that the pot will overshoot (covered in a later post), and will go back to 0. The program will try to  correct for this by continuing the motion until it gets back to a value of 1000. However, this may mean that as it continues to move, it tries to move through itself!

This problem only occurred with the wrists. Other potentiometer values were not affected as we limited the motion so that it was in between 0 and 1023.

However, when the wrist moved up or down (roll), or twisted left or right (pitch), it was possible that it would go through 0 to 1023 or vice versa. So firstly, we had a problem of it moving through the maximum/minimum value of the pot.

Furthermore, we initially used these pots to determine our position.We could map the value of the hand position from the Leap Motion to a potentiometer value. We wanted to use this same idea with the wrists, but we hit upon another problem! When you roll or pitch your hand, the potentiometer values will change differently. What this means is that it is possible for two or more different positions to have the SAME potentiometer values!

Through a graveyard of many possibilities, we hit upon a solution. Our solution for this problem was to instead define certain regions and wrist movements that our robot arm could go to, and use the pitch and roll functions that came with the SDK/API to determine how the hand had moved. A gesture of the hand will determine whether the wrist should move as the pitch and roll characteristics (which I detailed in an earlier post) will change.

It is unfortunate that we could not get the robotic Arm to move in real-time and mirror our wrist movements exacly. It is a limitation of the Arm that we have and the set-up that we were given. However, further work may be able to work around the problem that the wrist creates for us, and an even more functional arm may result!

Leap Motion V2 Tracking Beta

With the new beta version of leap motion API, skeletal tracking it enable us to receive additional for each overall hand and data is much improved compared to version 1.

Leap motion - skeletal tracking
https://community.leapmotion.com/t/v2-tracking-beta-new-sdk-and-tracking-release-v2-0-3-17004/1329

Frames and Data

Frame Extraction 

 
On the client side, we have this set of code that sends a frame of data over the socket connection to the server. The actual Leap Motion SDK uses the 'Leap.loop' to control the frames that come in from the Leap Motion, but since we have to send it over a socket connection, we can't use the functions that come with the SDK to process the data on the server-side (this would have many advantages, as the Frame object that is created from the SDK gives us more information than the raw data). Instead, we have to dump the raw JSON data from the websocket the Leap Motion creates itself, and process it ourselves.

But what's actually in the 'frame.dump()' function?

Here's an example of the mass of data that comes from the frame dump. Yes, it's as crazy as it looks, but we can actually extract what we want from this dump of data.

The start of the dump gives a summary of certain data, namely a summary of data from the Fingers, Hands and Gestures. However, there is also a portion of the data that starts as 'Raw JSON', which is what we are more interested in.

Using the above snippet of code, we can extract the raw JSON data from the Leap Motion, which is what we need to control the arm and hand. JavaScript has powerful string control functions to search and extract certain bits of data. As this data is in JSON format, but has been sent as a string, we 'JSONify' it by using 'JSON.parse(frame)'. This allows it to be a JSON object that we can then reference if we require certain parts of the data.

I think it's important to note that since the start of this project, the Leap Motion SDK and API have been updated twice. It is currently at V3, which incorporates the Virtual Reality side of technology, including the Oculus Rift. Our project has been based on V1, and we are in the process of rewriting for V2. The reason this is important is that each update incorporates more data into each dump of data. Furthermore, the API is updated with newer and better functions.

Thus, instead of having a static reference to the data as to where the JSON data is, we use the relative reference from the words 'Raw JSON', which is more adaptable to changes in the API.

Processing Frame Data

Here is an example of how we have adapted the data to be able to process what we want from it. The Frame data is processed and returns pitch, roll and yaw of the hand.
However, the raw JSON data does not return these values, as it is dependent on the functions that are called in the API.

Thus, we simply extracted the functions that we required from the API:

As I am still learning the intricacies of JavaScript and interfacing with the Leap Motion, I was not certain as how to send a custom 'Frame' object over the socket. I will continue searching to see whether this can be done in a better way!

Next up - vibration motor problems and IR sensor data!

Limitations of calculating the bending angle of the finger

In this article we will talk on some of the limitations in the Leap Motion SDK when calculating the bending angle of the finger.

As mentioned in the "controlling a servo using leap" post we calculated the bending angle of the finger by taking the dot product of finger direction and the palm normal.

Figure 1
[http://hand-illustration.com/82-pinky/240-pinky.html]

Programming Arduino with JS

As we mentioned earlier, we are using Javascript as the main programming language through out the project. The Arduino microcontroller can be controlled through the use of frameworks like Rick Waldron's Johnny-five for Node JS, and the "StandardFirmata"software package for Arduino. 

Yeah we will be most likely utilising the Node.JS framework, which includes packages that allow for ease of communication between hardware and software. This enables us to handshake between Arduino Firmware and the Firmata protocol. Structure of the whole system is shown below.

Presentation Day!

Yesterday we had to present what we had done in front of our supervisor, Jonathan Li. Andy Russell was secondary supervisor while Geoff Binns and a dozen other friends were there to support us.

We had four demos we wanted to show.

1. Mentor Arm responding to commands from the keyboard

2. Haptic feedback from proximity IR sensors

3. Robot Hand responding to hand gestures via Leap motion

4. Invisible wall - Haptic feedback and Leap motion

The demos went really well. There were some glitches here and there however the overall response was positive!

If you missed our presentation, here are the slides:

Identifying all the fingers seperately

In order to control each servo we have to identify fingers separately(thumb, index, pinky, ring, middle). The sdk assigns an ID to each finger however this randomise always. So I did some research regarding this and most of them were suggesting to identify each finger by taking their lengths and the angle between palm and the tip position.

However I managed to identify each finger by sorting the x coordinates. So the left most finger is the Thumb, then index and vice versa.

Controlling a servo

I managed to find the angle of the bending finger by taking the dot product of finger direction and palm normal.

In the Javascript code I am using the math library. Also this method enables to find the angle when you move the finger side to side.

Then I used the Johnny-five library to interface with the Arduino. Hope this video helps you. 

Next Job is to identify each finger. 

Viewing Frame Data

This is regarding viewing a frame from the leap motion. I found this web application, it basically basically converts JSON strings to readable format (JSON viewer). http://jsonviewer.net/

Each frame has x,y,z coordinates of Hand(palm) and the Pointables(fingers). 

Short update on the IR sensing equipment

Hey all! It's been a while - mostly due to the fact that my subsystem is waiting on the most important parts - VIBRATIONAL MOTORS!

We've purchased them from a US company, but there were a couple of complications in the purchasing bureaucracy that has led their shipping to be delayed. In the mean time, though, I've been learning JavaScript for use in NodeJS, which is what we'll be using in the LeapMotion interface. The source I have been using is Eloquent Javascript, quite a useful resource, and I'll be moving on to Node soon enough :)

Furthermore, I've been prototyping the infrared ("IR") sensors that Ray had left over from some design units from the last couple of years. These work quite well in terms of voltage - an Arduino can easily deal with them. However, there were problems we encountered in the process.

Firstly, the IR emitter itself is not strong. The receiver was not able to pick up the response from it very well. Instead, we substituted an IR LED. This was able to give a much stronger response that we could read from the Arduino. Next, the ambient light of a room can affect the response, and thus needs to be taken account in the code. Lastly, the colour of the object being put in front of the sensor matters as well - black is not a good colour for IR sensing as it absorbs so much of it!

Thus, we used a bit of a hack to work out if it was close to something or near something. See the short code snipped below:

#define IRreceive A1 
int IRinitial;
void setup() { 
Serial.begin(9600); 
pinMode(IRreceive, INPUT); 
delay(100); 
IRinitial = analogRead(IRreceive); 
} 

void loop() { 
long IR; IR = analogRead(IRreceive);
 Serial.print("NUMBER: ");
 Serial.println(abs(IRinitial - IR)); 
}

The initial read of the IR sensor is used to assess the ambient light of the room. This is then used to compare to response we receive from the sensor. The response to most colours is usually higher. However, for black, it does not sense that it has been put close to the IR receiver until it is pretty much covered - in which case it will give an even lower response. Thus the need for the in abs the code.

Once the motors have come in, I'll be working on making these IR sensors send information to them and hopefully emulate some sort of haptic feedback.

Interfacing Leap Motion with Arduino

Using the WebSocket server in leapmotion SDK we can analyse the tracking data.

Its super easy! Just connect the port to ws://127.0.0.1:6437

I have used 'ws' library from node.js. To install the 'ws' library the following command is used.

npm install ws --save

We communicate with Arduino using the 'johnny-five' library. It's installed using this 

npm install johnny-five --save

Yeah! Thanks to node.js :)

You can find the code on github: arduinoleapcar

Have FUN! :)

Haptic Feedback with the Kinect!

Seems very similar to what we're doing, except with a Kinect, which means it can track a couple more elements of a person. He incorporates flex sensors as well, which might be useful since the Leap Motion cannot sense the joints easily.

He also uses linear actuators, and from forum-trawling, it seems that some people use this site: Precision Micro Drives to find vibrational motors. We will probably be using some slimmer, smaller types of vibrational motors for the haptic glove.