Today we will be discussing how you can write/paint in the air using your finger. We will be using Python language and Opencv library for this purpose. Check out the video below.
Ok before starting we need to make a few assumptions for the proper working of this application.
You will notice a lot of small white dots are removed from the image. But this is not enough. Right? So now we apply dilation morphology. Look at the result below.
Looks great. Isn't it? Opening morphology is erosion followed by dilation. Now, what is erosion and what is dilation? When you apply erosion morphology, it erodes away the boundaries of the foreground. This is helpful in removing small patches of noise in the image. But it also decreases the foreground area. Dilation is the opposite of erosion. It increases the foreground object size. Thus when you use opening morphology it first erodes the image and then dilates it. But even after applying dilation, the image is still noisy.
Don't worry about the noise for now. The main objective of applying dilation was to remove those small black spots on the hand which you can see in the resultant image of opening morphology. We did this because we want the hand to look as one distinct structure, without any distortions.
Now, let us use find contours method on the above result.
If you see the image above, you will notice that there are only two objects in the image frame with a very large area. One is your face and the other is your hand. If you try not to be in front of the camera, the problem becomes much simpler. But we will try to do it the hard way. This method is not full proof but good enough for you to get started.
Let me tell you the trick. when you try to write using your finger in front of the camera, your hand will be in between the camera and your face. And since your hand is more near to the camera than your face(which is at the farthest end from the camera), the area of your hand will be more than the area of your face. Now you get the point?
So, let's find the contour with the maximum area.
Using the above code we select the contour with the maximum area(Which is your hand).
Now, let us find the centroid of this contour using the following code.
The first line returns the features of the contours using which you can calculate the center of mass of the object, area of the object, etc. We use it to find the centroid of the object (cx, cy) and draw a small circle around it.
The first argument in cv2.circle method is the image variable on which you want to draw the circle. The second argument is a tuple of x and y coordinates. The third argument is the radius of the circle. The fourth argument is the color of the circle and the fifth argument is the width of the circle boundary.
It's time for trick number two. You want to write in the air using the tip of your finger. Isn't it? If not, this tutorial is not for you. So when you hold your hand in front of the camera for writing, your fingertip is the farthest point from your hand. right? Do you understand where I am going? We need to find the coordinates of this topmost point of your hand.
The above code returns the coordinates topmost point of your hand i.e. fingertip. Now what? Draw circles on the coordinates?
Nope, one last step remaining.
The above condition comes from a little observation. How you hold your hand while writing? The first condition is the difference between the y coordinate of centroid and the y coordinate of the topmost point. Similarly, the second condition is the difference between the x coordinate of the centroid and the x coordinate of the topmost point. This is to ensure that the drawing is done by the fingertip only and not some arbitrary point in the image.
And finally, you are ready to draw.
Display the image using imshow method. The final image is shown below. Don't forget to release the memory at the end of the while loop.
To release the memory use the following code.
And now you are finally done with the project.
Check out code for selecting HSL values using trackbar(The code is for HSV image you will have to change it to HSL ):- https://github.com/vibhor69meshram/Set-HSV-values-using-Trackbar
Ok before starting we need to make a few assumptions for the proper working of this application.
- This background is always static i.e. there is no addition or subtraction of objects in the background scene.
- The background-color is always constant. It does not change with time.
- The object that will be used for writing/painting is of a different color than the background to give us sufficient contrast between foreground and background.
We are ready to begin now. Let us start by installing necessary python libraries for our project using pip install. We will be needing Numpy and Opencv libraries. Now create a python project and create a new script. Import the required libraries into python script as shown below.
import cv2 import numpy as np
To capture a video, we need to create a VideoCapture object. Its argument can be either the device index or the name of a video file. The device index is just the number to specify which camera.
cap = cv2.VideoCapture(0)
Now, we will write a while and capture image frames continuously. The following line of code is used to capture the frames.
ret, frame = cap.read() frame = cv2.flip(frame, +1)
The second line is used to mirror the image frame so that it becomes easy for writing. The image we read is in RGB format. We need to convert this image into HSL(Hue Saturation Lightness) format. Since here skin color is involved HSL image works better for me. If you do not know the difference between HSV and HSL image click here.
frame2 = cv2.cvtColor(frame, cv2.COLOR_BGR2HLS)lb = np.array([0, 0, 81])ub = np.array([185, 255, 255])
What are these lb and ub? These are the lower and upper bound thresholds. The thing is we want the background to be completely eliminated and only our object should be present in the image frame. For this purpose, we decide these threshold values. lb and ub are numpy arrays of 3 elements. Each element represents the value of Hue, Saturation, and Lightness. The HSL values of image which lie between lb and ub will be only kept in the image frame and rest will be discarded. The above values work fine for me. Your values might be different. The best way to define these values is by using trackbars.
Now we create a mask for our image which will only keep values lying between lb and ub and discard the rest of the values.
mask = cv2.inRange(frame2, lb, ub)
The masked image looks something like this.
We are only interested in hand and the rest of the background is useless for us. So somehow we have to remove all the noise from this mask. To remove that noise we can use morphology. Opencv has a method called opening morphology which removes the small patches in the image.
opening = cv2.morphologyEx(mask, cv2.MORPH_OPEN, Kernal)
The first argument here is the image on which you want to perform morphology. The second is the type of morphology you want to perform and the third is the kernal. Kernal is nothing but a matrix of any size defined by you with all the elements as '1'. After performing morphology the image looks like this.
Looks great. Isn't it? Opening morphology is erosion followed by dilation. Now, what is erosion and what is dilation? When you apply erosion morphology, it erodes away the boundaries of the foreground. This is helpful in removing small patches of noise in the image. But it also decreases the foreground area. Dilation is the opposite of erosion. It increases the foreground object size. Thus when you use opening morphology it first erodes the image and then dilates it. But even after applying dilation, the image is still noisy.
Don't worry about the noise for now. The main objective of applying dilation was to remove those small black spots on the hand which you can see in the resultant image of opening morphology. We did this because we want the hand to look as one distinct structure, without any distortions.
Now, let us use find contours method on the above result.
contours, hierarchy = cv2.findContours(opening, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
The first argument in the find contours method is the source image. The second argument is the mode for finding contours and the third argument is the method to approximate the contours. Find contours returns a list of all the contours found in the image. This array is stored in the contours variable.
Let me tell you the trick. when you try to write using your finger in front of the camera, your hand will be in between the camera and your face. And since your hand is more near to the camera than your face(which is at the farthest end from the camera), the area of your hand will be more than the area of your face. Now you get the point?
So, let's find the contour with the maximum area.
for contour in contours: if cv2.contourArea(contour) > 1500: cnt = contour
Using the above code we select the contour with the maximum area(Which is your hand).
Now, let us find the centroid of this contour using the following code.
M = cv2.moments(cnt) cx = int(M['m10'] / M['m00']) cy = int(M['m01'] / M['m00']) cv2.circle(frame, (cx, cy), 5, [50, 120, 255], -1)
The first line returns the features of the contours using which you can calculate the center of mass of the object, area of the object, etc. We use it to find the centroid of the object (cx, cy) and draw a small circle around it.
The first argument in cv2.circle method is the image variable on which you want to draw the circle. The second argument is a tuple of x and y coordinates. The third argument is the radius of the circle. The fourth argument is the color of the circle and the fifth argument is the width of the circle boundary.
It's time for trick number two. You want to write in the air using the tip of your finger. Isn't it? If not, this tutorial is not for you. So when you hold your hand in front of the camera for writing, your fingertip is the farthest point from your hand. right? Do you understand where I am going? We need to find the coordinates of this topmost point of your hand.
extTop = tuple(cnt[cnt[:, :, 1].argmin()][0])
The above code returns the coordinates topmost point of your hand i.e. fingertip. Now what? Draw circles on the coordinates?
Nope, one last step remaining.
if abs(cy - extTop[1]) > 200 and abs(cx - extTop[0]) < 150: x1.append(extTop[0]) y1.append(extTop[1])
The above condition comes from a little observation. How you hold your hand while writing? The first condition is the difference between the y coordinate of centroid and the y coordinate of the topmost point. Similarly, the second condition is the difference between the x coordinate of the centroid and the x coordinate of the topmost point. This is to ensure that the drawing is done by the fingertip only and not some arbitrary point in the image.
And finally, you are ready to draw.
for i in range(len(x1)): cv2.circle(frame, (x1[i], y1[i]), 4, (255, 155, 100), 5)
Display the image using imshow method. The final image is shown below. Don't forget to release the memory at the end of the while loop.
cv2.imshow('Resuting Image', frame)
To release the memory use the following code.
cap.release() cv2.destroyAllWindows()
And now you are finally done with the project.
Check out the full code here:- https://github.com/vibhor69meshram/Writing-in-air-using-finger
Check out code for selecting HSL values using trackbar(The code is for HSV image you will have to change it to HSL ):- https://github.com/vibhor69meshram/Set-HSV-values-using-Trackbar
brother what is contour intial value..??
ReplyDelete