Writing in Air using Python and Opencv | Computer Vision

Today we will be discussing how you can write/paint in the air using a colored object. We will be using Python language and Opencv library for this purpose. Check out the video below. I think you now get the idea of what we will be doing.

Ok before starting we need to make a few assumptions for the proper working of this application.

1. This background is always static i.e. there is no addition or subtraction of objects in the background scene.
2. The background-color is always constant. It does not change with time.
3. The object that will be used for writing/painting is of a different color than the background to give us sufficient contrast between foreground and background.

We are ready to begin now. Let us start by installing necessary python libraries for our project using pip install. We will be needing Numpy and Opencv libraries. Now create a python project and create a new script. Import the required libraries into python script as shown below.

import cv2
import numpy as np

To capture a video, we need to create a VideoCapture object. Its argument can be either the device index or the name of a video file. The device index is just the number to specify which camera. 

cap = cv2.VideoCapture(0)

Now, we will write a while and capture image frames continuously. The following line of code is used to capture the frames.

ret, frame = cap.read()
frame = cv2.flip(frame, +1)

The second line is used to mirror the image frame so that it becomes easy for writing. The image we read is in RGB format. We need to convert this image into  HSV(Hue Saturation Value) format. 

frame2 = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
lb = np.array([105, 163, 201])               
ub = np.array([110, 255, 255])               

What are these lb and ub? These are the lower and upper bound thresholds. The thing is we want the background to be completely eliminated and only our object should be present in the image frame. For this purpose, we decide these threshold values. lb and ub are numpy arrays of 3 elements. Each element represents the value of Hue, Saturation, and Value. The HSV values of image which lie between lb and ub will be only kept in the image frame and rest will be discarded. The above values work fine for me. Your values might be different. The best way to define these values is by using trackbars.

Now we create a mask for our image which will only keep values lying between lb and ub and discard the rest of the values. 

mask = cv2.inRange(frame2, lb, ub)

The masked image looks something like this.

The white circle in the above image is my object. The rest of the background is eliminated. The above image looks but when you set the upper and lower bounds there are a lot of chances that noise will be added to the mask. That will look like a lot of small white dots in the image. To remove that noise we can use morphology. Opencv has a method called opening morphology which removes the small patches in the image. 

opening = cv2.morphologyEx(mask, cv2.MORPH_OPEN, Kernal)

The first argument here is the image on which you want to perform morphology. The second is the type of morphology you want to perform and the third is the kernal. Kernal is nothing but a matrix of any size defined by you with all the elements as '1'. After performing morphology the image looks like this.

You won't notice the difference here because the masking was extremely good. Now we need to perform a bitwise AND operation between the original image and the masked image. This process will generate the image with the original color of the object. 

res = cv2.bitwise_and(frame, frame, mask = opening)

Here, the first argument is the input image. The second argument is the destination image and the third argument is the mask that we are using for ANDing operation. The resultant image is shown below.

Don't get tired, we are almost there. 

To draw, we need to detect the contours of this object.  The contours method in Opencv library works only on the binary image(Of course to optimize the process). Therefore we will use the opening image(which is already binary) and use find contours method on it.

contours, hierarchy = cv2.findContours(opening, cv2.RETR_TREE,

                                       cv2.CHAIN_APPROX_SIMPLE)

The first argument in the find contours method is the source image. The second argument is the mode for finding contours and the third argument is the method to approximate the contours. Find contours returns a list of all the contours found in the image. This array is stored in the contours variable.

If you still have noise in your image frames after this much of filtering process, there are chances that you will get a lot of contours in the image frame. Then your job is to find the most relevant contour in the list of contours. Once you find the relevant contour you will have to find the center of the desired contour for further process. This can be done using the following piece of code.

if len(contours) != 0:
    cnt = contours[0]
    (x, y), radius = cv2.minEnclosingCircle(cnt)
    x = int(x)
    y = int(y)
    x1.append(x)
    y1.append(y)

The first line checks whether you found contour or not. After the filtering process, only one contour will be left. The second line stores the value of contour in cnt variable. We now find out a minimum enclosing circle around cnt using the code in the third line. The coordinates of the center of the circle are stored in x and y variables. Radius variable stores the radius of the enclosing circle. x and y can be float values. Therefore, we have to convert x and y into integer values because for drawing we need to input the coordinates in integer values. x1 and y1 is the list containing all the values of x and y coordinates(since we do not want to lose the information of the last coordinate we drew). 

Now the last and exciting part. Drawing. For drawing, you need to use the following piece of code.

for i in range(len(x1)):
    cv2.circle(frame, (x1[i], y1[i]), 2, (0, 255, 0), 2)

The for loop iterates from the start of the list to the last element in the list. For each iteration, we draw a small circle on the original image frame using the cv2.circle method. The first argument in this method is the image variable on which you want to draw the circle. The second argument is a tuple of x and y coordinates. The third argument is the radius of the circle. The fourth argument is the color of circle and the fifth argument is the width of the circle boundary.

cv2.imshow('Original Image', frame)

And the last step. Display the image using imshow method. The final image is shown below. Don't forget to release the memory at the end of the while loop.

To release the memory use the following code. 

cap.release()
cv2.destroyAllWindows()

Hurrayyy...!!!! And you are done with the project. Happy Writing In The Air...

Check out the full code here:- https://github.com/vibhor69meshram/Writing-in-Air

Check out code for selecting HSV values using trackbar:- https://github.com/vibhor69meshram/Set-HSV-values-using-Trackbar