[K230 Embedded AI Development Board Review] + Simple Sign Language Recognition (Part 2)

jinglixixi

[K230 Embedded AI Development Board Review] + Simple Sign Language Recognition (Part 2) [Copy link]

 

For the five fingers to be identified, their corresponding relationships with colors are shown in FIG10 .

Figure 10 Color correspondence

The drawing procedure for marking fingers with colors is:

for i in range(5):
        j = i*8
        if i==0:
           R = 255; G = 0; B = 0
        if i==1:
           R = 255; G = 0; B = 255
        if i==2:
           R = 255; G = 255; B = 0
        if i==3:
           R = 0; G = 255; B = 0
        if i==4:
           R = 0; G = 0; B = 255
        pl.osd_img.draw_line(results_show[0], results_show[1], results_show[j+2], results_show[j+3], color=(255,R,G,B), thickness = 3)
        pl.osd_img.draw_line(results_show[j+2], results_show[j+3], results_show[j+4], results_show[j+5], color=(255,R,G,B), thickness = 3)
        pl.osd_img.draw_line(results_show[j+4], results_show[j+5], results_show[j+6], results_show[j+7], color=(255,R,G,B), thickness = 3)
        pl.osd_img.draw_line(results_show[j+6], results_show[j+7], results_show[j+8], results_show[j+9], color=(255,R,G,B), thickness = 3)

        gesture_str=gesture_res[k][1]
        pl.osd_img.draw_string_advanced( x_det , y_det-50,32, " " + str(gesture_str), color=(255,0, 255, 0))

So, in addition to the three gestures tested, what other gestures can be recognized?

There are 9 gestures in total, namely fist, five, gun, love, one, six, three, thumbUp, and yeah.

The procedure for defining and recognizing gestures is:

# 根据手掌关键点检测结果判断手势类别
def hk_gesture(self,results):
with ScopedTiming("hk_gesture",self.debug_mode > 0):
angle_list = []
for i in range(5):
angle = self.hk_vector_2d_angle([(results[0]-results[i*8+4]), (results[1]-results[i*8+5])],[(results[i*8+6]-results[i*8+8]),(results[i*8+7]-results[i*8+9])])
angle_list.append(angle)
thr_angle,thr_angle_thumb,thr_angle_s,gesture_str = 65.,53.,49.,None
if 65535. not in angle_list:
if (angle_list[0]>thr_angle_thumb) and (angle_list[1]>thr_angle) and (angle_list[2]>thr_angle) and (angle_list[3]>thr_angle) and (angle_list[4]>thr_angle):
gesture_str = "fist"
elif (angle_list[0]<thr_angle_s) and (angle_list[1]<thr_angle_s) and (angle_list[2]<thr_angle_s) and (angle_list[3]<thr_angle_s) and (angle_list[4]<thr_angle_s):
gesture_str = "five"
elif (angle_list[0]<thr_angle_s) and (angle_list[1]<thr_angle_s) and (angle_list[2]>thr_angle) and (angle_list[3]>thr_angle) and (angle_list[4]>thr_angle):
gesture_str = "gun"
elif (angle_list[0]<thr_angle_s) and (angle_list[1]<thr_angle_s) and (angle_list[2]>thr_angle) and (angle_list[3]>thr_angle) and (angle_list[4]<thr_angle_s):
gesture_str = "love"
elif (angle_list[0]>5) and (angle_list[1]<thr_angle_s) and (angle_list[2]>thr_angle) and (angle_list[3]>thr_angle) and (angle_list[4]>thr_angle):
gesture_str = "one"
elif (angle_list[0]<thr_angle_s) and (angle_list[1]>thr_angle) and (angle_list[2]>thr_angle) and (angle_list[3]>thr_angle) and (angle_list[4]<thr_angle_s):
gesture_str = "six"
elif (angle_list[0]>thr_angle_thumb) and (angle_list[1]<thr_angle_s) and (angle_list[2]<thr_angle_s) and (angle_list[3]<thr_angle_s) and (angle_list[4]>thr_angle):
gesture_str = "three"
elif (angle_list[0]<thr_angle_s) and (angle_list[1]>thr_angle) and (angle_list[2]>thr_angle) and (angle_list[3]>thr_angle) and (angle_list[4]>thr_angle):
gesture_str = "thumbUp"
elif (angle_list[0]>thr_angle_thumb) and (angle_list[1]<thr_angle_s) and (angle_list[2]<thr_angle_s) and (angle_list[3]>thr_angle) and (angle_list[4]>thr_angle):
gesture_str = "yeah"
return gesture_str

So what are the forms of these 9 gestures?

It can be seen from the program that these 9 gestures are expressed by the state of 5 fingers, that is, whether the fingers are in a straight state or a bent state.

Figure 11 Finger arrangement order

In Figure 11, if the relationship with the array angle_list[] is established from left to right, and "1" is used to represent the straight state and "0" is used to represent the bent state, then the judgment condition is to see whether the array angle_list[] exceeds the judgment angle. If it is greater than the judgment angle, it is "0", and if it is less than the judgment angle, it is "1".

In this way, we get 9 gesture expressions, as shown in Figure 12.

Figure 12 9 gestures

After mastering the previous gesture definition rules, you can rewrite the gesture recognition program according to the sign language specifications to achieve the goal of sign language recognition.

Of course, it is still quite difficult to recognize complete sign language information, because sign language not only contains static gestures, but also includes more complex dynamic hand gestures, such as moving, circling, swinging, flipping, flexion and extension, as well as gestures coordinated with the body state. It is a very challenging research topic.

Jacktang

The challenge is to fully recognize the complete sign language information.

jinglixixi

Jacktang posted on 2024-10-21 07:50 Being able to fully recognize complete sign language information is a challenge

It's true that it's voluminous and complex, but there are some simple attempts that can be made.