Design of SmartMouse intelligent gesture input device that can replace mouse

1.1 Introduction

The technology of computer standard peripherals mouse and keyboard is relatively mature, but its operation is monotonous and there has been no breakthrough in technology for many years; touch screen technology has recently appeared and quickly became a hot topic. This technology requires special display touch devices to meet input requirements; in addition, the above two input methods are limited by the principles of input devices, which inevitably constrain users around display devices and input devices. In this industrial context, a humanized intelligent wireless input device is urgently needed to meet the increasingly popular multimedia presentation, office and personal computer application environments.

Under this premise, our team has developed a gesture input system called Smartmouse. Users can conveniently wear the handheld end of the device on their hands and connect the host end to the PC through the USB interface, so that they can control the PC through gestures. All the functions of the mouse can be completed through gestures, and the popular daily application software can also be smoothly operated. The biggest difference between the operation mode of Smartmouse and the touch screen and traditional mouse is that it gets rid of the dependence on the screen and fixed input device, so that it can better adapt to today's multimedia environment. It allows computer users to break away from the limitations of the traditional mouse and keyboard and establish a more humane communication method between users and computer equipment.

1.2 Project Background

At present, domestic input device manufacturers are mostly improving existing input devices. For example, the multimedia mice and keyboards launched by Xingui and Shuangfeiyan add shortcut keys for multimedia software on top of the basic functions of the original keyboard or mouse. Although such products increase the ease of use for users, most of them are expensive, and more importantly, their input methods have not changed qualitatively.

Huazhong University of Science and Technology has recently developed a human upper limb motion input device based on spatial motion measurement technology. The input method of this device adopts human bionics design and is groundbreaking. However, the system has a lot of hardware and a complex recognition algorithm, and its industrialization prospects are slim.

As early as 2001, Samsung and SenseBoard demonstrated virtual keyboards under development. Samsung's product in the R&D stage was called "Scurry". The two companies' products differ in recognition methods, but both bring revolutionary changes, that is, they can abandon the traditional mouse and keyboard. The problem with this input device is that users still have to imitate the keyboard typing when using it.

As of the end of 2004, the research and development of this keyboard is still in the wired connection stage. So far, there is no news of the official launch of "Scurry" because the device is expensive, costly, and difficult to commercialize.

The Wii game console produced by Nitendo is similar to this, using the sense of acceleration to control the game, but it can only be used as a gaming device and uses infrared transmission, so it is limited by angle and distance.

In the 1970s and 1980s, a large number of studies on data gloves were conducted internationally to capture hand movements, but such research stagnated soon after. Currently, the more popular approach to gesture recognition is to use a method based on visual image recognition. A camera is used to capture the image of the action, and image processing methods are used to obtain the action information. However, this gesture recognition method is very expensive and has very high requirements for software and hardware. It cannot be used as an input device for ordinary PCs.

Against the background of the rapid development of MEMS technology and ultra-low power wireless communication technology, the development of ultra-small, ultra-low power MEMS acceleration sensors and wireless communication modules has provided a guarantee for us to design and develop an input device with broad application prospects and that is more user-friendly, convenient and fast.

1.3 The advancement of this design

Smartmouse intelligent gesture input device is not limited to computer input, but can also be used in many fields such as multimedia device control, and has a very broad application range. In addition, there is no input device on the market that uses acceleration sensors and integrates wireless communication and USB plug-and-play technologies. The emergence of Smartmouse intelligent gesture input device will definitely attract a group of mid-to-high-end users who pursue new things.

As a product, Smartmouse intelligent gesture input device has the advantages of existing products and overcomes their shortcomings; it combines humanity, ease of use and portability, and provides users with a new operating concept, which will surely be favored by the market.

2. Demand Analysis

2.1 Functional requirements

Smartmouse intelligent gesture input system is divided into handheld end (gesture data acquisition module) and host end (data receiving and processing module), and the handheld end and host end are connected wirelessly. The handheld end can be conveniently worn on the user's finger to collect the user's gesture data; the host end can realize the interface with the PC and can receive and intelligently process the gesture data.

The functional block diagram of the Smartmouse intelligent gesture input device is shown in FIG1 .

2.2 Performance requirements

Smartmouse intelligent gesture input device can accurately identify various gestures of people and smoothly perform corresponding operations such as moving the mouse, left button, right button, deleting, confirming, canceling, moving files, etc. according to the gestures. The signal acquisition end can collect the information of human hand movements in real time and completely, and transmit it to the host end on the PC or receiver through the 2.4G wireless frequency band in the corresponding data format after sorting; the host end can instantiate itself as a mouse and keyboard composite device in the form of a USB interface without any driver; after receiving the data, it processes the data intelligently, identifies the meaning expressed by the gesture, and feeds back the corresponding operation to the PC or receiver. The recognition rate of gestures should reach the level of consumer electronics.

3. Solution Design

3.1 System Function Implementation Principle

The Smartmouse intelligent input system uses an acceleration sensor to collect and preliminarily identify the user's hand movement data. Combined with wireless communication and USB interface technology, it transmits the obtained hand movement information to the PC or receiver through the USB interface through a simple and intelligent gesture recognition processing algorithm created by itself, allowing users to control the PC in a way that is more in line with their body movement habits, giving users a new operating experience.

The hardware structure block diagram of the Smartmouse intelligent input system is shown in Figure 2. (1) Gesture data acquisition

The handheld terminal of the Smartmouse intelligent gesture input system needs to continuously and accurately sample the user's hand motion data, and transmit the collected hand motion data to the host terminal connected to the USB interface of the PC in a wireless form. This system uses the dual-axis MEMS acceleration sensor chip MXC6202xMP produced by MEMSIC as the main gesture sensing device, which can achieve low power consumption and high precision while correctly sensing hand motion data.

(2) Data reception and processing

The host end connected to the USB port of the PC, that is, the data receiving and processing module, instantiates itself as a mouse and keyboard composite device of the PC, and intelligently processes the gesture data received wirelessly through the recognition algorithm, and transmits the processed results to the PC in the form of mouse and keyboard operation data. The connection process between the data receiving and processing module and the PC does not require the installation of any driver, and can be plug-and-play. The recognition rate of gesture data can meet the requirements of consumer electronics. In this system, the AVR microcontroller is used as the core processing unit, which is responsible for enumerating the device as a USB composite device according to the USB protocol, managing the wireless transceiver module, and managing the handheld terminal through the wireless module. At the same time, the received gesture data is analyzed by the self-created gesture recognition algorithm and the results are sent to the host end.

(3) Data Transmission

The transmission of the collected gesture data adopts the 2.4G frequency band which is currently widely used in the field of consumer electronics products. It accurately transmits data and control information while ensuring bandwidth, and can also realize power consumption reduction operations such as device sleep.

(4) Host and PC interface

The host side, namely the gesture data processing module, is connected to the PC through a USB interface, instantiates itself as a keyboard and mouse composite device of the PC, and realizes plug-and-play without any driver.

(5) Power supply

The handheld terminal uses a micro lithium battery as a power source, which is small in size and can be repeatedly charged and used. At the same time, it reduces system power consumption through operations such as sleep and shutdown.

The host side is powered by the host USB interface and can be put into sleep mode when not in use for a long time.

(6) Intelligent recognition rate

The Smartmouse intelligent gesture input system needs to operate the PC smoothly, has high requirements for the gesture recognition rate, and can perform complex operations on the popular demonstration software. In this design, the received gesture data is processed by a 32-bit AVR microcontroller, and the gesture recognition algorithm created by the team is combined to identify the meaning of the gesture.

3.2 Hardware platform selection and resource allocation

Since this system contains many modules and complex algorithms, and has high requirements for power consumption and stability, the platform selection is divided into two steps:

In the function realization stage, the main work is to improve the gesture recognition algorithm, USB interface protocol, wireless communication protocol, etc. All parts use ready-made modules. The wireless transceiver part uses the eZ430 wireless transceiver module produced by Ti; the algorithm hardware implementation and USB interface management use the EVK1100 development board provided by Atmel.

During the work improvement stage, on the basis of realizing gesture recognition algorithm, USB interface protocol, wireless communication protocol, etc., a handheld terminal was made using the ATXMEGA32D4 microcontroller and wireless transceiver chip AT86RF212 produced by Atmel; the AT32UC3L064 microcontroller and wireless transceiver chip AT86RF212 were used to realize data reception, gesture recognition algorithm and USB interface management on the host side.

3.3 System Program Architecture

In this system, the software design is mainly divided into gesture data acquisition, wireless communication protocol, gesture data recognition algorithm and USB interface protocol.

Gesture data collection: Under the control of the host, gesture data is collected and packaged in a certain format for transmission;

Wireless communication protocol: manages the connection and data transmission between the handheld terminal and the host terminal;

Gesture recognition algorithm: It is the core algorithm of this system. It uses a self-created intelligent recognition algorithm based on acceleration sensor data to process gesture data and derive the meaning of gestures.

USB interface protocol: manages the connection between the host and the PC or receiver, enumerates the host as a computer mouse and keyboard composite device, and implements plug-and-play without drivers.

3.4 System Program Flow

The programs in this system are mainly divided into handheld programs and host programs. The program flow chart is shown in Figure 3 below.

3.4 Expected Results of the System

With the help of the EVK1100 series development board and Ti eZ430 wireless module provided by Atmel, a prototype was developed to realize all the functions of the mouse and control simple browsing software, while achieving an accuracy rate of more than 90%. A primary model of the self-made product was developed to realize all the functions of the mouse and have the "air brush" function, which can smoothly control simple browsing software and achieve an accuracy rate of more than 90%.