Analysis of Multi-touch Smart Home Platform Application

introduction

In recent years, multi-touch has become a new way to replace the traditional human-computer interaction. It abandons the keyboard and mouse, realizes multi-person interaction at the same time, and is a revolutionary innovation in human-computer interaction. Unfortunately, the technology is still in its early stages, and many Multi-Touch products are only for high-end or military users, and the price is very high. This is unaffordable for the majority of consumers. In addition, there are currently very few software applications based on Multi-Touch, and most of them remain in the function of game entertainment, which also limits the development and application of this technology.

Therefore, it is urgent to make Multi-Touch technology affordable and marketable. Considering the fresh and efficient human-computer interaction of Multi-Touch, we apply it in daily office life and propose a set of Multi-Touch application solutions for the vast number of office workers to improve work efficiency and increase work fun.

Multi-touch smart home platform

1. Brief description of FTIR principle

As the original proposed implementation method of Multi-Touch, FTIR (Frustrated Total Internal Reflection) has always been the most popular and most thoroughly studied implementation method. The principle is to place an infrared LED on the side of a thicker transparent plexiglass so that the infrared light is totally reflected inside the glass. When a finger touches the surface of the plexiglass, the balance of total reflection will be broken, so some light will be scattered below and captured by the camera. Since plexiglass is completely transparent, a layer of rear projection screen must be attached to the upper or lower part of the plexiglass. Overall, although the effect is good, this solution is too complicated to implement because it requires reasonable adjustment of the side incident light and has high requirements for materials.

Figure 1

2. Brief Introduction of DI Principle

The principle of DI was first proposed by Microsoft, and its implementation principle is shown in Figure 2. In the figure, the surface that the user touches is not necessarily made of plexiglass, and the key to the entire technology lies in the infrared scattering light source below the screen, which evenly scatters infrared light on one side of the multi-touch screen. When the finger touches the surface, a shadow will appear in the image because the finger blocks the upward scattering route of the light. After further image processing, the position information of the finger can be extracted.

Figure 2

In terms of specific implementation, DI can be made simpler. Figure 1 shows the principle of Back DI (back projection DI), where you can see that infrared light is incident from below. The advantage of this Back DI is that the light source is below the touch screen and can be controlled by itself, and the brightness of the infrared light can be reasonably controlled; it not only achieves shadow capture, but also object recognition. However, this solution requires the reasonable placement of the infrared LEDs, and ensures that the brightness of all infrared LEDs is almost the same, and also achieves the most ideal uniform lighting effect, so it is very difficult to implement.

In actual operation, we tried a Front DI (external light DI) solution. In contrast to Back DI, Front DI is responsible for collecting infrared light from outside the touch screen. Similarly, due to the light shielding effect of fingers, shadows will also be captured. Although it does not have the high-definition resolution of Back DI, it is sufficient for desktop applications.

After analyzing and practicing the FTIR implementation solution, considering the cost and complexity of the product, we tried the DI solution, which was more ideal. In addition, DI is cheaper, has a simple system, and is easier to assemble and implement. Therefore, the DI solution was selected for the final work.

3. System Composition Structure

The whole system consists of four main parts: PC, camera module, projector and user's desktop. In general, the hardware composition of the system is relatively simple. The camera prototype uses a high-definition camera product from Logitech. After removing the visible light filter and adding an infrared filter, the system collects finger position information. The projector is used as the display part, and Epson's EB-X7 is selected. For the outer frame, we designed the composition structure of the platform frame and the installation and working methods of each component.

4. Software

4.1 Software Overview

In the system design, we chose to use C# to write the program, and used UP, DOWN, and MOVE to represent the finger touch events, the finger moving on the touch screen, and the finger leaving the touch screen. We also used CCV to write the finger recognition program for the entire touch screen and encapsulated it into a dynamic link library to facilitate the subsequent program calls.

CCV (Community Core Vision) is an open source/cross-platform software that solves computer vision and machine sensing. It can identify infrared light reflected by fingers, and after data processing, the image becomes clearer, and then the event is passed to the pre-written DLL dynamic link library file, so that the software can recognize the specific finger movement.

Users can draw graphics on the display buffer. The graphics system uses GDI+, which provides powerful drawing functions, which is more suitable for the applications provided by this system. A pointer to the GDI+ drawing object can be obtained through a function. After the user finishes drawing, the display buffer must be updated to the parent window to finally display on the screen.

Each message processing queue corresponds to an independent message processing thread, which is responsible for processing various events, especially user operation events. Events are sent to the message processing queue in the form of messages. The message processing thread takes the message from the queue, processes it and returns. There are two modes for message sending, one is to return after sending, and the other is to wait for the message to be processed before returning. The message processing thread is not destroyed until the window is closed. The message queue is the multi-threaded feature of the window system, which allows multiple windows to work simultaneously on Intel dual-core processors, greatly improving the performance of the window, especially when multiple users perform complex operations at the same time, which can greatly improve the parallel capability.

The operations of the message queue are mutually exclusive, and only one thread can add or delete elements at a time, so a lock mechanism is needed to protect it.

For user operations, the parent window sends the user operations in the child window to the child window through messages, and so on. Finger messages are particularly worth introducing. Finger messages are divided into finger down, finger move and finger leave. The window recognizes different fingers through the finger ID in the message parameters.

The operation of multiple fingers is based on an active finger table maintained by a window, which records all the fingers bound to this window. The table is implemented as a balanced binary tree to improve speed.

The window designed by this system can be rotated at any angle, so that users can watch from all angles, which is suitable for small meetings. The window has built-in support for a circular menu, which notifies the window user of the selection through a message. The window also supports keyboard operation, and the keyboard notifies the window through character messages.

4.2 Application Functions

4.2.1 Whiteboard

Whiteboard is a common function in meetings, especially in meetings related to design. It is very effective if designers can discuss face to face and express their opinions on the whiteboard. Traditional paper whiteboards can provide an environment for multi-person discussion, but it is not convenient to modify the design. General network whiteboards can provide a platform for communication, but they cannot allow many people to discuss and operate at the same time. Multi-Touch technology perfectly solves their problems and provides people with a platform for full communication.

4.2.1 Virtual Keyboard

In the application, users not only need to draw, but also need to type in many cases. The whiteboard function alone cannot meet the customer's requirements for typing standard Chinese characters, so a virtual touch keyboard is added. Text can be input on the touch screen using the keyboard. The touch keyboard also has Sogou input method, which can input Chinese characters.