Case Study of Visual Imaging Sensors in Smart Home

Many household appliances and devices promise to make our homes safer, more comfortable, and to save us time or make our lives easier. They can automate menial tasks or put the control of these devices at our fingertips by connecting to our phones. This functionality is made possible by a large number of digital sensors, from devices that measure air temperature, humidity and light conditions to motion detectors, gas sensors and more. From the information provided by such sensors, they have one thing in common: they are all point-by-point and lack spatial resolution. For example, a traditional temperature or light sensor will report the temperature or light intensity only at the point of installation in the room. It has no knowledge of the people in the room, their behavior,

In contrast, visual imaging sensors have a very fine spatial resolution. This is provided by the millions of detectors integrated into the sensor. In this way, they generate a rich unstructured information on which visual image analysis is applied to obtain structured information (for example, not just "there is someone at the doorbell", but how many people and who). Visual image sensors are already widely used in smart robots and home security systems. Now, the development of this technology is bringing other multi-pixel sensors to the field of home automation and smart homes. Thermal imaging sensors are a leading candidate, with the prospect of preventing fire hazards and guiding kitchen chefs, monitoring occupancy without sacrificing privacy, measuring our comfort levels, and remotely ensuring the health of infants and the elderly.

Next, let’s look at some specific cases.

Your safer, smarter kitchen

Why perform thermal imaging in the kitchen? The short answer is for safety, health, and convenience. In fact, cooking is the leading cause of home fires, accounting for 49% of such fires, according to the National Fire Protection Association (NPFA). Not all affected homes had faulty smoke detectors; rather, by the time smoke was detected, it was too late to eliminate the conditions that led to the fire. Thermal imaging can identify high temperatures that could cause a fire long before dangerous conditions develop, providing a clear path to minimize these costly and devastating incidents.

A natural way to provide thermal imaging in the kitchen is to integrate sensors into the range hood to implement a cooker monitor. Cooksy-Pro, shown in Figure 1, is the world's first smart cooking assistant product. In the picture, Cooksy-Pro is a copper-colored device installed under the range hood. From there, the thermal imager can continuously "see" the temperature distribution of the entire stove or stove.

Many cooking ingredients are sensitive to processing temperatures. Overheating them may not only deplete them of their nutritional value, but may even transform them into compounds that are not so friendly to our bodies. A thermal imaging camera above the cooktop can continuously "read" the temperature of the cooking meal and display it to us, helping us to make healthier meals.

“This led us to the concept of a cooking assistant that helps you create healthy and delicious meals every time,” said Steven Cartwright, CFO of Cooksy Corp. “Essentially, Cooksy enables you to share your cooking recipes or follow the recipes of others. Cooking times and temperatures are detailed in real time as the food is being cooked.”

Cartwright explains: "Cooksy combines visual and thermal camera sensors with sophisticated intelligence and connectivity in a device that easily fits above your stovetop. The device streams your cooktop in real time to your phone, allowing you to 'see' the temperature of your cooking ingredients and suggesting how and when to act on that information."

The benefits of stove monitors extend beyond the art of cooking and nutrition to the realm of kitchen safety. For example, monitors can detect unattended gas stove flames or overheated stovetops, overheated pots and pans (with or without oil), boiling and overflowing containers, and other hazardous situations. Figure 2 shows the boiling of a pot of water, with a lid, monitored over time by a thermal imager. While the directly observable temperature never approaches 100°C (212°F), the boiling point of water, the application of thermal image processing and analysis can detect when the water begins to boil, when it begins to overflow from under the lid, and when the gas burner has been turned off. In the context of smart homes and connected devices, the identification of such events can not only trigger an alarm, but also adjust or stop the power or gas supply if necessary to prevent accidents or fire hazards.

Thermal imaging analysis of stove applications (Source: Meridian Innovation)

Cooktop monitors aren’t the only place in the kitchen where thermal imaging sensors can be found. An early prototype of a thermal imaging-enabled microwave oven—originally demonstrated by Meridian Innovation at CES 2020—is shown in Figure 3 (a video of it in action can be viewed at facebook.com/watch/?v=685705472262649).

The device is able to heat food to the desired temperature, rather than for a fixed time at a fixed power. It is based on an off-the-shelf microwave oven that has been modified to integrate a thermal imager and alternative controls. This allows us to avoid the guesswork that we often have to do when choosing the power and time to reheat food. Finally, we can directly communicate our intention by selecting the desired food temperature and leave the technical details to the machine. This intention can be communicated via a wirelessly connected mobile device or through a touchscreen integrated into the modified microwave oven. Naturally, one can set the desired food temperature and monitor it as the food is heated.

Furthermore, with advances in TinyML and other initiatives to enable AI capabilities on embedded processors, this setup could allow a device to guess what we put in the microwave when heating it, automatically selecting the optimal temperature for us.

The necessity of timely delivery of thermal imaging cameras for detecting high temperatures and the complexity of building a good solution has led to many partners working to provide a number of reference designs to OEMs and ODMs. An excellent example of a thermal sensing solution powered by STMicroelectronics’ X-CUBE AI is described in the article “Arrow Electronics Launches AI Thermal Sensing Solution Powered by STMicroelectronics’ X-CUBE AI” published in European Business Magazine https://europeanbusinessmagazine.com/media-outreach/arrow-electronics-launches-ai-stmicroelectronics-x-cube-ai/. Another new solution made possible in collaboration with our partner Generalplus Technology is described in the section below titled “When Thermal Imaging Meets AI Chipsets”.

Air conditioning that cares about your comfort

HVAC systems are another example where integrated thermal imaging can have a positive impact. Most air conditioners in common use today have a simple temperature sensor embedded in the box to control the room temperature; this sensor provides the only feedback from the environment. Such systems can be considered first generation. In contrast, second generation systems with integrated thermal imaging cameras can know if there are occupants in the room, and where those occupants may be located in the room. This enables more efficient use of the air conditioning unit. For a small apartment or hotel room, a thermal imaging camera of a few hundred to a thousand pixels may be sufficient, while for larger living spaces, a sensor of several thousand pixels is required, coupled with a field of view of more than 90°.

Note that the content of the thermal image provides insight not only into the occupancy of the cabin, but also into the overall distribution of temperature throughout the room. Based on this more detailed information, the unit can automatically adjust the temperature, direction, and speed of the airflow to save energy and improve comfort.

As for comfort, there are various mathematical models that establish a direct relationship between a person’s average skin temperature and a person’s perceived comfort. We should remember that the purpose of air conditioning and HVAC systems is to ensure a certain level of comfort and well-being for the occupants, not just to cool or heat the air in a room to a preset temperature. This leads us to the concept of what we call 3rd generation HVAC systems. At this level, the heating or cooling units use higher resolution thermal imaging – around 5,000 to 20,000 pixels – and can obtain more detailed thermal images, directly measuring the average skin temperature of the occupants and estimating their level of comfort. This information allows for environmental controls that target true thermal comfort rather than temperature,

2nd generation air conditioners are already available on the market, thanks to the recent mass availability of cost-effective 1,000 to 5,000 pixel thermal imagers. We expect that rapid advances in thermal imaging technology will bring 3rd generation capabilities to high-end models within five years.

Health Monitoring for Infants and Elderly Baby monitors with video streaming have been on the market for some time and are a great way to share your child’s happy moments with friends and family and allow caregivers to monitor their babies remotely. However, embedded vision cameras cannot provide direct insight into the baby’s physiological state. Thermal imaging sensors can.

There is a lot of research and development going on to create baby monitors that integrate visual and thermal cameras and use rich AI computing engines to support the cameras. An example of a new product that is coming soon is shown in Figure 5. This field offers many possibilities.

For example, since elevated body temperature is a common symptom of illness, every parent would like to detect this symptom as early as possible, even in the middle of the night. To address this application, one can accurately track the normal temperature pattern of an infant when the infant is healthy and happy, and then automatically detect the occurrence of abnormal temperature patterns and alert the child's caregiver at the early stage of symptoms. It should be remembered that the healthy pattern of an infant's core body temperature exhibits significant diurnal variation, which may be close to 2° between its lowest and highest points. This pattern seems to be unique to each child and evolves slowly as the child's physiology develops, so capturing this pattern requires constant observation. So, in addition to health conditions, thermal and visual optical flow patterns can be further utilized in a feedback loop through smart baby phones to engage the developing brain in interactive games. Behavioral patterns and corresponding temperature markers can be analyzed simultaneously. These technologies can help caregivers nurture safe, smart and healthy babies.