Bridging solutions for mobile-related platforms such as smart car ADAS and in-vehicle infotainment systems

Over the past decade, the development of smartphones and their application ecosystem has had a profound impact on the innovation of mobile-related applications in automotive electronics. Automakers have begun to apply the same processor platforms used for smartphones to the next generation of cars, which has improved the driving experience while making cars safer and cheaper. In addition, automakers also want to take advantage of the scale and application support of today's mobile industry. Although many processor manufacturers can now provide automotive-grade mobile platforms, the processors are still designed with smartphones in mind. In many cases, such platforms need to be adjusted to meet the needs of automakers. The use of FPGAs can quickly implement low-cost bridging solutions, making existing platforms perfect for automotive applications.

Design engineers face many challenges whenever they try to adapt an existing platform to a completely new application. The automotive industry is no exception. FPGA bridging solutions can solve many of these problems.

Common Challenges in Automotive Platforms

Depending on the system design, specific concepts and requirements, automakers may need multiple components to achieve a suitable solution. Lattice Semiconductor offers a variety of FPGAs that can implement various solutions that meet the above requirements.

• Cross Link ™ devices – low - power , low-cost sensor and camera interface bridging solutions that can be used to aggregate and multiplex data from multiple camera interfaces ( vision or radar) and send it to the SoC.  

• ECP5 ™  – an  excellent and comprehensive bridge chip for automotive applications . The ECP5 is a low-power , low-cost device that uses SERDES to easily connect devices via Ethernet or fiber. The DSP blocks can perform pre- and post-processing of sensor data or convert video data into various standard display formats.

• MachXO ™  – Small form factor FPGA with the highest number of I/Os per LUT, enabling easy I/O expansion and simple conversion of video data. 

The following are applications of bridging solutions in specific areas.

Embedded Vision

While embedded vision may sound a bit like science fiction, it is actually one of the most exciting application trends today. As the name implies, it refers to the ability of machines to see and collect visual data about their surroundings. The technology enables manufacturers to realize machines and cars that can "see", sensors that can sense, and computers that can make informed decisions autonomously.

Data collected by embedded vision systems can help machine learning technologies become smarter and more capable. Human-machine interfaces (HMIs) support smarter operation and control of machines, while machines provide feedback to help with decision making. A focus of computer vision is the use of multi-dimensional images to measure distance and depth in target recognition and stereo vision applications. Using sensor fusion technology, these systems are able to combine data from different sensors into meaningful and useful information for processing. At the same time, these systems can communicate securely over high - speed connections to enable smart cities, smart factories, and smart cars.

Lattice's programmable products provide co-processing, bridging and interconnect solutions to enable intelligent functions for edge applications. The flexibility of FPGAs helps speed time to market, reduce cost and power consumption, and minimize the size of industrial and automotive equipment.

Advanced Driver Assistance System (ADAS) Bird's-Eye Panoramic Vision

Modern processors typically have only 2 camera interfaces, but many ADAS systems require at least 4 interfaces, and in some cases up to 8 cameras to accurately perceive the car's surroundings. Another challenge facing design engineers is how to process the image data collected by these cameras. These data requirements usually require a large-size ISP (Image Signal Processor) connected to the processor. Lattice ECP5 FPGAs are designed for parallel processing to accelerate the processing process. The device provides a large number of I/Os, which can easily connect to multiple cameras. In addition, their co-processing capabilities can improve the efficiency of the processor.

The ECP5 can connect to multiple cameras and perform basic or advanced image processing tasks to provide the processor with the highest quality image for decision making. An example is the bird's eye view camera and front and rear side view systems in modern cars. The bird's eye view system is able to achieve a panoramic view from 20 feet above the car looking down. This is achieved by stitching together the data from 4 (or more) wide angle (FoV) cameras. The ECP5 aggregates all the camera inputs, stitches the images together, performs fisheye correction (which is caused by the FoV wide angle lens), performs white balancing, maximizes image quality using HDR, etc., and then sends the best quality image to the processor.

In the above applications, a single ECP5 device can replace multiple processors with limited camera interfaces. This provides design engineers with a way to reduce system cost and power consumption.

The following factors must be taken into consideration when designing a system of this type:

• Required number of video channels and resolution

• Fast and stable transmission

• Pre-process images to reduce load on the main ADAS processor

Video channels and resolution

Lattice CrossLink devices enable design engineers to aggregate data from multiple image sensors and transmit the data to an application processor via a single CSI-2 interface. The CrossLink is small in size and can be placed close to the sensor to increase design flexibility.

Lattice also provides several off-the-shelf IPs for CrossLink devices that enable bridging and aggregation between MIPI D-PHY and other camera or display interface standards. This allows designers to use cameras or displays with legacy interfaces such as OpenLDI, CMOS, LVDS, as well as modern MIPI CSI-2 or DSI interfaces.

DisplayPort is another open standard that is gaining popularity in the automotive industry. It also has lower electromagnetic interference (EMI) due to the embedded clock, which reduces the number of channels required. It uses a micro-packet protocol and can be easily extended to support higher resolutions and longer distances. Using the dedicated SERDES channels provided by ECP5, DisplayPort (DP) or embedded DisplayPort (eDP) can be used to implement applications such as instrument systems, dashboard/navigation displays, and rear-seat entertainment systems.

Combined with Lattice's automotive-grade MHL/HDMI ASSP solution, it can easily realize connection applications to modern smart devices such as smartphones and tablets.

Lattice MHL/HDMI ASSP for interconnection between smart devices and cars Data transmission inside the car

Data transmission in the car

When there are many sensors in a car to manage, point-to-point wiring becomes much more complex and expensive. Connecting multiple sensors to the ECP5 at the rear of the car allows data to be sent quickly and efficiently to the front of the car using a single cable, reducing weight and cost, and simplifying servicing.

The ECP5's 3.2Gbps SERDES provides strong support for many network and transmission applications. The device can be used to drive in-vehicle networks such as BroadR-Reach or Ethernet, to connect PHY chips or to collect sensor data for use in the car. The ECP5 also supports an emulated CSI-2 interface that can be used to connect multiple cameras or radar devices.

Using ECP5 to realize in-vehicle sensor aggregation and network connection

Video Preprocessing

ECP5 can also be used to pre-process video. As the automotive industry begins to adopt mobile processors, design engineers have to face a variety of new interfaces. For example, although mobile phone processors usually have a single DSI output to connect to the display , the mainstream displays in the automotive market use LVDS . FPGAs can implement pre-processing of video signals of different resolutions and bridging between different interfaces. ECP5 can be used to build a video bridging solution between the DSI or FPD-Link output of the application processor and the LVDS input of most automotive displays . In addition , ECP5 can also be used in in-vehicle infotainment applications to split one video output into two outputs for rear seat displays , or to crop and format a single video output according to specific video resolution requirements .

Advanced driver assistance systems with radar/ lidar

Radar and LiDAR are not only used in autonomous vehicles , but also as driver assistance tools. Specifically, they are used to detect dangerous objects and conditions , allowing the car to notify the driver or automatically take action to protect the safety of passengers when necessary . Although these systems are still in development, it is foreseeable that future cars will not only be limited to image signal processing of images received by the camera , but will also adopt radar - based proximity sensors and LiDAR- based terrain sensors. Radar and LiDAR systems take full advantage of the high-speed MIPI interface and output data using CSI-2 . After combining this with the resources of the processor , design engineers will once again face the challenge of a limited number of MIPI CSI-2 interfaces or different interface types . Automotive-grade CrossLink devices can be used to aggregate data from multiple sensors, or simply used as a bridging solution to convert CSI-2 data into a format acceptable to the application processor interface .

For example , many modern 77 GHz radars use CSI-2 to interconnect with the ADAS MCU . Using the topology shown below, the ADAS system can connect to multiple radar devices through multiple CrossLink bridges to collect data in all directions around the car and send the data to the application processor through a parallel interface .

CrossLink for Radar Bridging Applications in ADAS Systems

ADAS also requires expensive image signal processing (ISP) resources to identify objects or focus on specific objects instead of general images. As machine learning algorithms for decision-making continue to advance , cars become more automated, and FPGAs provide design engineers with the flexibility they need . When the decision-making power belongs to the computer , it must decide how to deal with road conditions, objects on the road, and keep the driver safe in all situations .

Lattice ECP5 has a full set of HDR ISP from Helion Vision, GmbH, which can be used to improve the quality of captured images . Based on the higher quality images , the target recognition function can be easily realized using a microprocessor soft core .

Multi-screen video display for in-vehicle infotainment system ( AIS)

If automakers want to implement an AIS system to provide information and entertainment services for the entire car , this will need to support multiple screen outputs , a rear screen camera input, and video and data input from mobile devices.

Mobile processors can usually only drive one DSI display . The automotive field uses various displays with interfaces such as LVDS , DSI or DisplayPort . For traditional display interfaces like LVDS, ECP5 can convert DSI to LVDS and ensure that the output resolution is compatible with the display. ECP5 also supports displays with DisplayPort (DP) and Embedded DisplayPort (eDP ) interfaces . For processors that do not support DSI, CrossLink can drive DSI displays through bridging . Design engineers can also use MachXO devices to support multiple displays and use HDMI automotive ASSP to input video from mobile devices.

Customer application case studies

A Lattice customer has implemented a similar solution using an ECP5 FPGA. The customer’s product provides a bird’s eye view panoramic vision solution while processing image data and providing hardware acceleration. The solution uses 4 cameras (front, rear, and sides ) mounted on the vehicle body. The video data is processed and seamlessly stitched to provide a panoramic image around the car .

ECP5-based surround vision application block diagram

As shown in the figure above, one ECP5 can replace multiple ARM processors to achieve the bird's-eye view panoramic vision function . The images captured by all four cameras are processed and merged , and with the help of the ISP , a full set of 360- degree panoramic surround vision functions can be achieved , such as white balance, fisheye correction and defogging . This solution can be installed during the car manufacturing stage or added as an aftermarket product .

Each camera provides 720p analog HD images . The final surround view solution is 1080p 60 fps. Designers replaced multiple ARM processors with a low-cost, low-power ECP5 FPGA. A low-end ARM processor is also required for initial calibration and video encoding.

Summarize

The flexibility of FPGAs has proven to be extremely valuable in many industries . In the automotive market, FPGAs can help solve the mismatch between advanced entertainment and safety systems and mobile processors . This business model has obvious advantages, allowing automakers to use market-proven products from the smartphone world and quickly adapt to the changing needs of the automotive world .

FPGAs can also play a role in other areas . FPGAs have begun to appear in applications such as motor control and are proving useful in the automotive industry as well. It is certain that as long as the electronic systems of automobiles continue to develop and systems such as ADAS continue to drive the advancement of autonomous vehicles , design engineers will need to integrate more sensors into these systems. This will drive the demand for FPGAs that can implement cameras, sensors , video and higher-speed connectivity in the evolving market .