CMOS image sensors provide visual perception for autonomous vehicles

Enabling fully autonomous vehicles requires the integration of information from multiple sensors, with camera information being perhaps the most important. These cameras must be able to continuously capture even the tiniest details in various conditions to ensure the safety of vehicle passengers and other road users. This article will explore the key features to look for when selecting an image sensor to provide the excellent combination of features needed for autonomous vehicles.

Figure 1. Image sensors are an important part of autonomous driving

The image sensor is responsible for converting photons into electrons, which are then stored as digital image data. Today, image sensors are widely used in a variety of camera vision applications, including smart factories, medical imaging, and automotive. The choice of image sensor depends on the level of performance required for a given application. When deciding which image sensor to use, it is first necessary to understand the desired frame rate, expected lighting conditions, and desired system tolerances, but a lack of engineering expertise in vision systems can make the process daunting. Fortunately, there are several standards that can be used to compare the performance of different image sensors.

Sensor resolution and sensitivity

In applications where measurement accuracy is crucial, sensor resolution is a critical metric as it determines the number of pixels on the photosensitive surface rows and columns that capture the image. The minimum number of pixels required depends on the minimum features in the image that need to be detected. While it is theoretically possible to resolve a single feature of an object with only 2 pixels per dimension, the lack of contrast and image noise means that at least 4 or 5 overlapping pixels are needed in real-world applications to fully resolve a feature.

Sensitivity, which measures the efficiency of a sensor in converting photons into electrons, is important in almost all applications. It measures the time and illuminance (illumination) it takes for the sensor to recognize the available image. Sensitivity is also affected by sensor noise, so the signal-to-noise ratio (SNR) is an important metric. Sensors with high signal-to-noise ratios provide higher quality images in low-light conditions. Sensitivity is particularly important in surveillance and medical applications, where high-quality images are required even in low-light conditions. While image sensor fact sheets provide sensitivity metrics, it can sometimes be difficult to compare the sensitivity of sensors from different manufacturers. Autonomous vehicles require a high level of detail, even in low-light (near dark) conditions, so the two key metrics of resolution and sensitivity must be carefully considered when selecting an image sensor.

Dynamic range and frame rate

Dynamic range refers to the ratio between a pixel's minimum (noise) signal and its maximum trap capacity, which defines how many different brightness levels are present in an image. This is a particularly important feature in applications with extreme light conditions. Dynamic range metrics are based on the European Machine Vision Association (EMVA) 1288 standard and are measured in decibels (dB), so they can be easily compared against most sensor fact sheets.

 

Figure 2: The image on the right has a higher dynamic range

Frame rate is a sensor's speed metric, measured in images (frames) read per second (fps). High frame rates are important in applications that capture fast-moving objects because short exposure times are required to prevent blurring and reduce motion artifacts. As the number of sensor pixels increases, the maximum frame rate achievable decreases. For example, higher frame rates can be achieved with a low-quality video graphics array (VGA) image sensor, while the same effect cannot be achieved with a full-resolution 20-million-pixel sensor. Sensors that support "region of interest" (ROI) can combine high frame rates with high resolution. These sensors determine one or more areas of the image to be processed, ignoring all other areas of the image. This effectively reduces the overall image resolution, allowing for higher frame rates. Like standard cars, autonomous vehicles encounter various lighting conditions during high-speed driving, so high dynamic range and high frame rate are critical characteristics when selecting an image sensor.

Movement and color

The two readout methods used by CMOS image sensors are rolling shutter and global shutter. The rolling shutter method reads out the sensor's pixels line by line during exposure, making it a very fast technique. It uses fewer transistors per pixel, is less noisy, more sensitive, and less expensive than a sensor that uses a global shutter. Rolling shutter sensors are recommended for applications requiring high dynamic range. In the global shutter readout method, each pixel is exposed at the same time, so there is no snap delay between pixel rows. However, this method is expensive and difficult to implement.

In applications such as measurement recording and presence detection, the use of black and white image sensors is acceptable, but many applications now require color images. However, the black and white sensor also has some advantages. In order for the sensor to provide a color image, RGB filters need to be arranged in Bayer mode at the pixel level. However, Bayer color interpolation results in reduced detail and overall measurement accuracy. Therefore, color sensors are only needed when color information is required in the application. Obviously, autonomous vehicle applications need to capture color images and use rolling shutter sensors.

Pixel size

There is a misconception that the larger the pixels, the better the image quality. While larger pixels have more area to use to collect light, it doesn't mean you can get a higher quality image. It's important to note that factors such as resolution and pixel noise metrics also play a significant role in determining image quality. Smaller pixels tend to have lower dark signal inhomogeneity (DSNU); At higher temperatures, dark signal inhomogeneity can limit low-light performance. In some cases, sensors with smaller pixels will outperform sensors with larger pixels. When designing a camera system, it is essential to consider a good balance between speed, sensitivity, and image quality characteristics to achieve excellent performance.

CMOS Sensor Solutions

The Hyperlux sensor family is the second-generation Super Exposure pixel technology platform from onsemi that delivers outstanding performance in automotive imaging applications ranging from 3 million pixels to 8 million pixels and beyond. With over 120 dB of flicker-free (FF) single exposure and 150 dB of ultra-high dynamic range (HDR) with LED flicker cancellation (LFM) technology, the sensors deliver consistent image quality and dynamic range over automotive temperature ranges without changing settings in different lighting conditions, reducing latency and improving safety. The 2.1 μm image sensors in the series include flexible features such as intelligent ROI, pixel binning, array windowing, and dual outputs. These features can simultaneously output flexible data formats such as data at different resolutions (ROI) and different integration times. The sensor is designed specifically for ASIL-D systems, and its sophisticated real-time safety mechanisms and fault detection capabilities contribute to more advanced driver assistance systems (ADAS).

Understanding is the key to choice

This article describes the key features to consider when selecting a CMOS image sensor for autonomous vehicle applications, including resolution, sensitivity, speed, dynamic range, motion, and color. ON Semiconductor Hyperlux image sensors offer an excellent combination of these characteristics, making them ideal for a wide range of automotive applications.