Beyond RGB: The Wide World of Cameras

In our previous article, we explored the new wave of physical AI innovation driving advanced perception and visualization.

A typical physical AI deployment does not depend on a single camera – it combines various cameras in different conditions and for different objectives. Consequently, the physical AI hardware stack is multimodal – combining visible-light cameras with complementary depth and motion sensing, as well as non-visible sensors that can operate in poor lighting or occluded settings. Some of these are detailed here.

RGB (visible-light) cameras: These capture conventional 2-D imagery in the visible spectrum – the standard camera form we are all accustomed to. In most stacks, this remains the primary sensor for scene understanding, object detection and recognition, and behavioral intent identification, thereby acting as the backbone of the entire system. RGB cameras have been around for decades and are therefore the most mature technology in this ecosystem – offering high performance and reliability at low costs.

Recent advances include the shift towards global shutter, which came to the fore in late 2023 when Sony launched the first full-frame global-shutter mirrorless camera. Global shutter technology reduces distortion and enables high-fidelity, even during fast motion, by capturing the entire focal plane at once rather than reading one line at a time. This is especially valuable for robotics, autonomous vehicles, and industrial applications that require synchronization with other sensors.

RGB cameras are central to physical AI and serve as the key semantic layer, thanks to their semantic richness, which captures objects, text, colors, textures, and behaviors. However, they are especially sensitive to light conditions (glare, shadows, and low light), occlusion, and clutter; they are also prone to domain shifting when used across environments. Consequently, they are necessary but rarely sufficient.

Near-infrared (NIR) cameras: These are rarely used as standalone cameras but are often used alongside RGB cameras to support their operation in lower-light conditions. NIR cameras capture reflected light from objects, similar to a standard camera; however, they are sensitive to wavelengths between ~700 and 1200 nm (i.e., outside the visible spectrum). This enables them to work in low-light conditions and be less sensitive to changes in visible lighting. They are also less disruptive to humans due to their usage of illumination outside the visible spectrum. Consequently, NIR cameras are often used in conjunction with RGB cameras in driver monitoring systems (DMS), for industrial inspections in controlled-light settings, and for night-vision security applications.

Medium-wave infrared (MWIR) and long-wave infrared (LWIR): Unlike earlier cameras, MWIR and LWIR cameras detect emitted thermal radiation instead of imaging light. They distinguish objects based on their “emissivity,” a measure of how much heat a given surface radiates. Consequently, these infrared cameras operate independently of lighting conditions and are especially effective at identifying people and machines, regardless of lighting, camouflage, shadows, or weather conditions.

These cameras can be used for different applications, including:

• Temperature measurement – for example, in industrial settings to detect overheating equipment, or in healthcare to detect fevers.
• Night vision – for example, to detect intruders in a security perimeter.
• Redundancy alongside RGB cameras

MWIR and LWIR cameras need to be carefully calibrated based on the application and the ecosystem in which they are used. Furthermore, a thermal camera generally only acts as a state sensor and not as an identity sensor; it also has low spatial resolution and is therefore poor at identifying small objects or distinguishing between occluded emitters. Finally, these cameras tend to be very expensive, often costing up to 20x as much as a typical RGB camera. As a result of these factors, MWIR and LWIR are typically used only as safety and redundancy channels in a physical AI stack, except in specialized scenarios such as temperature measurement or leak detection.

Depth cameras: Typical RGB cameras capture the world in two dimensions; depth cameras are required to add a third dimension. Depth cameras are broadly segmented into two types: passive and active.

Stereo cameras are the typical passive depth cameras – they usually use two different imagers, which capture the scene in 2D; the depth of the scene is then calculated by correlating corresponding points across the two imaging views. This is a very mature and stable technology that has been proven to be reliably effective at long range and in outdoor conditions. However, calibration is critical, and the system must be rigidly mounted to prevent estimation errors; it also struggles with transparent or glossy surfaces and multiple reflections. While they are commonly used in autonomous robotics and vehicles, their failure leads to one of the most common yet-to-be-solved problems for autonomous robots – the ability to navigate indoor environments with glass doors, windows, or furniture (for example, a typical office).

Time-of-Flight and structured-light/infrared cameras are active depth cameras. These emit modulated waveforms (visible light or infrared) and estimate distance based on the time taken for the reflected signal to return. These can produce very precise 3D images with texture in real time; however, because they rely on light reflection, they are highly sensitive to ambient lighting and multi-camera interference. They also tend to be more expensive and more power-hungry than stereo cameras. However, they are the most effective choice for computer vision systems in constrained (usually indoor) environments for short-range tasks – for example, pick-and-place robots in factories, or for volume measurement of bins and pallets in a warehouse.

These depth cameras act as the geometry layer on top of the semantic layer provided by RGB cameras and enable a richer understanding of the shape, size, and position of objects.

Event-based cameras: Event-based cameras (often known as neuromorphic cameras) are one of the newest technologies in the world of cameras. They are inspired by biological sensors – for example, our eyes. Unlike other cameras that capture and output frames, neuromorphic cameras detect changes in light at a pixel level to identify movement or changes in intensity. Each individual pixel operates asynchronously, and the changes in light are correlated with changes in pixel coordinates, time, and other relevant factors. Consequently, a completely static scene will produce no data; however, any changes to that scene will be recorded. Neuromorphic cameras operate on sparse data, since they only record changes to individual pixels rather than to the entire frame – this results in images with high accuracy, low latency, and low power requirements.

Neuromorphic cameras are especially useful for high-speed navigation, motion tracking, and other motion-related applications under extreme lighting conditions. However, they need a strong ecosystem around them to enable precise clocking and timestamps, calibration, asynchronous processing, and data pipelines that handle differentiated data formats relative to the rest of the vision stack. Despite the technology and the surrounding ecosystem not being as mature yet, event-based cameras are growing in adoption, especially as a dynamic layer (sitting beside the semantic layer of RGB cameras) in physical AI applications such as autonomous vehicles, industrial robotics, moving target detection in cluttered defense environments, and other use cases that demand very high accuracy in real-time.

Hyperspectral and multispectral: The previous camera types have all focused on imaging based on shape, size, color, texture, and motion – visualizing the surface or exterior of a scene. Hyperspectral and multispectral cameras add another dimension by analyzing the material composition and chemical identity of elements within a scene. Both camera types work in a similar way – they split light into narrow, continuous spectral bands for each pixel and use this to identify different materials based on their spectral signatures. Although similar, these differ slightly primarily based on the volume of data and the spectral resolution. Hyperspectral imaging uses hundreds of narrow, contiguous bands, allowing very precise material identification based on detailed spectral signatures; multispectral imaging captures a smaller set of broad, discrete bands, which offer lower spectral resolution but faster data processing and easier data interpretation. Consequently, hyperspectral cameras are used more commonly in use cases that require precise spectral identification, but where the scene is not moving rapidly or producing large amounts of new data – for example, pharmaceutical inspection, mining, and recycling processes where material sorting is paramount. Multispectral cameras are used where the requirement for precise spectral analysis is lower, but spatial analysis (i.e., “where is it?”) or faster data processing is a greater need – for example, land-use mapping or vegetation monitoring.

Both hyperspectral and multispectral imaging systems are newer parts of the camera ecosystem, and tend to be more expensive as well as complex to set up, calibrate, and integrate into an overall vision stack – as a result, they are used only when needed as a materials intelligence layer on top of the other layers.

Short-wave infrared (SWIR) cameras can also be used for a similar purpose – many materials that look similar in visible light can be identified through spectral signatures based on a portion of the infrared spectrum. SWIR is the bridge between traditional semantic layers and materials intelligence layers – it can be used for both material identification and to improve visibility through haze and other obscurants (similar to other infrared cameras). However, these tend to be very expensive and are tougher to integrate into a broader vision stack – and higher-performance systems often need additional cooling infrastructure, noise management, and complex maintenance. 

Seeing everything: The unified vision stack

As we have seen, there are various cameras that can be applied based on the use case and requirements.

The core of any vision system remains the RGB camera, acting as the central semantic layer. This is often augmented with depth cameras to understand 3D geometry, as well as thermal cameras to support visibility in poor conditions, and add a layer of redundancy. Event cameras are the next most common, especially for adding a dynamic layer to objects in motion – for example, in fast-moving robotics or autonomous vehicles. Spectral imaging is used sparingly, only in cases where materials intelligence is a necessary requirement.

Building the ideal hardware stack for vision requires detailed analysis and planning, based on the use cases. Several questions need to be answered –

• Do I need identity, geometry, materials intelligence, spatial and motion intelligence, or a combination of these?
• What are the light conditions likely to be, and how will they change?
• What are the constraints around power, cooling, and interference with the scene (i.e., active or passive operation)?

A typical vision stack in robotics, defense, security, manufacturing, healthcare, or any other complex use case is multimodal by default, incorporating the various camera modalities above. Often, these are also paired with other non-vision sensors, such as vibration sensors (for industrial machinery) or radar, which is often an essential addition to the stack for detecting motion and presence independent of ambient conditions such as light, dust, or other obstructions. Physical AI companies can build meaningful and defensible moats by building or integrating unified and efficient hardware stacks that capture and combine data accurately and efficiently for subsequent decision-making.

In addition, cameras used in a physical AI stack need to be considered not just for resolution or megapixels, but also for other factors such as calibration stability, dynamic range, timestamp accuracy, and synchronization across the stack. The ability to integrate all these cameras using a unified software and data stack that can make sense of the captured scenes and provide true intelligence is critical – we will explore the ideal software and stack in the next article.