In-Pixel Intelligent Processing: Advances in Computer Vision

Sunday - June 11 2023, 08:00 UTC - 1 year ago

Computer vision is an AI field that enables computers to acquire, process, and analyze digital images, and make recommendations based on that analysis. Researchers at USC Viterbi's Information Sciences Institute and the Ming Hsieh Department of Electrical and Computer Engineering are looking to make advances in computer vision. They have developed In-Pixel Intelligent Processing (IP2), a low-cost, low-power technology that eliminates the need for backend processing. IP2 is suitable for both cars and drones, and can improve object detection accuracy.

You're in an autonomous car when a rabbit suddenly hops onto the road in front of you. Here's what typically happens: the car's sensors capture images of the rabbit; those images are sent to a computer where they are processed and used to make a decision; that decision is sent to the car's controls, which are adjusted to safely avoid the rabbit. Crisis averted.

This is just one example of computer vision—a field of AI that enables computers to acquire, process, and analyze digital images, and make recommendations or decisions based on that analysis.

The computer vision market is growing rapidly, and includes everything from DoD drone surveillance, to commercially available smart glasses, to rabbit-avoiding autonomous vehicle systems. Because of this, there is increased interest in improving the technology. Researchers at USC Viterbi's Information Sciences Institute (ISI) and the Ming Hsieh Department of Electrical and Computer Engineering (ECE) have recently completed Phases 1 and 2 of a DARPA (Defense Advanced Research Projects Agency) project looking to make advances in computer vision.

Two jobs spread over two separate platforms .

In the rabbit-in-the-road scenario above, on the “front end” is the vision sensing (where the car's sensors capture the rabbit's image) and on the “back end” is the vision processing (where the data is analyzed). These are conducted on different platforms, which are traditionally physically separated.

Ajey Jacob, Director of Advanced Electronics at ISI explains the effect of this: “In applications requiring large amounts of data to be sent from the image sensor to the backend processors, physically separated systems and hardware lead to bottlenecks in throughput, bandwidth and energy efficiency.” .

In order to avoid that bottleneck, some researchers approach the problem from a proximity standpoint—studying how to bring the backend processing closer to the frontend image collection. Jacob explained this methodology. “You can bring that processing onto a CPU [computer] and place the CPU closer to the sensor. The sensor is going to collect the information and send it to the computer. If we assume this is for a car, it's fine. I can have a CPU in the car to do the processing. However, let's assume I have a drone. I cannot take this computer inside the drone because the CPU is huge. Plus, I'll need to make sure that the drone has an Internet connection and a battery large enough for this data package to be sent.” .

So the ISI/ECE team took another approach, and looked at reducing or eliminating the backend processing altogether. Jacob states, “What we said is, let's do the computation on the pixel itself. So you don't need the computer. You don't need to create another processing unit. You do the processing locally, on the chip.” .

Front-end processing inside a pixel .

Processing on the image sensor chip for AI applications is known as in-pixel intelligent processing (IP2). With IP2, the processing occurs right under the data on the pixel itself, and only relevant information is extracted. This is possible thanks to advances in computer microchips, specifically CMOS (complementary metal–oxid–semiconductor) image sensors, which enable digital reading of analog signals, group analogue/digital converters, and low-powered, digital logarithmic computers.

Using IP2 for computer vision has several advantages: it is low-power, cost-effective and can support relatively large networks that can process large amounts of data quickly.

Jacob explains, “For example, with object detection, where you have a car and you need to know what is around the car, with IP2, you can quickly go through the entire image and only detect objects, thereby eliminating the need for a computer.” .