A new generation of sensors inspired by the human brain

Researchers have made strides in replicating the human brain's efficiency to enhance sensor technology and data processing in central computers. Compared to even the most powerful computers, the human brain outperforms them in terms of efficiency. With an average volume of approximately 1,260cm3, the human brain consumes only about 12 watts of power.

The remarkable capability of the human brain allows individuals to learn and recognize a vast number of faces in a short span of time. It can identify a familiar face instantly, irrespective of the expression. Furthermore, people can glance at a picture and identify objects belonging to countless categories. In contrast, let's consider the most powerful supercomputer. Frontier, located at Oak Ridge National Laboratory, Tennessee, USA. Spanning an area of 372m2 and consumes a staggering 40 million watts of power at its peak. Although Frontier processes massive amounts of data to train artificial intelligence (AI) models for recognizing numerous human faces, these models struggle when faced with unusual expressions.

The training process itself also requires a significant amount of energy, and while the resultant models can run on smaller computers, they still consume considerable power. Moreover, the models generated by the supercomputer are limited to recognizing objects from only a few hundred categories.

Moving beyond the Von Neumann architecture

Scientists have made certain discoveries about how the brain functions, such as the communication between neurons through spikes or thresholds of accumulated potential. Researchers have delved into the human cortex using brain probes to observe and record neuronal activity. These measurements have revealed that a typical neuron only spikes a few times per second, indicating sparse activation. While some basic principles of brain function are mostly understood, the specific mechanisms of neural computation, learning, and memory formation remain a mystery.

Nonetheless, the principles being investigated today will likely contribute to developing a new generation of chips that may replace current computer processing units (CPUs) and graphics processing units (GPUs) in the next decade. This transformation will involve shifting from the traditional von Neumann architecture, where processing and data storage are separate, and information is transferred through a shared bus.

Future computer designs are expected to incorporate concepts inspired by the brain, such as collocating processing and storage, mimicking the brain's structure. This approach aims to make computers faster and more power-efficient. The field of study focusing on these principles is known as neuromorphic computing, and significant research in this area is being conducted at the Interuniversity Microelectronics Centre (Imec) in Belgium.

Computations we don't understand

It is true that while spiking behavior is considered fundamental in the computation of biological neurons, there are likely more complex processes occurring at deeper levels, potentially even involving quantum effects. Between the quantum level and the high-level behavioral model of neurons, intermediate functions such as ion channels and dendritic calculations contribute to the brain's complexity. Our current understanding of the brain is limited, and there is still much to discover.

However, researchers in Belgium have made significant progress in replicating certain aspects of brain function using existing technology. Even though we may not fully understand the intricate workings of the brain, the ability to mimic some of its functions is already a significant innovation for humanity. These breakthroughs pave the way for further exploration and potential advancements in various study and technological development fields.

Indeed, various techniques and optimizations exhibit partial neuromorphic characteristics and have already been implemented in industrial settings. For instance, designers of graphics processing units (GPUs) have incorporated insights from the human brain into their designs, and computer architects have been addressing bottlenecks by utilizing multilayer memory stacks. The concept of massive parallelism, inspired by biological systems, is also prevalent in areas like deep learning in computer science.

Reinventing sensors

Researchers working on neuromorphic computing face significant challenges in making substantial progress within the computing field, as traditional architectures have already gained substantial momentum. Consequently, researchers have shifted their focus to sensors rather than attempting to disrupt the already innovative computing field. They are exploring methods to sparsify data and leverage this sparsity to accelerate sensor processing while concurrently reducing energy consumption. This shift allows for advancements in sensor technologies that align with bio-inspired principles and can contribute to enhanced performance and energy efficiency.

The researchers at Imec specifically concentrate on temporal sensors, encompassing various types such as audio, radar, and lidar sensors. One notable area of focus is event-based vision, which introduces a novel approach to vision sensing. Unlike traditional frame-based vision, event-based vision sensors operate based on the principles of the human retina. Each pixel in the sensor independently sends a signal when it detects a significant change in the received light intensity.

Drawing inspiration from these concepts, researchers have developed new algorithms and hardware specifically designed to support spiking neural networks. Their current emphasis is on demonstrating the potential of integrating these networks onto sensors in terms of low power consumption and low latency. By showcasing the efficiency and effectiveness of these integrated systems, they aim to highlight the advantages of leveraging spiking neural networks for various sensor applications.

Using circuits to emulate the spiking neurons

In the field of neuromorphic computing, researchers at Imec have successfully emulated the behavior of biological spiking neurons using digital circuits on a chip. These circuits mimic the characteristics of neurons in the brain. The spiking neurons accumulate inputs, lose a bit of voltage over time, and emit a spike when the membrane potential surpasses a certain threshold.

This mode of operation offers energy efficiency since no events or computations occur in the neural network until data changes. The sparse nature of spikes in the network contributes to low power consumption, as constant computing is unnecessary. Recurrent spiking neural networks exhibit memory capabilities as spikes reverberate through the network, enabling the recognition of temporal patterns similar to the brain's functioning.

In the case of sensors utilizing spiking neural network technology, data transmission occurs through tuples that include information such as the X and Y coordinates of the spiking pixel, the polarity of the spike, and the time it occurs. Sensors employ filtering mechanisms based on scene dynamics to handle the surge in transmission caused by simultaneous changes across multiple areas. This approach mimics the filtering algorithm observed in the human retina and can be seen as a form of edge AI.

Training spiking neural networks efficiently has been a significant obstacle in the past. However, researchers have made progress by developing software algorithms that demonstrate the performance of spiking neural networks based on specific configurations of neurons and connections. Subsequently, they implemented the hardware accordingly. This unconventional software and algorithm breakthrough, coupled with the use of standard CMOS technology, allows for quick industrialization of the developed technology.

The next direction is sensor fusion

Researchers are directing their efforts towards sensor fusion, a highly relevant topic in various industries such as automotive, robotics, and drones. Combining multiple sensory modalities aims to achieve high-fidelity 3D perception with low power consumption and minimal latency using spiking neural networks. The objective is to develop a dedicated chip for sensor fusion this year.

The goal is to create a comprehensive and coherent representation of the world by merging multiple sensor streams. Similar to the functioning of the brain, the researchers aim for an inherently fused representation where they do not need to distinguish between inputs from cameras and radar, for instance.

Exciting demonstrations are anticipated in the automotive industry, as well as in robotics and drones across various sectors, showcasing this technology's performance and low latency. The researchers will initially focus on addressing challenging scenarios in automotive and robotics perception that are currently impractical due to high latency or power consumption.

Event-based cameras, offering high dynamic range and temporal resolution, are expected to gain traction in the market, along with sensor fusion. Sensor fusion may involve a single module incorporating cameras, radar antennas, lidar, and data fusion on the sensor itself using spiking neural networks.

While the broader public may be unaware of the underlying technology, this could change when the first event-based camera is integrated into a smartphone. This integration could enhance human-machine interfaces, allowing for more natural interactions. For example, an event-based camera could recognize hand gestures as an interface, only initiating processing when significant activity is detected, thus leveraging the camera's intrinsic wake-up mechanism. Neuromorphic sensing holds the potential to revolutionize human-machine interfaces, enabling more intuitive and seamless interactions between humans and machines.