Toward Efficient Eye Tracking in AR/VR Devices: A Near-Eye DVS-Based Processor for Real-Time Gaze Estimation
IEEE Transactions on Circuits and Systems I Regular Papers, vol. 72, pp. 5748–5760
Abstract
This paper presents an efficient near-eye dynamic vision sensor (DVS)-based processor for real-time eye tracking in augmented reality/virtual reality (AR/VR) devices. The processor takes advantage of the sparse event data with fine time resolution from the DVS, addressing the need for high frame-rate, low-power, and accurate eye tracking on wearable devices with extended battery life. Exploiting the inherent sparsity of event data, we propose an event-density-based region of interest (ROI) determination method that operates directly on event stream, which requires$47\times $fewer operations than the traditional methods, effectively overcoming the latency problem caused by the heavy computational loads. To eliminate the issue of decreasing accuracy at the edges of the field of view (FoV), we customized and fine-tuned a neural network for gaze estimation, ensuring uniformly distributed sub-degree accuracy. An estimator with a streamlined output mapping strategy and an adaptive window-sliding convolution scheme is implemented for gaze estimation acceleration. The processor is designed and fabricated in UMC 40-nm LP technology with a core area of 2.52 mm2 and performs end-to-end eye tracking exclusively with the raw event stream from DVS, achieving an average accuracy of 0.91° within a$96^{\circ } \times 64^{\circ }$FoV. Operating at 200 MHz, it achieves a dynamic frame rate of up to 1.2 kHz and requires only$12.7~\mu $J of energy per gaze estimation. By integrating the DVS, the processor enables real-time, low-power, and accurate eye tracking, enhancing the immersive experience on AR/VR devices and offering intuitive and seamless interactions.
Authors 7
-
Affiliation as printed
State Key Laboratory of Integrated Chips and Systems, School of Information Science and Technology, Fudan University, Shanghai, China
School of Information Science and Technology, State Key Laboratory of Integrated Chips and Systems, Fudan University, Shanghai, China
-
Affiliation as printed
State Key Laboratory of Integrated Chips and Systems, School of Information Science and Technology, Fudan University, Shanghai, China
School of Information Science and Technology, State Key Laboratory of Integrated Chips and Systems, Fudan University, Shanghai, China
-
Affiliation as printed
State Key Laboratory of Integrated Chips and Systems, School of Information Science and Technology, Fudan University, Shanghai, China
School of Information Science and Technology, State Key Laboratory of Integrated Chips and Systems, Fudan University, Shanghai, China
-
Affiliation as printed
State Key Laboratory of Integrated Chips and Systems, School of Information Science and Technology, Fudan University, Shanghai, China
School of Information Science and Technology, State Key Laboratory of Integrated Chips and Systems, Fudan University, Shanghai, China
-
Affiliation as printed
Leiden Institute of Advanced Computer Science, Leiden University, Leiden, The Netherlands
-
Affiliation as printed
State Key Laboratory of Integrated Chips and Systems, School of Information Science and Technology, Fudan University, Shanghai, China
School of Information Science and Technology, State Key Laboratory of Integrated Chips and Systems, Fudan University, Shanghai, China
-
Affiliation as printed
State Key Laboratory of Integrated Chips and Systems, School of Information Science and Technology, Fudan University, Shanghai, China
School of Information Science and Technology, State Key Laboratory of Integrated Chips and Systems, Fudan University, Shanghai, China
Cited by 12 stored of 12
12 results
No patents citing this paper on Lens.org (checked 2026-10-11).