Real-Time Radar-Based Gesture Detection and Recognition Built in an Edge-Computing Platform

IEEE Sensors Journal, vol. 20, pp. 10706–10716

Abstract

In this paper, a real-time signal processing framework based on a 60 GHz frequency-modulated continuous wave (FMCW) radar system to recognize gestures is proposed. In order to improve the robustness of the radar-based gesture recognition system, the proposed framework extracts a comprehensive hand profile, including range, Doppler, azimuth and elevation, over multiple measurement-cycles and encodes them into a feature cube. Rather than feeding the range-Doppler spectrum sequence into a deep convolutional neural network (CNN) connected with recurrent neural networks, the proposed framework takes the aforementioned feature cube as input of a shallow CNN for gesture recognition to reduce the computational complexity. In addition, we develop a hand activity detection (HAD) algorithm to automatize the detection of gestures in real-time case. The proposed HAD can capture the time-stamp at which a gesture finishes and feeds the hand profile of all the relevant measurement-cycles before this time-stamp into the CNN with low latency. Since the proposed framework is able to detect and classify gestures at limited computational cost, it could be deployed in an edge-computing platform for real-time applications, whose performance is notedly inferior to a state-of-the-art personal computer. The experimental results show that the proposed framework has the capability of classifying 12 gestures in real-time with a high${F}_{\sf 1}$-score.

Authors 6

1. Yuliang Sun

   Hella (Germany) · Ruhr University Bochum

   Affiliation as printed

   HELLA GmbH & Co. KGaA, Lippstadt, Germany

   Institute for Integrated Systems, Ruhr-University Bochum, Bochum, Germany

   HELLA GmbH & Co. KGaA, Lippstadt, Germany; Institute for Integrated Systems, Ruhr-University Bochum, Bochum, Germany

2. Tai Fei

   Hella (Germany)

   Affiliation as printed

   HELLA GmbH & Co. KGaA, Lippstadt, Germany

3. Xibo Li Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University, Aachen, Germany

4. Alexander Warnecke

   Hella (Germany)

   Affiliation as printed

   HELLA GmbH & Co. KGaA, Lippstadt, Germany

5. Ernst Warsitz

   Hella (Germany)

   Affiliation as printed

   HELLA GmbH & Co. KGaA, Lippstadt, Germany

6. Nils Pohl

   Ruhr University Bochum

   Affiliation as printed

   Institute for Integrated Systems, Ruhr-University Bochum, Bochum, Germany