A Lightweight Deep Learning Approach for IMU-Based Human Activity Recognition: From Sensor Data to Action Classification

1. What Is Human Activity Recognition and Why IMU?

Human Activity Recognition (HAR) is a technology that determines a person’s various postures and daily activities through a series of observations and surrounding environmental data. With the arrival of the 5G era and the rapid development of IoT technology, HAR has been widely adopted in daily life and has become one of the core functions of intelligent products. HAR possesses enormous development potential and broad application scenarios, including human-computer interaction, sports activity monitoring, smart homes, healthcare, and crime surveillance.

Existing HAR methods can be classified into three categories: (1) Computer Vision (CV)-based HAR, which collects human motion data through video and images; (2) Channel State Information (CSI)-based HAR, which identifies human activities through the physical layer of WiFi signals; (3) Sensor-based HAR, such as IMU-based motion capture, which processes and analyzes multi-dimensional motion data collected by single or multiple sensors.

CV-based HAR requires pre-installing high-definition cameras, and imaging quality is affected by temperature, visibility, and other environmental conditions. CSI-based HAR requires being within WiFi signal coverage, limiting its application scenarios. With the development of IoT technology, more researchers have turned to HAR methods based on Inertial Measurement Unit (IMU) sensor data. IMU sensors include accelerometers, gyroscopes, and magnetometers, and are widely used in smartphones, automotive electronics, aerospace, and military fields. Compared with other HAR methods, IMU-based HAR offers advantages including low cost, low power consumption, and data collection unconstrained by environmental conditions. With the proliferation of mobile networks and smartphones, HAR algorithms based on smartphone built-in IMU sensors have drawn increasing attention, and the growing computing power of smartphone chips provides a favorable environment for running these algorithms.

2. From Machine Learning to Deep Learning for HAR

In the early days, HAR technology used Machine Learning (ML) models that extracted time-domain and frequency-domain features from raw IMU sensor data. Researchers fed IMU sensor data into ML models to learn a classifier capable of categorizing activities.

Zhang H et al. proposed a Support Vector Regression (SVR) model for accurately estimating basic gait parameters — stride length, speed, and foot clearance — from custom-designed instrumented insoles, achieving excellent intraclass correlation coefficients across all analyzed gait parameters. Maurer U et al. developed eWatch, an online activity recognition system embedding sensors and a microcontroller into a wrist-wearable device, using decision trees and time-domain feature extraction with execution times below 0.3 ms for real-time inference. Zhu C proposed a walking activity recognition system using Hidden Markov Models (HMM), employing coarse-grained classification to determine activity type followed by fine-grained classification to further distinguish activities. Xia K et al. developed a racket-sport recognition wristband system using a multi-layer hybrid clustering model, applying K-means clustering for feature extraction and DBSCAN clustering to identify feature centers of different motions without requiring data calibration. Jain A et al. proposed a smartphone built-in sensor HAR method extracting features via Fourier transforms and histogram of oriented gradients, then classifying activities using K-Nearest Neighbors and Support Vector Machines.

Although ML-based algorithms could already provide high accuracy, the features required in machine learning mostly needed domain experts to identify and hand-code, and the learned features were only low-dimensional, resulting in weak generalization capability.

3. The Challenge: Deep Learning Models Are Too Heavy for Deployment

With the arrival of the big data era, the volume of training data has grown enormously, and traditional machine learning algorithms can no longer satisfy researchers’ pursuit of performance. Consequently, more researchers have turned to deep learning-based HAR algorithms. Compared with machine learning, neural network models possess stronger generalization capability and do not require domain experts to hand-craft features — they can automatically extract high-dimensional features from data, thereby further improving model accuracy.

Deep Neural Networks (DNN) represent the simplest form, consisting of only a few fully connected layers. Bashar S K et al. proposed a DNN-based HAR algorithm that first uses neighborhood component analysis to select important features from time-domain and frequency-domain parameters, then models a dense neural network with four hidden layers to classify input features. Convolutional Neural Networks (CNN) are commonly used in classification tasks. Xu W et al. built a CNN-based model using raw tri-axial accelerometer data from smartphones directly as input, without requiring any complex preprocessing. Long Short-Term Memory (LSTM) networks possess the capability to mine information from long-distance temporal data, making them well-suited for continuous sensor-collected data. Uddin M Z et al. proposed a deep neural structured learning model based on time-series information, using kernel-based discriminant analysis to better observe feature clusters from different activity classes, then applying LSTM-based structured learning for activity modeling. Li H et al. proposed a bidirectional LSTM framework for multi-modal sensor fusion, achieving soft feature fusion between wearable sensor and radar data. Hamad R A et al. proposed a HAR method combining dilated causal convolutions with a multi-head self-attention mechanism, entirely omitting recurrent structures while maintaining step-wise sequence order for efficient computation.

Beyond supervised learning, semi-supervised and federated learning approaches have also been introduced to HAR. Yao L et al. introduced semi-supervised CNN for HAR, where a CNN-based encoder-decoder and convolutional ladder network learn better high-level features, using L2 regularization to resist outliers in noisy sensor data and reducing labeled data requirements by over 90%. Yu H et al. proposed FedHAR, a personalized federated HAR framework that performs distributed learning to keep training data local and protect user privacy. Li C et al. proposed Meta-HAR based on meta-learning, treating each user’s HAR problem as a different task and training a shared embedding network through a model-agnostic meta-learning framework, enabling the embedding network to generalize to any individual user.

However, these neural network models possess high temporal and spatial complexity, typically requiring substantial computational resources and training time. Although these networks have achieved remarkable results in HAR, their complexity makes deployment on devices such as smartphones challenging, limiting their practical usability.

4. Res-MLP: A Lightweight Deep Learning Architecture

Numerous deep learning neural network architectures exist today, including CNN, RNN, and LSTM. The perceptron, proposed by Rosenblatt in 1957, is the simplest neural network — a two-layer network with only an input layer and an output layer, without any hidden layers. In 2008, Nazzal J M proposed the Multi-Layer Perceptron (MLP). Unlike the traditional perceptron, MLP adds hidden layers between the input and output layers. All layers in an MLP are fully connected — each neuron in one layer connects to every neuron in the next layer — enabling it to handle non-linear tasks as a non-linear classifier.

The input layer feeds data into the MLP network. Before entering the MLP, data requires preprocessing — typically normalization, which compresses input data into the [0, 1] range and accelerates network convergence. A Batch Normalization (BN) layer is commonly used before each training layer to normalize data, forcing it toward a normal distribution with a mean of 0 and variance of 1. This not only keeps data distributions consistent but also avoids vanishing gradients and improves training efficiency.

In recent years, as neural networks have grown increasingly complex and computationally demanding, industry has begun pursuing more lightweight neural networks to meet practical deployment requirements. MLP, as the most fundamental network with the smallest computational footprint, has re-entered researchers’ focus. They discovered that after certain improvements, MLP can outperform more complex networks like CNN and LSTM while remaining significantly more lightweight. After a series of refinements, MLP is now fully capable of handling various classification tasks.

4.1 Residual Connections: Solving the Depth Problem

Residual Network (ResNet), proposed by He Kaiming et al., was designed to solve the problems of vanishing gradients and network degradation. When networks become very deep, parameter initialization approaches zero, causing gradients to vanish in shallow layers during backpropagation. Additionally, network degradation occurs: as the training loss decreases and saturates, adding more layers paradoxically increases the training loss. This is not caused by overfitting, but by redundant network layers learning non-identity mapping parameters. The solution is residual networks — the residual between two consecutive layer outputs serves as input to the next layer, ensuring information from shallow layers is not corrupted by deeper layers. A residual module consists of two parts: the identity mapping (skip connection) and the residual mapping, expressed as x_{n+1} = x_n + F(x_n, W_n).

4.2 GELU Activation: A Smarter Nonlinearity

In residual modules, the ReLU function is commonly used as the activation function. However, ReLU has a notable defect: when the input is negative, its output is directly zero, preventing those neurons from receiving effective training. In 2016, Hendrycks D et al. proposed the GELU (Gaussian Error Linear Unit) activation function. GELU offers three key advantages: (1) As a non-monotonic activation function, GELU effectively preserves small negative values, maintaining gradient flow while overcoming both gradient explosion and vanishing gradient problems. (2) GELU’s output range is [-0.21, +∞] — the upper bound avoids gradient saturation that sharply drops training speed, while the absence of a lower bound provides strong regularization. (3) GELU possesses excellent smoothness, enabling better generalization.

4.3 The Complete Res-MLP Network

To address the high complexity of deep learning models currently used for IMU sensor data, a lightweight model was designed based on MLP and ResNet. Because the model does not use convolution or self-attention mechanisms and is composed solely of simple fully connected layers, its complexity is significantly reduced.

Before data enters the model, it passes through a Batch Normalization (BN) layer to normalize the data and ensure consistent distribution, thereby avoiding vanishing gradients and improving training efficiency. The output tensor then enters the first fully connected layer, which uses GELU as the activation function. To prevent overfitting, a Dropout layer is placed after the fully connected layer, randomly deactivating a proportion of neurons so that each mini-batch effectively trains a different network.

The algorithm framework follows three stages: First, signal data samples are collected from the built-in IMU sensors. Second, the collected data undergoes preprocessing — including window segmentation, normalization, and feature extraction — to produce a dataset suitable for training. Finally, the prepared dataset is fed into the lightweight Res-MLP neural network for training. Once training is complete, the model can be used to recognize human activities.

5. Experimental Validation on UCI HAR Dataset

The experiments use the UCI HAR dataset, proposed by the University of Genoa team. The dataset was collected from 30 volunteers aged 19-48 wearing smartphones at the waist. It contains six activities: walking, walking upstairs, walking downstairs, sitting, standing, and laying. The smartphone’s built-in gyroscope and accelerometer captured tri-axial linear acceleration and tri-axial angular velocity data at a 50 Hz sampling rate. Data was divided into fixed windows of 2.56 seconds (128 data points) with 50% overlap. Gyroscope and accelerometer data together provide 9 channels. A total of 10,299 samples were collected, with 561 feature vectors designed from time-domain and frequency-domain variables.

The dataset was split 7:3 into training (7,209 samples) and test sets (3,090 samples). Each fully connected layer used L2 regularization (coefficient 0.001) with the Adam optimizer. The learning rate started at 0.01 with a decay factor of 0.1, enabling rapid convergence early in training and slower gradient changes later for better final performance.

5.1 Performance vs. Complexity Trade-off

By comparing performance across activation functions, GELU achieved 98.23% accuracy versus Linear (97.67%), Tanh (97.81%), ReLU (97.78%), and Leaky ReLU (97.85%), confirming GELU’s significant advantage. After 300 training epochs, validation accuracy and loss closely matched training values, confirming no overfitting occurred.

Compared with four other HAR methods using smartphone built-in IMU sensors — C4M4BL, BLSTM, CNN+LSTM, and DNN — Res-MLP achieved the lowest values in model parameters (spatial complexity), FLOPs (computational complexity), training time (300 epochs), and test time. This demonstrates three key advantages: (1) More lightweight — without convolution operations, computational complexity is lower, requiring less processing power. (2) Shorter training time — as dataset size grows, the 84× training time reduction versus C4M4BL (289s vs 14,054s) is transformative for iterative development. (3) Faster inference — with only 0.45s test time, the model supports real-time applications.

Res-MLP achieved at least 0.36% higher average accuracy across six activities than other methods, and its precision, recall, and F-score were also the highest. DNN — composed solely of fully connected layers like Res-MLP but without residual connections — showed the closest recognition performance, confirming the critical contribution of the residual architecture. In summary, Res-MLP not only has the most lightweight model but also achieves the best classification performance, demonstrating an excellent trade-off between performance and cost for real-world deployment.

5.2 Confusion Matrix Analysis

C4M4BL, BLSTM, and CNN+LSTM showed varying confusion across all six activities, while DNN and Res-MLP achieved 100% accuracy on walking, walking upstairs, walking downstairs, and laying — with zero confusion for these four activities due to their highly distinctive signal features. However, sitting and standing remain the most confusable activities, as their tri-axial acceleration and angular velocity patterns are very similar.

6. Out-of-Distribution Generalization: The Real-World Reality

The previous experiments assumed training and test data are identically and independently distributed (i.i.d.), meaning P_train(X,Y) = P_test(X,Y). However, in real-world scenarios, it is typically impossible to collect training data from all potential users — the test distribution is unknown. Test data is generated in the future, and each test sample represents a new user, requiring the model to possess out-of-distribution (OOD) generalization capability.

The UCI HAR dataset was collected from 30 volunteers, each with independently numbered data records. For OOD evaluation, the dataset was re-partitioned by volunteer ID to create four OOD datasets: each training set randomly selected 21 volunteers’ samples, and the test set used the remaining 9 volunteers’ samples. This ensures that training and test data come from different distributions (P_train ≠ P_test), specifically testing the model’s ability to predict activities for unseen users.

Under the various OOD dataset partitions, model classification performance dropped to varying degrees — the largest drop being nearly 6%, from 98.23% on the i.i.d. dataset to 92.48% on the worst-performing OOD dataset. Sitting and standing remained the most confusable activity pair across all OOD tests. This indicates that the Res-MLP algorithm requires further improvement in generalization capability, and that when encountering new users in real-world scenarios, the model’s activity recognition is not yet sufficiently accurate.

7. Summary

The Res-MLP algorithm demonstrates that a lightweight neural network — combining MLP with residual connections and GELU activation — can achieve state-of-the-art HAR accuracy while requiring only a fraction of the parameters and computation of more complex models such as CNN+LSTM and DeepConvLSTM. With just 97,542 parameters and 327,224 FLOPs, Res-MLP achieves 98.23% accuracy on the UCI HAR dataset, outperforming heavier architectures with training times up to 48× shorter while being suitable for deployment on resource-constrained edge devices such as smartphones. The OOD generalization experiments reveal that when facing unseen users, accuracy can drop by up to 6%, indicating that further techniques such as deep metric learning and ensemble learning are needed for robust real-world deployment across diverse user populations.