[WACV 2024] UNSPAT: Uncertainty-Guided SpatioTemporal Transformer for 3D Human Pose and Shape Estimation on Videos

3D human pose and shape estimation is the task of reconstructing a human mesh from images and video. Research on recognizing and inferring human posture and behavior from these inputs and videos has been gaining traction because of its applicability in a variety of industries, including computer graphics and healthcare.

In this study, we propose an efficient framework for 3D human pose and shape estimation in video^[5]. Unlike previous video-based methods that use global average pooling to compress spatial information and then consider temporal relationships, we solved the excessive complexity of spatio-temporal attention through two modules: the 'Spatial Alignment Module' and 'Space2Batch.' We also utilized the uncertainty-guided attention re-weighting module to improve the robustness of the model in challenging environments such as occlusions and crowded backgrounds. Using these techniques, we achieved SOTA on widely used benchmark datasets and presented detailed results at WACV 2024, the leading conference in the computer vision.

1. Introduction

Since the introduction of  SMPL^[1], a parametric representation of 3D human pose and shape, the field of 3D human pose and shape estimation has made significant performance progress in the form of estimating SMPL parameters.

Figure 1. Visualization of introduction to SMPL in 3D human pose and estimation^[1]

However, single-frame-based 3D human pose and shape estimation is vulnerable to situations such as motion blur and occlusion, which can lead to unstable estimation performance along the time axis when estimating a sequence of images. To solve this problem, various video-based approaches^{[2, 3, 4]} have been proposed. Recent video-based studies extract features from each frame individually and then combine them to form temporally-aware features for a direct extension to the time axis. However, while these extensions contributed to improving temporal errors, they significantly increased reconstruction errors. This is because, to address the excessive complexity of spatio-temporal attention, global average pooling was used to compress the spatial information and then model the temporal relationship.

Figure 2. Comparison image with and without global average pooling applied

This study aimed to build a 3D human pose and shape estimation framework that considers all spatio-temporal dimensions without excessively increasing complexity. To this end, we proposed 1) a 'Spatial Alignment Module (SAM)' that spatially realigns adjacent frames, and 2) a 'Space2Batch' that decomposes spatial and temporal relationships to significantly reduce the complexity of spatio-temporal attention. We also proposed an uncertainty-guided attention re-weighting module to improve model robustness in difficult environments such as motion blur and occlusion.

2. Methods

This study aims to estimate the 3D human pose and shape of the center frame using a sequence of continuous images. To represent the pose and shape, we use SMPL, a parametric human body model, to estimate the shape parameters and pose parameters.

Figure 3. Schematic diagram of the proposed model^[5]

Spatial Feature Extraction and Spatial Alignment Module

First, the bounding box of the person is extracted from the image, and then the image features within the box are extracted. In contrast to previous video-based methods that use a global average pooled vector, pooling on features is not done to avoid losing spatial information. Rather, the spatial dimensionality is preserved. Since these features contain spatial dimensions, additional processing is required.

Specifically, because the bounding boxes of people are not spatially aligned, simply combining these features leads to poor performance, which often occurs when the camera or people in the scene are moving rapidly. To solve this problem, we proposed the Spatial Alignment Module (SAM), which spatially aligns each adjacent feature to a central feature. The SAM takes the adjacent features and the center feature as input to estimate the affine transformation matrix. This matrix consists of scale and translation parameters for the x-axis and y-axis. Using the estimated affine transformation matrix, adjacent features are warped to spatially align with the center feature.

Space2Batch

When a sequence of features spatially aligned by SAM is used to model the spatio-temporal relationship between frames using the transformer architecture, the complexity of $O\left(d{w}^2{h}^2{T}^2\right)$occurs because attention complexity increases quadratically with respect to its input dimension. To deal with the excessive complexity of calculating attention for each point on the spatio-temporal axis, we proposed Space2Batch. As the spatial positions of features within the time axis have been aligned by SAM, we can decompose the temporal correlation from the spatial positions by treating the spatial dimension as a batch, thus only calculating attention between the same spatial positions. This results in significantly reducing the complexity from $O\left(d{w}^2{h}^2{T}^2\right)$ to $O\left(dwh{T}^2\right)$while achieving better performance than considering full spatial-temporal attention.

Uncertainty-Guided Attention Re-Weighting

When modeling spatio-temporal correlation, it is important to prevent errors in a specific frame from propagating to the entire sequence. This is especially true if some frames in the video are occluded or blurred, which can lead to incorrect estimations. Previous studies suffered from error propagation by simply modeling temporal relationships using transformers. To solve this, we proposed uncertainty-guided attention re-weighting. The core of this module is for the model to estimate uncertainty about its spatio-temporal position. During training, we intentionally create synthetic artifacts by replacing random spatiotemporal positions with those of other videos in the batch. We then train a small network to distinguish between these artifacts. When inferring, the estimated uncertainty map is used to re-weigh spatio-temporal attention.

This uncertainty-guided attention re-weighting serves to identify areas that need help from adjacent frames in estimating the current frame and prevents incorrect information from propagating to adjacent frames.

3. Experiment

Attention Visualization

Figure 4. Visualization of attention weight, uncertainty map, and re-weighted attention^[5]

Figure 4 above shows how the attention weight, uncertainty map, and uncertainty-guided attention re-weighting behave when a noise patch is added to the frames. In the second row of Figure 4, you can see that the uncertainty estimator correctly estimates a high uncertainty value for the occluded area. The estimated uncertainty map is also used to re-weight the attention map from the third row to the fourth row in the figure.

Comparison with State-of-the-Art Methods

To make a quantitative comparison with existing video-based methodologies, we measured reconstruction error (PA-MPJPE, MPJPE, MPVPE) and temporal error (Acceleration) on the 3DPW, MPI-INF-3DHP, and Human3.6M benchmark datasets.

Figure 5. Quantitative comparison with existing methodologies^[5]

As shown in the above Figure 5, LG AI Research’s model outperforms the previous methodologies for all benchmarks. In particular, we see a large performance gap in terms of reconstruction error because our methodology models temporal relationships without compressing spatial information.

Figure 6. Qualitative comparison with existing methodologies (Source : 3DPW Dataset)

The proposed methodology also demonstrates more robust estimation performance compared to other methodologies in severely occluded situations, as shown in Figure 5 above.

4. Conclusion

This research, presented at WACV 2024 by LG AI Research, suggests an efficient framework for video-based 3D human pose and shape estimation. The published paper also achieved the selection of Oral session at WACV 2024. Specifically, we spatially aligned input features to efficiently calculate spatio-temporal relationships and reweighted attention based on uncertainty to improve model robustness.

We also achieved SOTA performance on benchmark datasets widely used in the field. 3D human technology is expected to be applied in more and more industrial environments, such as AI agents, logistics, and manufacturing, and will provide direct benefits to users. LG AI Research will also continue researching efficient approaches for real-world applications of 3D human pose and shape research.

▶UNSPAT: Uncertainty-Guided SpatioTemporal Transformer for 3D Human Pose and Shape Estimation on Videos (Link)