Rethinking Matching-Based Few-Shot Action Recognition

Abstract

Few-shot action recognition, i.e. recognizing new action classes given only a few examples, benefits from incorporating temporal information. Prior work either encodes such information in the representation itself and learns classifiers at test time, or obtains frame-level features and performs pairwise temporal matching. We first evaluate a number of matching-based approaches using features from spatio-temporal backbones, a comparison missing from the literature, and show that the gap in performance between simple baselines and more complicated methods is significantly reduced. Inspired by this, we propose Chamfer++, a non-temporal matching function that achieves state-of-the-art results in few-shot action recognition. We show that, when starting from temporal features, our parameter-free and interpretable approach can outperform all other matching-based and classifier methods for one-shot action recognition on three common datasets without using temporal information in the matching stage.

Project page: https://jbertrand89.github.io/matching-based-fsar

A Appendix

In this appendix, we present more formally the matching functions used as baselines for our study in (Sect. A.1), as well as extra experiments that study the impact of different hyper-parameters and present results of different task setups (Sect. A.2).

1.1 A.1 Baseline Matching Functions

In this section, we describe the different matching functions used as our study’s baseline,  6. Some matching functions use temporal information by leveraging the absolute or relative position of the pairwise similarities $m_{ij}$. The matching functions that use temporal information are called temporal whereas the others are called non-temporal.

Fig. 6.

Matching functions on the temporal similarity matrix M . We show how each method estimates a scalar video-to-video similarity given the input pairwise similarity matrix. The functions can be classified as a) temporal or b) non-temporal whether they use the temporal position of the features or not.

Temporal Matching Functions. We provide a list of the temporal matching functions implemented in this study as baselines. Some of them were already introduced in prior work.

Diagonal (Diag) is used as a baseline in prior work [3]. It is given by $s(M)= \displaystyle \nicefrac {\sum _{ij} m_{ii}}{n}$. It assumes temporally aligned video pairs.

OTAM  [3] uses and extends Dynamic Time Warping [20] to find an alignment path on M over which similarities are averaged to produce the video-to-video similarity. A differentiable variant is used for training.

Flatten+FC (Linear) is a simple baseline we use to learn temporal matching by flattening M and feeding it to a Fully Connected (FC) layer without bias and with a single scalar output. Video-to-video similarity is therefore given by $s(M) = \sum _{ij} w_{ij} m_{ij}$, where $w_{ij}$ are learnable parameters which are $n^2$ in total.

Table 4. Impact of learning a feature projection on performance of matching-based methods. $^\dagger$ denotes hand-crafted matching methods, i.e. no training is performed for the cases where a feature projection is not learned.

ViSiL  [16] is an approach originally introduced for the task of video retrieval. We apply it to few-shot action recognition for the first time. A small Fully Convolutional Network (FCN) is applied on M. Its output is a filtered temporal similarity matrix, and the Chamfer similarity is applied on it. The filtering is performed according to the small temporal context captured by the small receptive field of this network.

Non-temporal Matching Functions. We provide a list of the non-temporal matching functions that were implemented in this study as baselines.

Mean is used as a baseline in prior work [3]. It is given by $s(M)= \displaystyle \nicefrac {\sum _{ij} m_{ij}}{n^2}$. It supposes all the clip pairs should contribute equally to the similarity score.

Max is used as a baseline in prior work [3]. It is given by $s(M)= \max _{ij} m_{ij}$. It supposes that selecting the best matching clip pair is enough to recognize the action.

1.2 A.2 Additional Ablations and Impact of Hyper-parameters

In this section, we present additional ablations to evaluate the impact of the feature projection head, the ordering of the tuples, and the number of examples per class used in the support set. We also report the impact of using the different variants for the Kinetics-100 and UCF-101 datasets.

Table 5. Impact of the dimension size of the feature projection head for Chamfer++ using ordered-tuples and $l=3$.

Table 6. Chamfer++ variants on the three datasets: UCF-101, Kinetics-100 and SS-v2.

The Impact of the Projection Layer for matching methods is validated in Table 4. The performance is consistently improved on all setups and methods by including and learning a projection layer. Although the backbone is trained with TSL on the same meta-train set, the projection layer allows features and values in the temporal similarity matrix to better align with each matching process.

Projection Head Dimension. To match the work from [21], we set the projection dimension to $D=1152$. This section evaluates the effect of using different values for D. The results are reported in Table 5. A minimum value of $D=1024$ seems enough and could be used for future experiments.

Impact of Using Different Variants. We report the accuracy for the different variants of Chamfer++ for the Kinetics-100 and UCF-101 datasets in Table 6. As for SS-v2, both variants improve performances compared to the vanilla approach.

Impact of Ordering the Clip-Tuples. In this section, we evaluate the impact of using ordered clip feature tuples $\textbf{t}^{l}$ versus using all the clip feature tuples $\textbf{t}_{all}^{l}$. The comparison between ordered tuples and all tuples is presented in Table 7. On the SS-v2 dataset, using ordered clip feature tuples boosts the accuracy. On the Kinetics-100 and the UCF-101 datasets, using ordered tuples doesn’t provide a boost and can even slightly harm the performance. Since the number of tuples is significantly lower when they are in order, using the ordered clip feature tuples is preferable.

Table 7. Impact of using ordered clip feature tuples vs all the features for different values of l.

Impact of the Number of Examples per Class. The impact of k is shown in Fig. 7 by measuring performance for an increasing number of support examples per class while keeping the number of classes fixed, $C_{t}=C_{f}=5$. We observe that TSL and TRX have inferior performances for the low-shot regime, while their performance increases faster with the number of shots. In the low-regime, Chamfer-QS++ outperforms the other methods and still keeps some benefits while the number of shots increases.

Fig. 7.

Evolution of the accuracy with the number of examples per class in the support set for the three datasets.