In this section, we first introduce the VideoQA datasets used for our experiments as well as implementation details. We then compare the proposed method with other state-of-the-art methods and conduct ablation studies to justify each component of our method.
4.1 Datasets
Eight benchmark datasets for VideoQA are adopted for evaluation which contain videos of different lengths, scenarios, and question-answering types.
MSVD-QA [
55] contains 50,505 question-answer pairs and 1,970 video clips, with questions divided into
what,
who,
how,
when, and
where, all of which are open-ended tasks.
MSRVTT-QA [
55] contains 243k question-answer pairs and 10k videos, with questions in the same form as MSVD-QA but with more complex visual scenarios and longer video lengths of around 10 to 30 seconds.
ActivityNet-QA [
65] contains 58k open-ended question-answer pairs and 5.8k videos. All videos are untrimmed web videos with an average length of 180 seconds.
TGIF-QA [
18] contains 165k question-answer pairs and 72k animated GIFs. This dataset covers four types of questions:
Action,
Trans,
FrameQA, and
Count.
Action is a multi-choice task to identify the repetitive action;
Trans is another multi-choice task to identify the temporal transition of the action;
FrameQA is an open-ended task, with answers being particularly obvious on a single frame;
Count is a repetition counting task to figure out the number of repetitions of the action.
Youtube2Text-QA [
61] contains 9.9k question-answer pairs, with video data from MSVD-QA. Questions are divided into
what,
who,
other. We select the multi-choice task in the dataset for evaluation.
SUTD-TrafficQA [
56] contains 62k question-answer pairs and 10k videos. This dataset includes six challenging traffic-related reasoning tasks, with each question providing four candidate answers, only one of which is correct. To solve these questions, the model needs strong common sense and logical reasoning abilities.
Social-IQ [
66] contains 7.5k question-answer pairs and 1.25k videos. All questions are multi-choice tasks and are specific to in-the-wild social situations, aiming to evaluate artificial social intelligence through question-answering. This dataset has a larger proportion of questions starting with
why and
how, which often require strong reasoning abilities.
NExT-QA [
49] contains 52k manually annotated question-answer pairs and 5.44k videos. This dataset contains three types of questions:
Causal,
Temporal, and
Descriptive. The
Causal questions are designed to explain actions or uncover the intentions of previously occurring actions. The
Temporal questions are designed to assess the model’s ability to reason about temporal relationships between actions. The
Descriptive questions focus on the description of the scene in the video. All questions are divided into multi-choice and open-ended tasks, where the latter requires the model to generate answers in short phrases.
4.2 Implementation Details
Experiment Settings. To split the datasets, we use the standard training, validation, and test sets provided in each dataset. Similar to existing works, we take the open-ended questions in MSVD-QA, MSRVTT-QA, ActivityNet-QA, and TGIF-QA datasets as open-ended classification tasks. We treat the open-ended questions in the NExT-QA dataset as open-ended generation tasks, truncating each answer to a maximum length of 6 and considering each word in the training set as a member of the answer vocabulary.
We use a ResNet with 152 layers to extract appearance-motion features, and the maximum sampling scale \(N\) is set to 3 by default. The feature dimension \(d\) is set to 512. During network training, each VideoQA dataset has a pre-defined vocabulary composed of the top \(K\) most frequent words in the training set. We set the \(K\) value for the MSVD-QA dataset to 4,000 and that for other datasets to 8,000. We use the Adam optimizer to train the network, with the initial learning rate set to \(1e-4\) . When the loss is not decreased for 5 epochs, the learning rate is reduced by half. The maximum number of epochs is set to 50, and the batch size is set to 32. We implement our model with the PyTorch deep learning library on a PC with only one GTX 1080 Ti GPU.
Evaluation Metrics. To evaluate the performance of a model, we use different metrics depending on the task. For open-ended classification and multi-choice tasks, we adopt accuracy (%) to evaluate the performance. For repetition counting tasks, we use MSE computed between the predicted answer and the ground truth answer to evaluate the performance. For open-ended generation tasks, we determine the
Wu-Palmer similarity (
WUPS) score [
49] to evaluate the quality of the generated answers.
4.3 Comparison with State-of-the-art Methods
In this section, we compare our HMRNet with the state-of-the-art methods on eight popular VideoQA datasets. Unless otherwise stated, the reported results are those obtained from experiments done on the testing set.
Table
1 reports the comparison results on MSVD-QA, MSRVTT-QA, and ActivityNet-QA datasets. For videos with varying durations and complex scenes, our proposed HMRNet still achieves the best performance, i.e., 41.8% (+0.6%), 39.5% (+0.9%), and 40.6% (+3.3%) on MSVD-QA, MSRVTT-QA, and ActivityNet-QA datasets, respectively. The results of these experiments demonstrate that our method is well adapted to both short and long untrimmed video question answering.
Table
2 shows the comparison results on the TGIF-QA dataset. Our proposed HMRNet achieves the best performance on the Action (+2.9%), Trans (+1.7%), and Count (
\(-0.15\) ) tasks. The improvement is particularly noticeable for Action and Trans tasks. These tasks challenge the modeling of temporal relations between videos and questions. The PKOL [
68] obtained the best performance on FrameQA tasks, in contrast, our method achieves 12.7 and 9.7 percentage points of improvement on Action and Trans tasks, respectively. The better performance of HMRNet could be owed to its ability to extract multimodal interactive relations from questions and videos and search for answer clues at different semantic levels.
We note here that, in addition to using the necessary encoders to extract video and question features, some of the compared methods apply other complex feature extraction procedures to promote the video–question interaction. For example, Bridge2Answer [
39] additionally extracted compositional semantic relations between words using an NLP tool [
38]. GMN [
11], HAIR [
34], HOSTR [
4], MASN [
44], PKOL [
68], and HQGA [
50] applied a Faster R-CNN [
43] for object detection per frame to obtain fine-grained spatio-temporal features. Conversely, ACRTransformer [
72] adopted a BSN [
33] to specifically locate high-action frames in a given video, and MSPAN [
12] extracted motion features in a frame-by-frame style despite the large parameter size of the motion network. Furthermore, PKOL [
68] used additional prior knowledge from a massive corpus and thus achieved strong results on the FrameQA task. By contrast, our proposed relation-oriented interaction module can effectively extract multimodal interactive representations without these complex feature pre-processing steps. Moreover, most of the previous methods applied dense sampling to the input video. For example, HCRN [
25], Bridge2Answer [
39], and HQGA [
50] sampled 8 clips with each comprising 16 frames; E-SA [
65], MAR-VQA [
74], DSAVS [
36], and DualVGR [
48] sampled 20 clips for long videos. However, with fewer clips sampled (only seven clips are sampled when
\(N\) is set to 3 by default), our proposed HSM is able to search for answer cues across multiscale semantics to achieve the best performance.
Tables
1 and
2 also provide comparison results with recent large-scale pre-training approaches. Specifically, SSML [
1] was pre-trained on the HowTo100M video–text corpus. ClipBERT [
27] proposed a sparse sampling strategy and used the COCO Captions and Visual Genome Captions datasets for pre-training. With the same pre-training approach as in [
27], SiaSamRea [
64] proposed a Siamese sampling mechanism to enhance answer reasoning. HD-VILA [
57] built a large-scale, high-resolution video–text dataset to improve reasoning performance. Moreover, their models are huge, complex, and demand ample computing resources to implement training. As a comparison, our proposed HMRNet is compact and does not require any extra video–text corpus for pre-training to achieve superior performance.
Nevertheless, we also recognize that large-scale pre-training indeed largely improves the performances of several methods [
59,
67] on the above datasets. e.g., [
59] built a domain-specific dataset for pre-training and achieved a higher recognition rate. [
67] collected a very large video–text corpus called YT-Temporal-180M for pre-training and achieved a performance superior to that of our approach. However, we believe that our method, which focuses on designing compact and efficient components for the problems present in VideoQA is equally valuable. Moreover, it provides a way to promote broader research, especially while powerful computational resources are still expensive to access.
Tables
3–
5 show evaluation results on Youtube2Text-QA, SUTD-TrafficQA, and Social-IQ datasets, respectively. For the Youtube2Text-QA dataset, we achieved an improvement of 3% in overall performance compared to the previously reported best results. For the SUTD-TrafficQA dataset, we obtain the highest accuracy rate, i.e., 40.91%. We also show the performance of our HMRNet for binary and four-class classification tasks on the Social-IQ dataset: here, HMRNet improves the recognition rate by 10.1% and 10.4%, respectively, compared with Tensor-MFN [
66]. This outstanding performance suggests that our proposed HMRNet has the ability to perform causal reasoning in in-the-wild social situations and complex traffic scenarios.
Table
6 shows evaluation results for the open-ended generation task conducted on the NExT-QA dataset. Similar to [
49], we compared experimental results across all question types.
\(\mathrm{WUPS}_C\) ,
\(\mathrm{WUPS}_T\) , and
\(\mathrm{WUPS}_D\) denote the generated answer scores for
Causal,
Temporal, and
Descriptive questions, respectively.
\(\mathrm{WUPS}\) denotes the overall score for all questions. The higher the score, the more accurate the generated answer. Our method achieves better overall performance compared with other state-of-the-art approaches. Table
7 shows evaluation results for the multi-choice task conducted using the NExT-QA dataset. We experiment with the multi-choice task on the validation set.
\(\mathrm{Acc}_C\) ,
\(\mathrm{Acc}_T\) , and
\(\mathrm{Acc}_D\) denote the recognition rates for
Causal,
Temporal, and
Descriptive questions, respectively.
\(\mathrm{Acc}\) denotes the overall accuracy of all questions. We denote the approach introduced in reference [
52] as VTQG, which encompasses three graphical reasoning architectures: VQG, VTG, and TQG. The results demonstrate that our HMRNet outperforms the majority of existing methods across all question types and achieves comparable performance to VTQG.
Similarly, as we have analyzed in Tables
1 and
2, our method offers the advantage of not requiring complex feature pre-processing steps, such as those used in VTQG that rely on object detectors on each frame and seq-NMS to generate target trajectory features. Additionally, our method achieves powerful performance with fewer video clips, as compared to VTQG which samples a relatively large number of 16 clips. These benefits are due to our inter- and intra-modal relationship modeling and multiscale collaborative reasoning structure. However, it is important to note that object-level interactions, as explored in methods [
52] and [
51], offer advantages, particularly for questions that involve relational reasoning between objects in the NExT-QA dataset. Also, the approach in [
51] of constructing graph interactions of objects and leveraging BERT fine-tuning to enhance performance is worth considering for our work.
To further demonstrate the advantages of our proposed HMRNet, in Table
8, we compare it with several methods in terms of parameter size and computational efficiency. The last four methods in the table, i.e., ClipBERT [
27], VQA-T [
59], MERLOT [
67], and HD-VILA [
57], are recent large-scale pre-training methods. We use nParams and GFLOPs [
15,
56] to denote the number of learnable parameters and the computational efficiency of the model, respectively. As shown in the table, while leveraging off-the-shelf features for answer reasoning, our proposed HMRNet has the lowest number of parameters. Furthermore, HMRNet requires significantly fewer parameters than HME and HD-VILA. In terms of computational efficiency, our method also has a greater advantage compared with other methods. The GFLOPs of our HMRNet are only 0.25, which is remarkably less than those of MERLOT and HD-VILA, and less than half that of VQA-T.