End-to-end training of Multimodal Model and ranking Model

We select two modalities: visual modality from video frames, and text modality from titles, descriptions, ASR and OCR. Given the item $A$, we use $\textrm{img}_{m}(A)$ and $\textrm{txt}_{k}(A)$ to represent the raw content materials. Next, we extract single-modal features from vision encoder $f_{\textrm{ve}}$ and text encoder $f_{\textrm{te}}$:

where $M$ indicates the number of visual modalities, $K$ indicates the number of text modalities, $\overline{v}_{m}(A)$ and $\overline{t}_{k}(A)$ represent the output of single-modal encoder.

Then, we downsize their dimensions through several FC layers:

in which $f_{v}$ and $f_{t}$ represent the FC layers, ${v}_{m}(A)$ and ${t}_{k}(A)$ are the final single-modal representations.

Motivated by (Li et al., 2023), we propose Fusion-Q-Former (FQ-Former) to fuse different modalities, which is made up of transformers and a set of trainable queries.

Given the sets of single-modalities $\boldsymbol{v_{A}}=\left\{v_{1}(A),\dots,v_{M}(A)\right\}$ and $\boldsymbol{t_{A}}=\left\{t_{1}(A),\dots,t_{K}(A)\right\}$, we concatenate them with the globally shared queries $\boldsymbol{q}=\left\{q_{1},\dots,q_{Q}\right\}$, where $Q$ is the number of queries. Then we input them into the transformers, which are abbreviated as TRM in eq. 6, aiming to learn the relationships and importance between different modalities using self-attention (SA). Finally, we slice the first $Q$ output tokens to generate the multimodal content features $c_{A}$:

FQ-Former has advantages over traditional fusion methods: (i) Fixed-length: the output size of FQ-Former is fixed and regardless of the number of modalities, making it more suitable for the industrial variable-length modalities. (ii) Robust: since the queries participate in SA, FQ-Former can relieve the potential negative impacts, which might be caused by the low-quality materials or already enough interaction data, by assigning more attention weight to queries.

It is widely acknowledged that recent open-source backbones have been sufficiently trained and very close to the ceiling. Therefore, freezing the weights in single-modal encoders and only training the top layers or additional modules are gradually adopted by recent works (Ge et al., 2018; Chen et al., 2022; Hu et al., 2021; Li et al., 2023). We also choose this hybrid training paradigm by freezing $f_{\textrm{ve}}$ and $f_{\textrm{te}}$; and end-to-end training the $f_{v}$, $f_{t}$, $\boldsymbol{q}$ and $f_{\textrm{FQ-Former}}$, to balance the trade-off between efficiency and effectiveness.

Earlier works (Zhou et al., 2018; Pi et al., 2020) have shown that sequential modeling can improve the performance, guiding us to model user content interest in a similar manner.

Given the candidate item $A$ and the user behavior sequence $\boldsymbol{u}=\{u_{1},u_{2},...,u_{N}\}$, we first utilize the multimodal model to generate their content embeddings $c_{A}$ and $\{c_{u_{1}},c_{u_{2}},...,c_{u_{N}}\}$. Then we calculate the attention score between $c_{A}$ and each $c_{u_{i}}$, and utilize the scores to weighted average the content sequence:

where $u_{A}$ represents the user content interest, a($\cdot$) represents the attention score function, $N$ is the sequence length, $w_{i}$ is the softmaxed attention score.

In practice, we encounter difficulties with GPU OOM when we intend to increase the sequence length. Given the hidden-size $d$, each FC layer in transformers requires $O(d^{2})$ consumption. Taking the sequence length $N$ into consideration, the consumption inflates to $O(N\cdot d^{2})$, greatly limiting the sequence length.

We notice that the parameters in multimodal model will stabilize after a few days of training, which inspires us to utilize the Low-Rank Adaptation (LoRA) technique (Hu et al., 2021) by switching from full tuning to partial tuning. Let’s consider the weights $W^{d\times d}$ that need to be optimized, we first train it on a short sequence length. After a period of sufficient training, we freeze $W$ and add a trainable LoRA module to continuously follow the time-varying online distribution.

As shown in fig. 2, the LoRA module contains two bypassed FC layers $W_{a}^{d\times r}$ and $W_{b}^{r\times d}$, where $r\ll d$. Compared with the $W^{d\times d}$, they have the same output size but fewer trainable parameters. We add their outputs together:

where $\textrm{sg}(\cdot)$ is the stop gradient operator. By the way, we reduce the number of trainable parameters from $O(d^{2})$ to $O(rd)$, allowing us to increase the sequence length $N$ so that we can model user long-term content interest.

We conducted experiments on two state-of-the-art production models in our system:

• •

  An e-com online-learning CTR model that predicts whether a user will click on the item.

• •

  An ad batch-training CVR model that estimates whether a user will stay in the app in the next day.

Our experimental datasets come from our RS. In the e-com scenario, we use 2.1 billion records as the training set and 250 million as the test set. In the ad scenario, we use 82 million records as the training set and 2.2 million as the test set. Due to the confidentiality policy, we do not specify the beginning and ending time of dataset.

In offline evaluation and ablation studies, we use AUC as the primary performance indicator. In online A/B test, we consider three key metrics for the e-com system: the Gross Merchandise Value (GMV), Order Volume, and CTR. For the ad system, we consider two important metrics: the Revenue Per Mille (RPM) and Income.

We utilize the Swin-T-22K¹¹1https://github.com/microsoft/Swin-Transformer as the vision encoder, and choose the RoBERTa-wwm-ext²²2https://github.com/ymcui/Chinese-BERT-wwm as the text encoder. Both have been proven to be powerful single-modal models in practice.

The values of the hyperparameters that we select are as follows: the number of queries is 2, the number of FQ-Former layers is 1, the number of SA heads is 4, and the weight of CIC loss is 0.1, the CIC temperature is 0.1.

We combine the following modalities in various ways to evaluate how different modalities impact the performance:

• •

  Text: a sentence composed of title, description, ASR and OCR.

• •

  Image: the cover frame of a short video.

• •

  Video: several frames from a short video.

The results are listed in table 2, demonstrating that multi-modalities can facilitate the modeling on recommendation, and increasing the number of modalities can further enhance performance.

We compare performance of the following fusion methods:

• •

  1flow (Kim et al., 2021): concatenates modalities and fuses them using transformers. The variable-length sequential outputs with paddings are used as the content features.

• •

  2flow (Lu et al., 2019): utilizes two independent transformers with cross-attention layers to allow the modalities to interact with each other. The output of text flow is used as the content feature.

• •

  Masked FQ-Former: the queries interact with modalities via Q & K, but their V do not participate in pooling.

• •

  FQ-Former: our proposed method.

As shown in table 3, FQ-Former outperforms traditional fusion methods, achieving the highest AUC gain, and taking queries into attentive pooling can further boost the improvements.

We compare the AUC before and after using LoRA. As shown in table 4, when training all parameters with a sequence length 20, the AUC gain is 0.179%; when using LoRA with a sequence length 50, the AUC gain is 0.256%, which verifies that a longer sequence with fewer trainable parameters can improve the performance.

We conduct this ablation experiment to assess the improvement of CIC. As shown in table 5: when training without CIC, the AUC gain is 0.143%; when training with CIC, the AUC gain is 0.242%, demonstrating that CIC can benefit the performance.

The final content features in our method are composed of two parts: the item-side $c_{A}$ and the user-side $u_{A}$, this ablation study aims to understand the benefits of each. As shown in table 6, either item feature or user feature can improve the performance, combining them together achieves the best.

The gains of E2E may come from three aspects: (1) the injection of content information; (2) the guidance from recommendation task; (3) the parametric matching between the multimodal model and the ranking model. Accordingly, we set up three experimental groups to gradually split the benefits:

• •

  PE: pre-extracts frozen features from a multimodal model, which has the similar structure with FQ-Former and has been fine-tuned in our system.

• •

  Task-specific PE: we first fine-tune the content model through E2E method, and pre-extract content embeddings for all items. Finally, we feed them into the ranking model as frozen features.

• •

  E2E: continuously trains the multimodal model with ranking model together.

As shown in table 7, the improvement of (1) is 0.071%; (2) adds another 0.114% increase; (3) provides an additional promotion of 0.057%. Combining them all together can result in a total promotion of 0.242%.

Cold-start IDs always lack generalization. In order to analyze the differences in ItemID embeddings between the baseline ranking model and EM3, we dump all their ItemID embeddings from the parameter server. Next, we randomly sample thousands of cold-start items that are created within 5 days, and search for similar items in their respective vector spaces using cosine similarity. Then we calculate the material similarities between each candidate item and its top3 similar items. This is done in an indifferent vector space from a frozen multimodal model, which has been fine-tuned in our system and has been utilized in section 4.3.6.

As the results shown in table 8 and the typical cases shown in fig. 4(a), the cold-start ItemIDs of EM3 achieve a great improvement in material similarity.

On popular items, we mainly focus on whether behavioral information has an impact on the content embeddings. For fairness, we choose the frozen multimodal model mentioned above as a comparison for EM3. First, we utilize the frozen model and EM3 to calculate content embeddings for all items. Next, we randomly sample thousands of popular items from top 30% GMV, and search for similar items in their respective vector spaces. Then we calculate the behavioral similarities between each candidate item and its top3 similar items. This is done in the vector space of baseline ItemID embeddings, which have only been trained by user-item interaction data and can be used to measure the behavioral similarity.

As the results shown in table 9 and the typical cases shown in fig. 4(b), we can conclude that the content embeddings of EM3 achieve a significant improvement in behavioral similarity.

The Amazon dataset has been widely used in previous studies, providing both interaction records and multimodal information. We choose 2 categories: Baby and Sports. For each category, we randomly split 80% of historical records as a training set, 10% for validation and the remaining 10% for test.

We integrate the fusion and CIC modules into the FREEDOM (Zhou and Shen, 2023), which is one of the most effective works at the time of writing. In details, we first use FQ-Former to calculate the multimodal embeddings of all items. Then we utilize GCN to process them on both item-side and user-side. Next, we align the item-side multimodal embeddings with ItemID embeddings using CIC. Finally, we concatenate them for retrieving.

Same as the previous works, we fix the embedding size of both users and items to 64, and use the negative sampling strategy to pair each user-item interaction with one negative item.

We compare our proposed model with the following methods: BPR (Rendle et al., 2012), LightGCN (He et al., 2020), VBPR (He and McAuley, 2016), SLMRec (Tao et al., 2022), LATTICE (Zhang et al., 2021), BM3 (Zhou et al., 2023a), FREEDOM. For a fair comparison, we retrain FREEDOM and some other models in our environment.

We perform a grid search to find its optimal settings on different datasets: we search for the number of queries from {1, 2}, the dropout rate of transformer from {0.5, 0.8}, and the CIC temperature from {0.5, 0.1}.

We select two widely-used evaluation metrics for top-K recommendation: Recall@K and NDCG@K, which are abbreviated as R@K and N@K. We use R@20 on the validation data as the training stopping indicator, and report the average metrics of all users in the test sets for both K=10 and K=20.

As shown in table 10, EM3 outperforms the baselines on both datasets. It shows that the method proposed by us is generalizable. The source code and evaluation logs are available at https://github.com/em3e2e-anonymous/em3.

Because the public evaluation is not the main focus of our research, we only simply integrate our method with one of previous works. Considering the flexibility of EM3 in decoupling from the original recommendation models, we believe that it can be combined with other models for a similar improvement.