HiBid: A Cross-Channel Constrained Bidding System with Budget Allocation by Hierarchical Offline Deep Reinforcement Learning

RTB attracts much attention and has been widely studied for various applications [18]. Some efforts have been devoted to designing bidding mechanisms to enhance the effectiveness and fairness of advertising auctions from the platform perspective. For example, Zhou et al. in [19] introduced a novel deep distribution network for optimal bidding in both open and closed online first-price auctions. Zhang et al. in [20] proposed a succinct and effective bid shading algorithm without parametric assumptions for the win distribution. Ren et al. in [21] proposed a comprehensive framework to jointly optimize user response prediction and bid landscape forecasting. Furthermore, there have been studies that approach the optimization of bidding strategies from the advertiser perspective, to improve the effectiveness of their ads during auctions. For example, Wu et al. in [22] developed a model-free DRL framework “DRLB” for constrained bidding to cope with the volatility of the auction environment. Yang et al. in [23] abstracted the essential demand of advertisers in RTB and proposed an effective linear programming solution. Those works focused on optimizing bidding prices under the given constraint for a single ad channel, to adapt to the unpredictability of the auction environment and satisfy advertisers’ requirements. However, there exist multiple advertising channels with significant quality differences in practical deployment. Also, some studies developed ways to allocate budget across multiple ad channels given a total budget constraint, e.g., some works [9, 10] formalized it as well-stated optimization problems. These methods required accurate estimation of outcome distributions (e.g., the expected number of clicks from choosing a particular budget), which is impracticable in a dynamic auction environment.

Different from the above works, this paper explicitly focuses on simultaneous budget allocation and constrained bidding for multiple ad channels, to maximize the advertising effectiveness for advertisers while ensuring platform revenue.

DRL has been widely applied in real-time systems, including user-item recommendation [24], ad-slots allocation [25], and real-time bidding for ad impression auctions [26, 7, 8, 27]. Cai et al. in [26] and Zhao et al. in [28] utilized DRL to learn the optimal bid for a single ad in display advertising and sponsored search, respectively. He et al. in [27] formulated the budget and financial constraints simultaneously and leveraged DRL to find a unified optimal bidding function on behalf of an advertiser. Unfortunately, the optimal bidding function may not yield optimal results in uncertain auction markets. Wang et al. in [8] proposed a curriculum-guided Bayesian DRL method to generalize to highly dynamic ad markets with ROI constraints. However, the mentioned DRL-based works only focus on single ad channel bidding and have not considered joint modeling across multiple channels. Due to the joint constraint settings of the advertiser’s financial requirements across channels, bidding individually cannot yield optimal results. It is insightful to model the relationship between channels jointly for bidding through a unified approach. Therefore, we leverage hierarchical reinforcement learning to jointly model cross-channel bidding and adjust bidding strategies through high-level budget allocation, which is one motivation of our work.

DRL provides a promising approach to address the c³-bidding problem by interacting with the environment and updating policy iteratively. However, it is not suitable for training the agent in an online setting due to the potential financial risk involved. In offline DRL, the agent learns from a fixed dataset of past interactions, rather than learning online in real-time. The main challenge of offline DRL is the distribution shift [29] of state-action visitation between the learned policy and behavior policy. Recent work [30] utilized distributional penalties to regularize the learned policy to stay close to the behavior policy. Other methods [31] used generative models to approximate the behavior distribution to stay within the support of offline data during the value back up. Ajay et al. in [13] proposed a hierarchical offline reinforcement learning method with unsupervised primitive extraction. However, directly applying existing offline DRL algorithms may not effectively solve our considered c³-bidding problem, as there is no effective method to solve the cross-channel CPC constraint and the changing allocated budget.

Constrained DRL focuses on designing efficient algorithms to find optimal policies for CMDP problems under the given constraints [32]. Some works converted the CMDP problem into a Lagrangian dual problem [33], and then found an optimal Lagrangian multiplier $\lambda$ as well as the corresponding policy which satisfied the constraint. Here $\lambda$ can be manually adjusted as a hyperparameter, which is policy-sensitive and hard to fine-tune. In recent works, gradient descent [34] or bisection search [35] were developed to get the optimal value of $\lambda$. Unfortunately, the policy needs to be retrained every time when the value of $\lambda$ changes until the constraint is satisfied. The iterative training process is unacceptable in c³-bidding problem due to the frequently changed budget that brings huge computational overhead. Thus, in this paper we adopt a $\lambda$-generalization [17] method to learn diversified bidding strategies that can dynamically respond to the changed budget constraint. Nevertheless, for the cross-channel CPC constraint, we need a more appropriate way to solve it, which is the key contribution of this paper as CPC-guided action selection in Section V-C.

The high-level planner needs to allocate the budget to maximize the number of user clicks while ensuring the revenue on each channel stays within an acceptable range. Thus the objective of the high-level planner is:

where $a_{m,p}^{h}$ denotes the allocated budget on channel $p$ for advertiser $m$, $Click^{h}(a_{m,p}^{h})$ and $Cost^{h}(a_{m,p}^{h})$ represent the number of clicks and cost given the budget $a_{m,p}^{h}$ within the interval, respectively. Meanwhile, the revenue constraint in Eqn. (7) is also regarded as a channel-capacity constraint that prevents advertisers from engaging in severe competition for high-quality channels.

We formulate the high-level MDP for each advertiser $m$ as a tuple $(\mathcal{S}^{h},\mathcal{A}^{h},\mathcal{T}^{h},\gamma^{h},\mathcal{R}^{h},\mathcal{C}^{h})$. That is, the high-level planner allocates the channel-level budget at each interval according to the advertiser’s budget requirements, while each channel allocation is regarded as a decision step. The details are defined as follows:

State $\mathcal{S}^{h}$: Let $\mathcal{S}^{h}$ denote the higher-level state space, consisting of the allocated budget and the historical statistics on the advertiser-level (e.g., the average CTR and CVR).

Action $\mathcal{A}^{h}$: The action $a_{p}^{h}\in\mathcal{A}^{h}$ is the budget assigned to channel $p$. We discretize the action by using the percentage of the budget and mask invalid actions that go beyond the total budget.

Reward $\mathcal{R}^{h}$: Reward $r^{h}_{p}\in\mathcal{R}^{h}$ is defined as the sum of number of user clicks in channel $p$ within the interval.

Constraint $\mathcal{C}^{h}$: The channel-capacity constraint and total budget constraint are given in Eqn. (6) and Eqn. (7), respectively.

After receiving the allocated budget $a^{h}_{p}$ on each channel, the low-level executor aims to maximize the number of clicks while satisfying that budget and CPC constraint simultaneously. For each advertiser $m$, the objective function of the low-level executor can be formulated as:

where $Cost(a^{l}_{p,i})$ and $Click(a^{l}_{p,i})$ denotes the real cost and whether the user clicks the ad after giving a bidding price $a^{l}_{p,i}$, respectively. Each request $i$ comes from a specific channel and only affects the cost of that channel, and thus we model each channel individually. Then, the CMDP for channel $p$ can be formulated as a tuple $(\mathcal{S}^{l},\mathcal{A}^{l},\mathcal{T}^{l},\gamma^{l},\mathcal{R}^{l},\mathcal{C}^{l})$, which is defined as follows:

State $\mathcal{S}^{l}$: The low-level state space is a collection of allocated budgets $a^{h}_{p}$, request and advertiser level information. The request-level information includes time, and current advertising status (e.g., budget consumption rate and financial constraints satisfactory ratio). The advertiser-level information is identical to the high-level planner state.

Action $\mathcal{A}^{l}$: Following [27], an action $a_{i}^{l}\in[a^{l}_{\min},a^{l}_{\max}]$ represents the bidding ratio and the final bidding price is calculated by $a_{i}^{l}*CPC_{m}^{set}$.

Reward $\mathcal{R}^{l}$: For each request $i$, reward $r_{i}^{l}\in\{0,1\}$ is set to $1$ if the bidding is successful and the user eventually clicks the ad.

Constraint $\mathcal{C}^{l}$: Single-channel budget constraints and cross-channel CPC constraints are given in Eqn. (9) and Eqn. (10), respectively.

Recall that the high-level planner’s objective is to maximize the number of user clicks under the advertiser’s budget while ensuring the revenue on each channel stays within an acceptable range. However, each channel’s request is limited and cannot accommodate all advertisers. If all of them compete for the request on high-quality channels, it will certainly lift the bidding price and reduce advertisers’ CPC satisfactory ratio. A well-designed planner may allocate different budgets for each channel based on the advertiser preference, as well as prevent budget over-allocation on specific channels.

A common solution is to leverage an advertiser-level constraint to limit the amount of budget allocation on the channel. It may result in reducing overall revenue since the bidding abilities of advertisers are quite different and thus one cannot allocate equal budget to all of them. Meanwhile, our design also faces a key challenge as how to restrict the updated policy to satisfy the channel-capacity constraint because it is hard to design a suitable reward function. Inspired by [16], we design a batch loss for the high-level planner to ensure that the budget allocated to each channel fluctuates within an acceptable range. Consider that we sample a batch of experiences containing multiple tuples $(s_{p}^{h},a_{p}^{h},r_{p}^{h},c_{p}^{h},s_{p+1}^{h})$ from the high-level dataset $\mathcal{D}^{h}$, then the batch loss is calculated by:

where $\kappa_{p}$ is calculated by multiplying historical CTR, CPC and impression count based on all advertisers in the batch, and a weight $w_{p}$ is introduced on the batch loss for learning stability. Since the argmax function is not differentiable, a soft version is applied:

where $\beta$ is the temperature coefficient and $Z=\sum_{n=1}^{|\mathcal{A}^{h}|}\exp[\beta Q^{h}_{\theta}(s_{p}^{h},a_{p,n}^{h})]$ is the normalization factor. While the high-level planner updates its strategy using randomly sampled batch experiences to maximize the number of clicks, the batch loss encourages to reallocate budgets across different channels based on historical statistics within the batch experiences. For example, it prompts some high-impact advertisers (i.e., those with higher conversion rates) to reduce their budget allocation on high-quality channels, thereby allowing low-impact advertisers to benefit from the superior advertising effects that these high-quality channels provide. This approach ensures that revenue on the channels does not fluctuate dramatically. Additionally, it prevents the high-level policy from consistently favoring higher investments in high-quality channels, which could lead to detrimental competition.

As the advertiser’s total budget and the platform’s allocated budget may change over time, we need an adaptive bidding strategy that can dynamically respond to the budget changes. However, under the offline DRL training paradigm, the low-level executor learns the bidding strategy from the low-level dataset $\mathcal{D}^{l}$ which cannot be generalized to unseen budget cases, reducing the effectiveness of the high-level budget allocation. To this end, we adopt “$\lambda$-generalization” [17] to learn an adaptive bidding strategy for dynamically changing budgets.

In order to learn a bidding strategy that satisfies the budget constraint on each channel individually, the problem of c³-bidding without CPC constraint can be converted into its Lagrangian dual problem [38] as:

where $\lambda$ is the Lagrangian multiplier that controls how much the bidding strategy spends. Thus, we take it as part of the input to the Q-network and modify the low-level reward function by:

Then the low-level policy can be formulated as:

Therefore, the low-level executor can adaptively respond to changing budgets by selecting the optimal $\lambda^{*}$. Next, we need to deal with the training of the corresponding policy under different $\lambda$ offline, and getting an optimal $\lambda^{*}$ under the budget constraint during online prediction.

During the offline training, we perform data augmentation by $\lambda$-generation, allowing the policy to learn how to bid under different $\lambda$. Given a fixed training dataset consisted of multiple tuples, we extend it by enlarging each tuple $(s_{i}^{l},a_{i}^{l},r_{i}^{l},c_{i}^{l},s_{i+1}^{l})$ into $\{(s_{i}^{l},a_{i}^{l},r_{i}^{l,\lambda_{n}},c_{i}^{l},s_{i+1}^{l},\lambda_{i}^{n})\}_{n=1}^{N}$, where $N$ is data repetition times and $\lambda_{n}$ is a uniformly sampled value from $[0,\lambda_{\max}]$. The range $[0,\lambda_{\max}]$ can guarantee that the cost of the learned policy falls within a controllable range [17]. We also construct two additional evaluation networks $\hat{Q}^{c}_{\phi}$ and $\hat{Q}^{u}_{\eta}$ to accurately evaluate the expected cost and number of clicks under the low-level policy $\pi^{l}$. We use the action from $\pi^{l}$ instead of $\max$ operator when computing the target value in two evaluation network updates by:

where $\tau^{l}$ is sampled trajectories from $\mathcal{D}^{l}$ and $c_{i}^{l}$ denotes the budget consumption under action $a_{i}^{l}$. Note that we utilize the same loss function $\mathcal{L}_{\hat{Q}}(\eta)$ for $\hat{Q}^{u}_{\eta}$ but compute number of clicks as $c_{i}^{l}$.

During the online prediction, two evaluation networks are used to perform $\lambda$-selection to ensure that the low-level policy does not exceed the budget allocated by the high-level planner. We uniformly sample $\{\lambda^{n}\}_{n=1}^{N_{p}}$ for each request $i$ within the range $[0,\lambda_{\max}]$, and select the $\lambda_{i}^{*}$ which satisfies the allocated budget $a_{p}^{h}$ as well as maximizing the number of clicks:

In this way, the low-level executor can adaptively respond to the allocated budget $a_{p}^{h}$ and ensure the effectiveness of the budget allocation made by the higher-level planner.

Due to the varying target users and advertising quality, the competition situation differs among channels, resulting in a significant discrepancy in terms of CPC between channels (i.e., high-quality channels have higher CPC). Therefore, we cannot simplify the cross-channel CPC constraint by setting the same target CPC for each channel. When we use Lagrangian relaxation to deal with both budget and CPC constraints simultaneously, it is impossible to find an effective pair of Lagrangian multipliers to satisfy them due to the explosion of the solution space. We design a CPC-guided action selection (CPC-AS) scheme to help the low-level executor choose the action that satisfies the CPC constraint by considering both the past and the future.

When making a decision for a request $i$, the final CPC for an advertiser $m$ is divided into two parts:

where $Cost_{m,t}$ and $Click_{m,t}$ denote the costs and number of clicks already happened up to now, respectively. As we describe in Section V-B, two evaluation networks $\hat{Q}^{c}_{\hat{\theta}}$ and $\hat{Q}^{u}_{\hat{\theta}}$ are developed to estimate the expected discounted costs and the number of clicks, respectively. Given the current state-action pair $(s^{l}_{i},a^{l}_{i})$, we approximate the expected costs and number of clicks through $\hat{Q}^{c}_{\phi}(s^{l}_{i},a^{l}_{i},\lambda)$ and $\hat{Q}^{u}_{\eta}(s^{l}_{i},a^{l}_{i},\lambda)$. Therefore, we define $CPC_{m}^{pred}(a_{i}^{l})$ by combining the past and expected future by: