Amortizing Pragmatic Program Synthesis with Rankings

Finding correct programs given a sequence of examples $\mathbf{u}=u_{1},u_{2},\dots$ is the primary challenge of program synthesis, with solutions ranging from enumeration (Feser et al., 2015), constraint solving (Solar-Lezama et al., 2006), neuro-symbolic (Polosukhin & Skidanov, 2018; Balog et al., 2016), and using large language models for code (Li et al., 2022). In this work, we assume the a set of $k$ consistent programs $w_{1},w_{2},\dots,w_{k}$ can be found using any of these techniques.

A global ranking $\sigma$ is an un-normalized prior (a score) over all programs. The global ranking is example-agnostic: given two programs $w_{a}$ and $w_{b}$, either $\sigma[w_{a}]\succ\sigma[w_{b}]$ or $\sigma[w_{a}]\prec\sigma[w_{b}]$, irrespective of the given examples $\mathbf{u}$.

As we can see, ranking the consistent programs under $\sigma[w]$ can be very efficient. In practice, efficient synthesis algorithms are built using either domain-specific heuristics for rankings (Singh & Gulwani, 2015; Polozov & Gulwani, 2015), or a learned prior from a code corpus (Li et al., 2022).

Rather than relying on heuristics or learning from a large corpus, RSA automatically derives a ranked synthesizer $L_{1}(w|\mathbf{u})$:

To rank the consistent programs, $L_{1}$ uses $S_{1}(\mathbf{u}|w)$, an example-dependent ranking function, that ranks the satisfying programs differently depending on the sequences of examples $\mathbf{u}$ given. In this setting with multiple examples, there could be cycles where a pair²²2or a triple or larger cycles of satisfying programs $w_{a}$ and $w_{b}$, which is ranked $S_{1}(\mathbf{u}_{1}|w_{a})>S_{1}(\mathbf{u}_{1}|w_{b})$ under some examples $\mathbf{u}_{1}$ and ranked $S_{1}(\mathbf{u}_{2}|w_{a})<S_{1}(\mathbf{u}_{2}|w_{b})$ given different examples $\mathbf{u}_{2}$. In this work, we assume that $S_{1}$ can be tractably computed at non-interactive speed.

In this work, we explore whether the example-dependent ranking of $S_{1}(\mathbf{u}|w)$ can be approximated — to have similar top-$k$ responses — with an example-agnostic ranking function $\sigma[w]$. Note that due to the existence of cycles, it may be impossible to find a global ranking that is consistent with all example-dependent rankings. Our key findings are as follows:

Key Finding 1: One can distill a pragmatic ranking $\sigma_{L_{1}}$ from $L_{1}$. While this is an approximation, it nonetheless retains much of the $L_{1}$’s communicative accuracy when interacting with end-users, and running orders of magnetudes faster.

Key Finding 2: In the special case where only a single example is used, RSA_single, the approximation can be made exact: There exists a global ranking $\sigma^{*}$ that perfectly matches the top-k responses of $L_{1}$ over any example $u$.

In our experiments, the users (real or simulated) will be given a target program, and attempt to communicate it to the synthesizers using examples. The synthesizers will be measured on their communication accuracy — whether the synthesizers can infer the target program from the examples given. A synthesizer is better than another if it can recover the target program using fewer examples.

For a sequence of listeners in the RSA algorithm $L_{0},L_{1},\dots$ over a boolean-valued lexicon $M$, there exists a sequence of global pragmatic rankings $\sigma_{L_{0}},\sigma_{L_{1}},\dots$ such that:

This means the partial rankings produced by any $L_{i}$ over consistent hypotheses are example-agnostic, where a global ranking preferring certain hypotheses unconditionally over others (e.g. a convention) is sufficient to explain the relative rankings of $L_{i}$ resulting from RSA_single.

Let $M$ be a boolean lexicon of size $m$ rows and $n$ columns. Let $r_{0}=r_{0}^{1}\dots r_{0}^{m}$ be the row-normalizing vector such that $r_{0}^{j}=(\sum M[j,:])^{-1}$, which is to say, each element $r_{0}^{j}$ is the normalization term for row $j$ of $L_{0}$. Let $\operatorname{*_{\leftrightarrow}}$ denotes row-wise multiplication:

Which is to say, starting from $M$, $L_{0}$ can be obtained by scaling each row $j$ by their respective normalization constant $r_{0}^{j}$. Let $c_{1}=c_{1}^{1}\dots c_{1}^{n}$ be the col-normalizing vector such that $c_{1}^{j}=(\sum L_{0}[:,j])^{-1}$, which is to say, each element $c_{1}^{j}$ is the normalization term for column $j$ of $S_{1}$. Similarly, let $\operatorname{*_{\updownarrow}}$ denotes column-wise multiplication

Computing $L_{i}$ under RSA amounts to applying row and column normalization alternatively multiple times:

Let $*$ be element-wise multiplication, let $\otimes$ be outer-product, we can rearrange the terms:

Here, $r_{0\dots i}=r_{0}*\dots*r_{i}$ is a vector of size $m$, and $c_{1\dots i-1}=c_{1}*\dots*c_{i-1}$ is a vector of size $n$. As we can see, following the RSA algorithm, $L_{i}$ can be decomposed to to multiplication of 2 parts: the lexicon $M$, and a matrix that is formed by the outer product $r_{0\dots i}\otimes c_{1\dots i-1}$ ⁵⁵5note that any prior over hypotheses and utterance can be similarly absorbed into these outer products terms.

Claim: The ordered indexes of $c_{1\dots i-1}$ is the global pragmatic ranking $\sigma_{L_{i}}$:

Proof: We show both sides of the $\iff$. Suppose that for some $w,w^{\prime},u$, both $L_{i}(w|u)>0$ and $L_{i}(w^{\prime}|u)>0$ (i.e. $M[u,w]=M[u,w^{\prime}]=1$).

(1) Show $\Rightarrow$: Suppose $L_{i}(w|u)>L_{i}(w^{\prime}|u)$. We have

As $r_{0\dots i}[u]$ is a constant, we have

(2) Show $\Leftarrow$: Suppose $c_{1\dots i-1}[w]>c_{1\dots i-1}[w^{\prime}]$.

Thus, $c_{1\dots i-1}$ is the global ranking $\sigma_{L_{i}}$ as claimed $~{}~{}~{}~{}\blacksquare$.

We check the our proof using simulations on $10000$ randomly generated boolean lexicons size ranging from $10\times 10$ to $20\times 20$, and running a chain of $100$ listeners on top. A total ordering can be found for all of them (Appendix B.1). We further study the stability of these ranks as they are formed, finding that the formed rankings tend to be stable across different RSA iterations (Appendix B.2).

Prior work such as that by Monroe et al. (2017) and Andreas & Klein (2016) has largely focused on sample and re-rank as a way of scaling RSA, making the example-dependent ranking function $S_{1}(u|w)$ more efficient at a cost of accuracy. Recent work by Key et al. (2022) and Vaduguru et al. (2024) apply the sample and re-rank approach to program synthesis, resulting in neural program synthesizers that also rank programs in an example-dependent way. Our work enables a different kind of synthesis algorithm altogether — that of a distilled pragmatic ranking that rank consistent programs agnostic to examples given. We view these works as complimentary, able to efficiently produce a simulated communication dataset $D$ which our approach can distill from.

RSA has been applied to improve the performance of language interfaces in a variety of other domains, such as image description (Andreas & Klein, 2016; Cohn-Gordon et al., 2018a, b), instruction generation and interpretation (Fried et al., 2018a, b), and grounded interaction (Fried et al., 2021; Lin et al., 2022). These works all use speaker models trained on labeled data from people. Our approach requires no human-produced data, and can be run entirely from the lexicon $M$ of the synthesis problem. On the other hand, we can easily integrate human data within our approach by training similar speaker models on the collected interactive data.

Prior works on resolving ambiguity in program synthesis have relied on example-agnostic ranking functions. Works such as Singh & Gulwani (2015); Polozov & Gulwani (2015) use scoring functions to penalize certain properties of programs (e.g. discouraging the use of constants), effectively inducing a global ranking over all programs;  Ellis & Gulwani (2017) uses a set of hand-crafted features to learn a naturalistic ranking from data. Synthesis algorithms that use a large neural code model to sample a large number of programs (Chen et al., 2021; Li et al., 2022) implicitly rank the programs based on their naturalistic distributions in its training data. Our work is unique in that (1) the learned ranking is rooted in efficient communication rather than hand-crafted features and (2) our approach does not require human annotated data.

Recent work by Muggleton FREng (2023) shows that in the case of single-example, the MAP estimate of the learner can be completely ranked by $sz(H)+\ln{g(H)}$ an example-agnostic global ranking. Our work can be viewed as a strict generalization in the following sense: They consider the chain of recursive bayesian reasoners of the form $M\rightarrow S_{0}\rightarrow L_{1}$, whereas our result applies to any alternating chains speakers and listeners of arbitrary depth. Their notion of “specificity” and “program length” also has direct analogies to the normalization terms in Equation 5, except these analogies do not carry over to deeper recursive depths.