Antibodies are a critical part of the immune system, having the function of recognizing and mediating the neutralization of undesirable molecules (antigens) for future destruction. Being able to predict which amino acids belong to the paratope, the region on the antibody that binds to the antigen, can facilitate antibody engineering and predictions of antibody-antigen structures. The suitability of deep neural networks has recently been confirmed for this task, with Parapred outperforming all prior models. In this work, we first significantly outperform the computational efficiency of Parapred by leveraging à trous convolutions and self-attention. Second, we implement cross-modal attention by allowing the antibody residues to attend over antigen residues. This leads to new state-of-the-art results in paratope prediction, along with novel opportunities to interpret the outcome of the prediction.