How to do an evaluation: pitfalls and traps

Erratum to: J Comput Aided Mol Des DOI 10.1007/s10822-007-9166-3

Due to an error in the transcription of the formula for calculating DPI several items in the paper are incorrect. The erratum corrects these errors, but the conclusions of the paper are not changed.

What happened:

A mistranscription of the formula for the calculation of the DPI resulted in an error in Eqs. 1 and 2 and some data discussed in the paper. The correct versions of Eqs. 1 and 2, the related text that follows Eq. 1 in the paper, Table 2 and Figs. 3 and 4 are given below.

Table 2 Resolution and DPI for selected structures from the Kirchmair dataset

Fig. 3

Uncertainty in atomic positions for 572 structures from the Kirchmair dataset

Fig. 4

The nominal resolution versus the coordinate error for a subset of the Gold (structures with resolution <2.5 Å) and the Glide data sets

The altered text and tables:

$$ \sigma {\left( {r{\text{, }}B_{{{\text{ave}}}} } \right)} = 2.2 N{_{\text{atoms}}} ^{{1/2}} V_{{\text{a}}}^{{1/3}} n_{{{\text{obs}}}}^{ - 5/6} R_{{{\text{free}}}} $$

(1)

Eq. 1

Goto et al.’s formula to calculate DPI.

In this treatment the standard error of position, σ(r) or DPI, is related to the number of atoms in the unit cell, N_atoms, the volume of the asymmetric unit cell, V_a, the number of crystallographic observations, n_obs and the R_free. It should be noted that the formula presented by Goto et al. is not precisely the same formula that Blow derives in his paper. In Eq. 1 the prefactor is given as 2.2, while in the original work by Blow the prefactor is given as 1.28. This is because Blow is calculating coordinate error for a particular axis, σ(x, y, or z), while Goto et al. are calculating the error in the distance, σ(r), giving rise to the √3 difference between the two prefactors. As such it is appropriate to consider the error in the coordinates, σ(x, y, z) as a measure of the uncertainty in the atomic positions, and the error in inter-atomic distances, σ(r), when comparing a computed and an experimental structure. Goto et al. also make the assumption that the errors in the computed pose are the same as for the experimental pose. In this analysis a computed and experimental pose must be different by an RMSD of at least √2 * DPI (which is the uncertainty in the atomic positions) for the difference to be meaningful.

The resolution of a crystallographic model, as has been mentioned, is often used to select protein structures for pose prediction by docking or conformer generation studies, on the assumption that resolution imparts some information on the quality and precision of the model. The DPI is a much more direct estimate of the reliability of crystallographic models when it comes to comparing experimental and computed atom positions (as is done in conformer reproduction or pose prediction). It is therefore of interest to compare the nominal resolution for a large number of “good quality” crystal structures with the DPI (σ(r)) and the positional uncertainty for the same structures. A good dataset for this comparison is provided by the extensive investigations performed by Kirchmair et al. [4]. Here 778 co-crystal structures were used to provide experimental ligand structures that were then compared to sets of computed conformations from conformer generation applications. From this set 750 could be downloaded from the RCSB and for 572 of these 750 crystal structures there exists sufficient data to allow the DPI to be calculated. The relationship between the nominal resolution for these structures and their average uncertainty in the atomic positions is shown in Fig. 3. It is apparent from Fig. 3 that the statement by Kirchmair that “0.5 Å approximately represents the accuracy of protein X-ray crystallography” is not supported by the actual properties of the crystal structures they studied. In fact, in those cases where the DPI can be calculated, 47% of the structures from their paper have atomic uncertainties >0.5 Å.

Figure 3 illustrates a number of other interesting points. While the expectation that greater coordinate precision will arise from structures with better nominal resolution is generally borne out by the data, there are many exceptions. Table 2 shows some examples of structures where the nominal resolution gives an unexpected estimation of uncertainty in atomic positions. The top half of the table shows structures with good nominal resolution but unexpectedly high DPIs, while the lower half of the table shows some structures with low nominal resolution and either unexpectedly low or unexpectedly high DPIs. Accordingly, simply using nominal resolution as a metric of quality for structures to be used in a pose prediction or conformer generation study is insufficient to guarantee that structures of appropriate quality will be used.

With the DPI for a structure in hand one can set a lower limit on the precision with which a computed conformation can reproduce an experimental one—an RMSD between the two conformations of less than the average atomic uncertainty for the experimental structure is not meaningful. It can be seen that almost half (47%) of the structures in this set has uncertainties >0.5 Å, while Kirchmair et al. report pose reproduction statistics both at <0.1 Å and at <0.5 Å RMSD. Since in close to half of the structures in this dataset the uncertainty in the atomic positions is greater than 0.5 Å, in these cases Kirchmair et al. report pose reproductions that are more precise than the accuracy of the source data allows.

$$ \sigma {\left( {r{\text{, }}B_{{{\text{ave}}}}}\right)} = 0.31{\left({1 + s}\right)}^{{1/2}}V_{M}^{{-1/2}} C^{{-5/6}}R_{{{\text{free}}}} d_{{\min }}^{{5/2}} $$

(2)

Eq. 2

Blow’s derivation for the relationship between nominal resolution and atomic precision using the Goto et al. [18] coefficient.

The variables found in Eq. 2 are as follows: s is the percent solvent present in the crystal, V _M is the asymmetric unit volume to molecular weight ratio, C is the completeness of the data, and d _min is the nominal resolution of the structure. The ideal lines shown in Fig. 4 were calculated using Eq. 2 and assuming an s of 0.0, a V _m of 3.5, a C of 100%, and that R _free is equal to the resolution/10.