See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/44799693

Least squares in calibration: Dealing with uncertainty in x

Article  in  The Analyst · August 2010

DOI: 10.1039/c0an00192a · Source: PubMed

CITATIONS READS
11 1,373

1 author:

Joel Tellinghuisen
Vanderbilt University
326 PUBLICATIONS   7,644 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Isothermal Titration Calorimetry View project

Analysis of dPCR data View project

All content following this page was uploaded by Joel Tellinghuisen on 04 June 2014.

The user has requested enhancement of the downloaded file.

 Least squares in calibration: Dealing with uncertainty in x
by

Joel Tellinghuisen
Department of Chemistry
Vanderbilt University
Nashville, Tennessee 37235

ABSTRACT

The least-squares (LS) analysis of data with error in x and y is generally thought to yield best

results when carried out by minimizing the “total variance” (TV), defined as the sum of the properly

weighted squared residuals in x and y. Alternative “effective variance” (EV) methods project the

uncertainty in x into an effective contribution to that in y, and though easier to employ are considered

to be less reliable. In the case of a linear response function with both sx and sy constant, the EV

solutions are identically those from ordinary LS; and Monte Carlo (MC) simulations reveal that they

can actually yield smaller root-mean-square errors than the TV method. Furthermore, the biases can

be predicted from theory based on inverse regression — x upon y when x is error-free and y is

uncertain — which yields a bias factor proportional to the ratio sx2/sxm2 of the random-error variance

in x to the model variance. The MC simulations confirm that the biases are essentially independent of

the error in y, hence correctable. With such bias corrections, the better performance of the EV method

in estimating the parameters translates into better performance in estimating the unknown (x0) from

measurements (y0) of its response. The predictability of the EV parameter biases extends also to

heteroscedastic y data as long as sx remains constant, but the estimation of x0 is not as good in this

case. When both x and y are heteroscedastic, there is no known way to predict the biases. However,

the MC simulations suggest that for proportional error in x, a geometric x-structure leads to small bias

and comparable performance for the EV and TV methods.

 -2-

Introduction
Classical univariate calibration is the default calibration method in most analytical work. It is

implemented by least-squares (LS) fitting n calibration points (xi,yi) to a response function, usually

linear, and then estimating the amount of the unknown (x0) from one or more measurements of its

response (y0). The procedures for carrying out such an analysis have been well established for a long

time,1-4 and many authors have considered its optimization and its limitations — especially the need

for nonlinear response functions and for proper weighting to handle heteroscedastic data (data of

varying uncertainty).5-13

Somewhat less attention has been given by the analytical community to another problem:

uncertainty in x. Most algorithms for linear and nonlinear LS employ the standard assumption that x

is error-free. Although this is seldom rigorously true, the controlled variable may be sufficiently

precise that ignoring its uncertainty has negligible consequences. Sometimes closer examination of

the data forces the conclusion that x is the more uncertain variable, making reversed regression

better.14 In any case it is desirable to know the effect of neglected uncertainty in the independent

variable. And when uncertainty in x and y are comparable, proper analysis requires methods that go

beyond the capabilities of the standard LS algorithms.15-18

In the fitting of data with error in x and y, it is generally thought that the "best" solution is

obtained by minimizing the quantity STV (TV = "total variance"; see Table 1 for a notation summary)
STV = S wxidxi2 + w yidyi2 , (1)

which was proposed by Deming19 and implemented in iterative algorithms as early as 1972.20-23 In

eqn (1) the di's are residuals in x and y. Minimum-variance estimates of the model parameters are

assumed to result from the use of weights inversely proportional to variance — wxi = Csxi-2 and wyi

= Csxi-2 — as can be proven to hold for conventional linear least squares (LLS) with a single

uncertain variable (y).

A complicating factor in specifying and minimizing STV is the need for "calculated" values (Y

and X) for both y and x, since the residuals are yi - Yi and xi - Xi. This, in turn, means that both x and

y must be adjusted in an iterative minimization of STV. To get around this difficulty, a number of
 -3-

authors have recommended "effective variance" (EV) methods, in which the variance in x is projected

into an effective contribution to that in y = f(x) using error propagation,24-27

sy,eff2 = sy 2 + (dy/dx)2 sx2. (2)

For y = a + bx, this becomes sy,eff2 = sy 2 + b 2sx2; evaluated at each point, it yields weights wi =

syi,eff-2, to be included in the minimization target,

SEV = S widyi2. (3)

SEV is minimized wrt the adjustable parameters, with the weights wi iterated to consistency, but with all

computations done with the observed xi values.

The EV method is easy to use with standard programs for weighted linear and nonlinear LS,

but it has been criticized for some obvious shortcomings.28,29 These include the observation that the

EV straight-line solution becomes just the unweighted or ordinary LS (OLS) solution for regression

of y upon x when the weights in x and y are constant (but not necessarily the same), true even when the

uncertainties in x greatly exceed those in y, making regression of x upon y more sensible. On the other

hand, with a subtle change in the minimization procedures, we obtain

dyi2
SEV2 = S 2 (4)
sy !+!b 2sx2

which is written in a way to emphasize that now the dependence of the weights on the slope b is

included in the minimization. Williamson long ago demonstrated that for the linear response function,

the EV2 and TV solutions are identical.30 Yet the computations still employ the observed xi,

maintaining the appeal of the EV approach.

Does minimizing STV truly yield statistically best estimates of the parameters? The question is

relevant, because for nonlinear LS (NLS) there is no guarantee that minimizing the variance will also

yield minimum-variance estimates of the parameters,31 in part because many NLS parameters do not

even have finite variance. And the TV method indeed qualifies as nonlinear, even for straight-line

fits.32 Further, it has long been known that inverse regression — x upon y when x is error-free and y

is uncertain — yields more precise estimates of unknowns in calibration.33-37 Thus we might suspect

that neglect of uncertainty in x in the present situation might actually lead to more precise parameter
 -4-

estimates. That can in fact be true, as is illustrated here in Figs. 1 and 2 for a model with constant

error in x and y, structured like one used previously by Kalantar, et al.,32 where similar effects were

present but were less obvious due to smaller data error (their Table 1, a check of results obtained

earlier by Christian and Tucker38). Note that although the biases are greater for OLS, the parameter

standard deviations are enough smaller than predicted to make the rms error (srms2 = s2 + bias2) fall

close to the predicted standard error (SE) for all sx.

The results shown in Figs. 1 and 2 carry the surprising implication that it is better to neglect

the error in x! That is to say, OLS may yield more precise parameter estimates than the more

demanding TV method. However, this result needs some qualification; and there are significant

parameter biases in the EV method that must be considered. Below, I will use Monte Carlo (MC)

simulations to show that the theory of Shukla for reversed regression36 reliably predicts the parameter

biases, for a range of data error structures in x and y, provided only that the uncertainty in x be

constant. Since the parameters in straight-line calibration are just means to an end, this seems to mean

that the result holds any time at least one of the two variables has constant uncertainty, being thus

designated as x. However, the MC simulations show that good performance estimating the calibration

parameters does not necessarily extend to estimating x0 in calibration, and we find that only with

constant error in both x and y and with correction for the biases does the EV method match or

outperform the TV method in this application. There is one important proviso to these results: To

correctly predict uncertainties of parameters and unknowns, one must know the uncertainty in x.

However, this requirement is anyway present for proper use of the TV method, as is clear from the

statements about weights given after eqn (1).

The bias correction for OLS treatment of data with constant error in x and y is found to be

proportional to the square of the ratio of the uncertainty in x to the range of x, a quantity that is easy to

evaluate, facilitating simple checks of the significance of uncertainty in x in typical calibration

situations. Further, the MC simulations demonstrate that when bsx < sy , the OLS treatment yields

reliable estimates of the uncertainties, requiring only corrections for bias, usually small.
 -5-

Theory and computational methods

Least squares with the EV methods

The theory and procedures of LLS and NLS, with and without weighting, are well summarized

in the literature, including in previous works by me in this journal;11-14 so only a brief review is

needed here. In particular, the matrix expressions given in ref. 13 remain valid for the EV and TV

methods, with only a few modifications needed for the latter. For example, one can obtain the EV

solutions for the straight line with any weighted LS code, by simply defining the weights as indicated

after eqn (2) and iterating to consistency. This iteration can be done manually — by running the fit,

recomputing the wi with the obtained slope b, and repeating until b stops changing. Adequate

convergence is usually obtained in fewer than 10 cycles. For the TV method in general, there are

additional equations for adjusting the xi;21-23 however, for linear response functions, such adjustment

is not needed, since the EV2 and TV results are identical. Thus one can obtain the solutions using any

code for NLS which permits the user to define the fit function.39,40 Usually the latter must be

expressed in the form F(x,y; a,b) = 0. Then the EV2 solution for the straight line is obtained using
a!+!bx!-!y
F(x,y; a,b) = (5)
(sy 2!+!b 2sx2)1/2

with the weights (if required) formally set as wi = 1. [In the general case, sx and sy vary with i, so

they may need to be defined as additional "variables" to incorporate this dependence.]

Importantly, the expression for the variance-covariance matrix, which contains the parameter

variances as diagonal elements, remains unchanged. Thus, with the sxi and syi considered known, we

use Vprior, given by

Vprior = (XT W X)–1 . (6)

The weight matrix W is diagonal, with Wii = wi = (syi2 + b 2sxi2)-1 for the EV method. If the EV2

method is implemented via eqn (5), the weights are folded into X, and wi = 1. [In this case, the

elements of X are ∂F/∂a and ∂F/∂b, instead of 1 and x.] Vprior is appropriate for MC simulations,

since sxi and syi are set at selected values. Recall that Vprior is exact for LLS, for any data error, as

long as the wi are taken as syi-2. The present applications being NLS, we expect it to hold in the limit
 -6-

of zero data error, but to fail increasingly as the data error increases.41 A point of the MC simulations

is to assess the significance of this failure.

With commercial codes, the computations of eqn (6) may be done automatically, and a user

should ascertain whether the program uses eqn (6) or

SEV
Vpost = n (XT W X) –1 , (7)

where n is the number of statistical degrees of freedom, here n - 2. In LLS the sum of weighted,

squared residuals is c2-distributed under the normal assumptions about the model and the data. Then

the prefactor in eqn (7) should be a sampling estimate of the reduced c2, having mean 1 and variance

(2/n). We anticipate that these properties should hold also for the present MC simulations in the limit

of small data error and again ask the extent to which they fail with increasing data error.

While Vpost is the default approach to parameter uncertainty estimation in most physical

science work, its use is questionable in LS treatments of data with uncertainty in x and y. There are

many situations where the weighting can realistically be known only to within a scale factor when

fitting a single uncertain variable — e.g., when homoscedastic y values of unknown sy are

logarithmically transformed, giving s(ln y) = sy /y and requiring weighting as y2. With error in both x

and y, the user must supply weights (or ss) for all xi and yi, that are valid to within a single unknown

scale factor (C after eqn 1). This will normally require actual information about these uncertainties,

making Vprior the proper choice.

Bias under OLS treatment with constant sx.

Shukla treated the problem of inverse calibration — regression of x upon y when x is error-free

and sy is constant — giving equations for bias and variance valid to order 1/n.36 The same treatment

can be applied to OLS for constant sx and sy = 0. However, Shukla's results are expressed in terms

of the true intercept and slope, a and b, of the original regression of y upon x rather than of his actual

regression of x upon y; since here we are interested in the true parameters for the regression of y upon

x — y = a + bx — with error in x, the true slope enters as the reciprocal of Shukla's b. We thus obtain

a bias factor,
 -7-

s 2x b 2 2
fbias= ( -1), (8)
s 2ymq nq 2

from which the biases can be calculated as

bias(a)=
† fbias (a - y ) and bias(b) = fbiasb (9)

s 2x b 2
with q = 1+ . (10)
s 2ym

Here sym2 is the model variance in y, defined as

† s 2ym (n -1) = Â (yi - y ) 2 ; (11)

it represents the present counterpart to Shukla's sx2 for error-free x. [In the parlance of analysis of

variance, the total variance is†partitioned into model variance and residual variance; the latter

corresponds to our random-error variances,sx2 and sy 2.] Since these predictions are to order 1/n, they

become progressively less reliable for small n.37 Shukla's treatment also correctly predicts reduced

variance for inverse calibration; however, I will confine my variance comparisons here to numerical

illustrations.

We extend this treatment by relating sym2 from eqn (11) to the corresponding model variance

in x. Neither xi nor yi is known in an actual analysis situation with finite sx and sy , but let us assume

that the intended x-structure is known. Then

sym2 = b2 sxm2 , (12)

where sxm2 is defined for x as sym2 for y in eqn (11). Then the true slope b drops out of the

expressions for q and fbias, giving

s 2x
q =1+ 2 . (13)
sx m
2
s Ê 2 ˆ
and f bias = 2 x Á 2 - 1˜ . (14)
s x mq Ë nq ¯

These expressions make it easy to recognize when uncertainty in x is likely to be a problem and to

correct for it. Their validity for a range of assumptions about the actual errors sx and sy is tested

below through the MC simulations.

 -8-

I know of no theoretical treatment of this problem for other than constant variance in one

variable and zero in the other. However, sometimes a data transformation can be used to achieve the

required constant s from a different error structure. Thus, for example, if x has proportional error, sx
dx'
= Cx, a logarithmic transformation yields a variable with constant uncertainty: x' = ln(x), sx' = dx sx

= C. Then if the fit model is linear in x', the theory applies, though results may be affected to some

extent by nonnormality in x' from the transformation.

Correcting for parameter bias in the EV method

In calibration, the fit parameters are commonly used to estimate an unknown (x0) from one or

more measurements of its response (y0), obtained from x0 = (y0 - a)/b for a linear response function.

The EV parameter biases can be large, so for reliable estimation of x0 they require correction. This can

be done easily using eqns. (9). Since a = a + bias(a) and b = b + bias(b), the estimated true

parameters are
b
bˆ = and ^ = a + -x b fbias ,
a (15)
1 + f bias
^ is obtained by again treating -y and -x as properties of the model.
where the expression for a

The corrections to a and b imply corresponding changes to the V matrix, needed to estimate

uncertainties in the corrected parameters and in x0. The estimates of a and b are related to a and b via

a linear transformation having a 2¥2 matrix L with elements L11 = 1, L12 = g, L21 = 0, L22 = h, where

h = (1 + fbias)-1 and g = -x fbias h. The transformed V is V' = LVLT, where LT is the matrix

transpose of L;42 the elements of V' are

V11' = V11 + 2g V12 + g 2V22 ; V12' = V21' = h V12 + ghV22 ; V22' = h2V22 , (16)
^ and ^b . The uncertainty in x0 comes from two sources:
where V11' and V22' are the variances for a

the calibration function and the measurement of y0.43 The two contributions are independent, hence

additive in the variance, giving for the bias-corrected EV results,11

sx0 2 = ^b -2 [V11' + 2 x0 V12' + x02 V22' + sy 0 2] , (17)
 -9-

where the first three terms in brackets constitute the variance sf2 in the response function at x0, and the

last is the y-variance at y0 divided by the number m of measurements averaged. With increasing data

error, the nonlinearity of x0 leads to additional contributions to the rhs of eqn. (17).36

The uncertainty of measurement of y0 can be reduced by increasing m and will anyway be

independent of the manner used to fit the calibration data. Accordingly, in the MC simulations

discussed below, I will examine just the uncertainty from the calibration fit. At the level of

approximation of eqn (17), this is the uncertainty of the response function at x0 divided by the slope,

or s (x )/ ^b for the bias-corrected EV results.

f 0

Monte Carlo simulations

Simulated data are generated by adding random normal error in x and y to points on the true

curve, using methods that have been described in detail before.13,41 Here it suffices to note that the

MC precisions are governed by the standard errors from the MC statistics, e.g., sa/(N)1/2 for the

intercept a when it and its standard deviation are estimated from N analyzed data sets. The uncertainty

in sa is governed by the statistics of c2, so it is uncertain by 1 part in (2N)1/2, which is 0.224% for N =

105. Also, when the MC results are binned to yield histograms and the latter are fitted, the bin counts

are weighted in accord with Poisson statistics, meaning they are assigned uncertainties = count1/2.

The computations were performed using algorithms written in Microsoft FORTRAN and run

on PCs operating in Windows XP and Vista. My general-purpose TV code has been tested on a

number of linear and nonlinear problems in the literature. Under tight convergence criteria, in tests on

York's strongly heteroscedastic linear model,44 it yields 10-figure agreement with "exact" values as

given by Cantrell,45 and 12-figure agreement with STV.23 The parameter SEs for this model agree

with those given by Reed at his stated precision (7 and 8 digits in his footnote 15, evaluated at

calculated points).46 Also, it gives identical results for parameters and standard errors in all

mathematically equivalent ways of expressing the fit relation among the variables and parameters (e.g.,

both implicit and explicit versions of quadratic equations, and different linearized versions of the

Langmuir isotherm47). This invariance is to be expected from the manner in which STV is defined in
 -10-

terms of the variables in eqn (1). The simpler EV2 code was similarly checked for its agreement with

the TV results for the linear model.

Results and discussion

Bias and precision in estimated parameters

From Figs. 1 and 2, with constant error in both variables the parameters are biased and

nonnormal for analysis by both the TV (= EV2) and the EV (= OLS) methods. Since LLS estimators

are normal and unbiased for any sy when sx = 0, we can ask how small sx must be for these

properties to hold approximately. For this model, 105 MC data sets run with sx = 0.05 (= sy /10)

yielded histograms that were adequately fitted by Gaussians, and ensemble statistics for both the

parameters and their standard errors that agreed with predictions, for analysis by both methods. If we

reverse the axes of major and minor uncertainty, it turns out that sx = 0.25 and sy = 0.025 yield

almost identical parameter SEs; however, now the parameter estimates are clearly biased and

nonnormal. We can see why this is true by re-expressing the fit relation with reversed definitions of

the variables,
y a
x=b -b , (18)

which is nonlinear in a and b. Further, if y were error-free, b would be the reciprocal of a normal

variate, which explains its bias and nonnormality.41,48 By redefining the parameters,
By A
x= 4 -2 . (19)

we regain normality for the same numerical values of the parameters, and B has the same SE as b.

Some of these results are illustrated in Fig. 3.

Figure 4 shows the binned S/n values from the same MC runs that yielded the results in Fig. 2.

Remarkably, the values from the TV analysis are adequately fitted by the reduced c2 distribution, even

though the parameter distributions are very far from Gaussian. This result — that the S values are c2-

distributed when the parameters are very nonGaussian — has been noted before in single-uncertain-

variable NLS,41 but the degree of nonnormality displayed here in Fig. 2 greatly exceeds that in the
 -11-

previous analogous comparisons. The S/n distributions for EV analysis in Fig. 4 do not follow the c2

distribution — a general result for this method.

For the examples so far, the biases are greater for OLS analysis, but the corresponding

parameter SEs are smaller than predicted by Vprior. Results for a range of assumed y-error structures

are compared with the predictions from eqns (8) - (11) in Table 2. When sy is constant, the biases for

EV analysis are remarkably insensitive to the value of sy , even when the nominal relative SE in a

exceeds 1. The predictions are less reliable for non-constant sy , but still lead to discrepancies that are

much smaller than the parameter SE. Note that the sy values assumed for the SP tests lead to weights

spanning a dynamic range exceeding 30, which is moderate heteroscedasticity. For TV analysis, the

biases are much smaller, but are not predictable. And the sampling estimates of sa and sb typically

exceed those for EV analysis by 15-60%. Given these two properties — the predictability of the

biases and the greater precision — these tests support the earlier indication that the EV method is

better for estimating the parameters than the TV method when sx is constant.

Complementary results for proportional rather than constant error in x are given in Table 3.

Shukla's theory cannot be used to predict the EV biases in this case, which now also depend on the

error in y. The biases in both intercept and slope are noticeably smaller for the geometric x-structure

than for the arithmetic, but always still exceed those for the TV method. The EV estimates are mostly

more precise than the TV ones, but not by as much as for constant x error in Table 2. We can

conclude that the EV method is at best comparable to the TV method in this case.

As previously was noted, the EV method degenerates to OLS when sx and sy are both

constant, as then the wi (eqn 2) become constant for all i. Similarly, when syi µ sxi, the EV method

becomes simple weighted LS, with relative weights given by syi-2. In all other cases, including

proportional error in both x and y, no such simplification occurs, and the wi depend upon i and b and

must be iterated to consistency. In particular, the uncertainties in x must be known in this case, as they

contribute to the wi and thus affect the EV results. Of course the uncertainty in x must be known for

correct estimation of the parameter SEs in all cases.

 -12-

To lend specificity to the comments in the preceding paragraph, consider the analysis of a

single data set like those in the simulations that gave the third set of results in Table 2 (sx = 0.75,

sy = 1.0). Naive use of OLS in this case will give the EV solution exactly (since the weights wi are
the same for all points, no matter what the value of b). The estimated slope and intercept are enough

biased as to make the rms errors in a and b now greater than those for the TV method, so correction of

these biases will be necessary for satisfactory estimation of x0.

Bias and precision in estimating the unknown

In comparing the EV results in Tables 1 and 2 with predictions, we knew the true parameters a

and b; an analyst would have to estimate these from the observed a and b. This can be done using

eqns (15), provided one can estimate both sx and sxm for the data. How well this can be done will

depend on the nature of the study, but recall that the first of these is anyway required for proper use of

both analysis methods. Most of the constant-sx results in Table 2 were obtained with a base x

structure of 5 evenly spaced xi values from 1 to 5. These were repeated 4 times to achieve the 20-point

data sets, giving sxm2 = 2.1053, q = 1.2672 and fbias = -0.19772.

Figures 5 and 6 compare the TV and EV methods, for the model behind the third set of results

in Table 2. As anticipated, the uncorrected EV results are the poorest, as a consequence of the large

parameter biases. The bias-corrected EV method is virtually identical to the TV method in its

performance in estimating x0. Note that both methods are slightly biased in these estimates. As was

mentioned earlier, this stems from the nonlinearity of the estimator of x0, and it holds also for

conventional calibration with error-free x.36,37 For the TV and EVcor results, sx0 exceeds sf(x0)/b by

~10%.

When similar MC comparisons are made for the model behind Figs. 1 and 2, where the EV

method seemed clearly better, we see (Fig. 7) that this performance holds also for estimating y(x). In

this case, correcting for the bias actually leads to increased rms error. However, Fig. 8 shows that the

uncorrected EV method fails to estimate the unknown x0 as well as the TV method in this case, though

with the bias corrections, the EV rms error remains the smallest for all x0. The SEs in y and x0 are
 -13-

relatively large over most of the investigated range of x, and sx0 is comparable to sf(x0)/b for EVcor

but actually significantly smaller than sf(x0)/b for the TV results. Examination of the distributions of

a, b, and x0 revealed the reason for this surprising result: very nonnormal a and b but nearly normal

x0. In the effective variance, the contribution from the x uncertainty is 90% of the total in this case,

meaning that regression of x upon y should yield roughly normal B and A in eqn (19) and reciprocal

normal a and b in eqn (18). The TV analysis correctly reflects this dependence and hence yields near-

normal x0, as expected for x0 = x(y0) from eqn (19); the EV treatment, being OLS of y on x, does not.

Figure 9 extends the comparisons to the SP1 model of Table 2, where the x-error remains

constant, but y is heteroscedastic. In this case the EV method requires iteration on the wi, since

changes in b affect the different wi differently. There are two ways of accommodating the bias

corrections now — within these iterations, where they contribute to sy,eff and hence to the changing

wi, or at the conclusion of these iterations. The MC computations gave slightly better results for the

first choice, so just these results are compared in Fig. 9. Regardless, the EV estimation of x0 is now

significantly poorer than that from the TV method over all x0 but a narrow range near the weighted

average of the calibration xi values. Thus the respectable performance of the EV method in estimating

the parameters in Table 2 for this model does not extend to their use in calibration.

When both x and y are heteroscedastic, there is no theoretical method for correcting the biases,

as has been noted. However, for the geometric x-structure, results in Table 3 showed good parameter

precision and small bias for the uncorrected EV results, so we ask whether this behavior extends to the

estimation of x0. Indeed, for the geometric-x models in that table (having both constant and

proportional error in y) there was only small loss of precision (~10%) and insignificantly greater bias

in x0 for the EV method. These losses would not be important in many applications, so the EV

method can be judged acceptable in this case though still poorer than the TV method.

Reliability of predicted uncertainties

All of the illustrations so far have been for actual performance of the two fitting methods, as

determined from the MC sampling statistics for the parameters and for the calculated y(x) and x0(y0).
 -14-

The analyst working with a single data set would have to estimate the uncertainties from the V-based

expressions, so we need to know how well those predictions match the observations. Note that if this

were LLS, the Vprior-based SEs would all be the same for all simulations on a given model; and they

would agree statistically with the observed sampling estimates for all of the normal quantities (all but

x0). With error in x, there is variability in Vprior over the simulations and no guarantee of agreement

with the sampling statistics. For the purpose of the comparisons, I compute the MC rms estimates of

the SEs — the square roots of the MC average variances (e.g. eqn 17 w/o the last term in brackets, for

0 ^ ), b (^b ), and for y = f(x) and x (y ) at several

the calibration variance in x ). This is done for a (a 0 0

points over the calibration range. Since we presume to know sx and sy , the Vprior-based SEs are the

appropriate quantities.

For most of the test models, these comparisons showed consistency between predicted and

sampling estimates of the SEs within ~10%, for both the TV results and the bias-corrected EV results.

The exceptions were as follows: (1) For the 5-point model of Figs. 1 and 2, both methods

overestimated the SEs for the parameters and for sf(x) by about 20% but predicted sx0 within ~5%.

This behavior is another manifestation of the preference of this model for regression of x upon y,

making x0 nearly normal but the other quantities roughly the reciprocals of normal variates. (2) For

the 20-point proportional error models having 20% error in x and 40% in y, there were divergence

tendencies in the sampling statistics of x0 (like factor-of-10 increases in the sampling rms errors on

going from 20% error in both x and y), and these were reflected in clearly divergent Vprior-based SEs

for x0 in both methods. (3) The latter behavior occurred also for a model having 20% error in x and

sy = 1, but only 5 points. In all of these cases the TV and EVcor methods performed comparably.

When sx is small

Most results discussed up to now have involved data that are very noisy in x, y, or both. While

these conditions may prevail in some situations (e.g., some biomedical work, and methods

comparisons near the detection limit), in most analytical work uncertainty in x is more likely to be a
 -15-

nuisance than a fatal flaw. I next address the matter of when that nuisance is enough of a problem to

require special attention, in the common use of OLS to fit calibration data.

From eqns (13) and (14), the quantity that determines the significance of bias is the ratio of the

random-error variance in x to the model variance. For example, for the base 5-point arithmetic x

structure used in much of this work, suppose the uncertainty in x is about 1/10 the range of xi values, or

sx = 0.4. Then a quick computation shows that the key variance ratio will be about 0.08, leading to an
8% bias factor for large n (which is worse than small n, thanks to the inconsistency of the estimators

in this situation). Even this bias is much larger than is likely to be encountered in most routine

calibration, yet still small enough to be easily corrected.

MC simulations were run on data sets like this for a range of sy values, using OLS (hence

Vpost, eqn (7), to estimate the uncertainties). These computations confirmed that an analyst working

with such data and unaware of the uncertainty in x would still obtain uncertainties for the parameters

and for estimates of x0 that are typically within 5% of the actual rms errors. The reduced sums of

squared residuals (S/n) were also quite close to sy 2 + b 2sx2, permitting an analyst with knowledge of

sx to both correct for the biases and reliably estimate sy 2. The biases become more significant
contributors to the rms errors as the sy 2 term in S/n decreases; and when S/n ≈ b2sx2, they require

attention. Alternatively the analyst should then consider fitting x(y) instead of y(x).

Conclusion

Data having comparable uncertainty in x and y normally require special methods for least-

squares analysis. However, when the uncertainties in both variables are constant, OLS can actually

give better results in linear calibration. Both OLS and the more complex TV method yield biased

estimates of parameters and unknowns; however, the parameter biases are reliably predicted and hence

correctable for OLS, from the theory of inverse calibration — regression of x upon y when x is error-

free and y is uncertain. MC simulations demonstrate that the biases are insensitive to the uncertainty

in y as long as both uncertainties are constant. When sy varies with y, OLS can be replaced by the

effective-variance method, in which the data are weighted inversely as the effective variance, syi2 + b 2
 -16-

sxi2. The MC simulations show that the EV method can again yield more precise, bias-correctable
parameter estimates; however, in this case the good results for the parameters do not translate into

comparable performance in estimating x0 in calibration. When both x and y are heteroscedastic, the

TV method must generally be used. In one exception, the MC simulations indicate that the EV method

performs comparably when the x-structure of the data is approximately geometric and x has

uncertainty proportional to its magnitude.

In OLS calibration, the quantity that signals the need for concern about uncertainty in x is the

ratio of the random-error variance in x to the model variance, sx2/sxm2. The biases will be less than

this ratio, so either negligible (when sy 2 exceeds ~3 b2 sx2), or easily corrected. When b2sx2

exceeds ~3sy 2, the error in x dominates, and analysts who prefer to work with unbiased estimators

should then reverse the regression (at some loss of precision).

The TV method can be trusted in all circumstances but is not free of bias, particularly for noisy

data. For linear response functions, the TV solutions can be obtained easily using many standard

nonlinear LS programs, by defining the fit function to include seff in the denominator. While this

approach is rigorously correct for only linear response functions, there are indications that it is

statistically nearly equivalent to the TV method more generally, for any response function.
 -17-

References
1 J. N. Miller, Analyst, 1991, 116, 3.

2 K. Danzer and L. A. Currie, Pure Appl. Chem., 1998, 70, 993.

3 M. Meloun, J. Militky, M. Hill, and R. G. Brereton, Analyst, 2002, 127, 433.

4 A. Sayago, M. Boccio, and A. G. Asuero, Crit. Revs. Anal. Chem., 2004, 34, 39.

5 K. Baumann and H. Wätzig, J. Chromatogr. A, 1995, 700, 9.

6 S. M. Gort and R. Hoogerbrugge, Chemom. Intell. Lab. Syst., 1995, 28, 193.

7 M. Mulholland and D. B. Hibbert, J. Chromatogr. A, 1997, 762, 73.

8 E. Desimoni, Analyst, 1999, 124, 1191.

9 L. Kirkup and M. Mulholland, J. Chromatogr. A, 2004, 1029, 1.

10 N. Francois, B. Govaerts, B. Boulanger, Chemom. Intell. Lab. Syst., 2004, 74, 283.

11 J. Tellinghuisen, Analyst, 2000, 125, 1045.

12 J. Tellinghuisen, Analyst, 2005, 130, 370.

13 J. Tellinghuisen, Analyst, 2007, 132, 536.

14 Q. C. Zeng, E. Zhang, and J. Tellinghuisen, Analyst, 2008, 133, 1649.

15 B. D. Ripley and M. Thompson, Analyst, 1987, 112, 377.

16 J. Riu and F. X. Riux, J. Chemom., 1995, 9, 343.

17 W. Bremser and W. Hässelbarth, Anal. Chim. Acta, 1997, 348, 61.

18 P. J. Ogren, A. Meetze, and W. C. Duer, J. Anal. Toxicol., 2009, 33, 129.

19 W. E. Deming, Statistical Adjustment of Data, (Wiley, New York, 1938, 1943; Dover, New

York, 1964).

20 D. R. Powell and J. R. Macdonald, Computer J., 1972, 15, 148.

21 H. I. Britt and R. H. Luecke, Technometrics, 1973, 15, 233.

22 W. H. Jefferys, Astron. J., 1980, 85, 177.

23 M. Lybanon, Am. J. Phys., 1984, 52, 22.

24 M. Clutton-Brock, Technometrics, 1967, 9, 261.

25 D. R. Barker and L. M. Diana, Am. J. Phys., 1974, 42, 224.

 -18-

26 D. S. Riggs, J. A. Guarnieri, and S. Addelman, Life Sci., 1978, 22, 1305.

27 J. Orear, Am. J. Phys., 1982, 50, 912; erratum, 1984, 52, 278.

28 J. P. Chandler, Technometrics, 1972, 14, 71.

29 M. Lybanon, Am. J. Phys., 1984, 52, 276.

30 J. H. Williamson, Can. J. Phys., 1968, 46, 1845.

31 E. Di Cera, Meth. Enzymol., 1972, 210, 68.

32 A. H. Kalantar, R. I. Gelb, and J. S. Alper, Talanta, 1995, 42, 597.

33 R. G. Krutchkoff, Technometrics, 1967, 9, 425.

34 R. G. Krutchkoff, Technometrics, 1969, 11, 605.

35 J. Berkson, Technometrics, 1969, 11, 649.

36 G. K. Shukla, Technometrics, 1972, 14, 547.

37 J. Tellinghuisen, Fresenius J. Anal. Chem., 2000, 368, 585.

38 S. D. Christian and E. E. Tucker, Am. Lab., 1986, 18, 33.

39 P. R. Bevington, Data Reduction and Error Analysis for the Physical Sciences, 1969,

McGraw-Hill, New York.

40 W. H. Press, B. P. Flannery, S. A. Teukolsky, and W. T. Vetterling, Numerical Recipes, 1986,

Cambridge Univ. Press, Cambridge, U. K.

41 J. Tellinghuisen, J. Phys. Chem. A, 2000, 104, 2834.

42 W. C. Hamilton, Statistics in Physical Science: Estimation, Hypothesis Testing, and Least

Squares, 1964, Ronald Press, New York.

43 D. B. Hibbert, Analyst, 2006, 131, 1273.

44 D. York, Can. J. Phys., 1966, 44, 1079.

45 C. A. Cantrell, Atmos. Chem. Phys., 2008, 8, 5477.

46 B. C. Reed, Am. J. Phys., 1992, 60, 59.

47 J. Tellinghuisen and C. H. Bolster, J. Phys. Chem. B, 2009, 113, 6151.

48 J. Tellinghuisen, J. Phys. Chem. A, 2000, 104, 11829.

 -19-

Table 1. Summary of abbreviations and notation, in approximate order of occurrence.

Abbreviations Notation

LS least squares x, y independent and dependent variables

OLS ordinary least squares x0, y0 unknown and its response
LLS linear least squares xi, yi, Xi, Yi observed (lc) and adjusted values

NLS nonlinear least squares dxi, dyi , wi LS residuals and weight for ith point

TV total variance S sum of weighted, squared residuals

EV effective variance sx2, sy 2 (s2) true and sampling estimates of variance

MC Monte Carlo sxm2, sym2 model variances in x and y

SE standard error a, b, a, b intercept, slope and their true values

srms root-mean-square error, (s2 + bias2)1/2

n, p numbers of points and fit parameters

Vprior , Vpost covariance matrices, p¥p

W weight matrix, n¥n

X Jacobian matrix, n¥p

 -20-

Table 2. Comparison of predicted and Monte Carlo results for linear model having intercept (a) and
slope (b) = 1, and error in both x (sx = 0.75) and y. Each MC run comprised 105 20-point data sets (4

points at each of 5 x values), analyzed by the EV method (first of each pair of lines, = OLS for
constant sy ) and the TV method (second of each pair). Predicted EV values of a and b are from Eqs.

(8) - (11). The x values were 1, 2, 3, 4, 5, except for the last results (SP2), obtained with a geometric x
structure — 0.2, 0.5, 1, 2, 5. Results are shown for constant sy (first four), proportional error in y

(next three), and a special y-error structure — 4.4, 0.8, 1.8, 3, 0.2 — used for SP1 and SP2. For
results marked with an asterisk, the analyses employed calculated values (Yi) to assign weights.

sy a (MC) a (pred) sa sa (MC) b (MC) b (pred) sb sb (MC)

0.0 1.5889 1.5932 0.3933 0.2758 0.8036 0.8023 0.1186 0.0803

0.9543 0.4155 1.0151 0.1263

0.5 1.5886 1.5932 0.4727 0.3676 0.8037 0.8023 0.1425 0.1076

0.9501 0.5114 1.0165 0.1562

1.0 1.5883 1.5932 0.6555 0.5586 0.8038 0.8023 0.1976 0.1644

0.9454 0.7286 1.0181 0.2236

2.0 1.5878 1.5932 1.1201 1.0085 0.8040 0.8023 0.3377 0.2975

0.9415 1.2668 1.0193 0.3903

10% 1.5439 1.5932 0.4255 0.3058 0.7977 0.8023 0.1364 0.1016

0.9501 0.4600 1.0164 0.1486

25% 1.4902 1.5932 0.5423 0.4238 0.7820 0.8023 0.1978 0.1647

0.9378 0.6106 1.0202 0.2240

25%* 1.6205 1.5932 0.5423 0.4506 0.7911 0.8023 0.1978 0.1642

1.0980 0.7135 1.0967 0.2599
SP1 1.7674 1.5932 0.8632 0.8229 0.8130 0.8023 0.2105 0.1826
0.8824 0.9918 1.0358 0.2526
SP2 1.2810 1.2420 0.5231 0.5127 0.9172 0.8609 0.1370 0.1259
0.9704 0.5402 1.0131 0.1455
 -21-

Table 3. Comparison of Monte Carlo results (105 20-point data sets) for linear model having intercept
(a) and slope (b) = 1, and error in both x (sx = 0.2 x) and y. In each pair of results lines, the first come

from analysis with the EV method, the second with TV. An arithmetic x-structure — 1, 2, 3, 4, 5 —

was used for the first six sets, while a geometric structure — 0.2, 0.5, 1, 2, 5 — was used for the rest.

Results are shown for constant and proportional error in y.

sy a (MC) sa sa (MC) b (MC) sb sb (MC)

0.0 1.1926 0.1541 0.1691 0.9509 0.0834 0.0844

0.9881 0.1580 1.0073 0.0861

0.5 1.2937 0.3309 0.3248 0.9128 0.1253 0.1165

0.9852 0.3412 1.0087 0.1317

1.0 1.3872 0.5716 0.5524 0.8771 0.1898 0.1777

0.9808 0.6009 1.0108 0.2041

2.0 1.4450 1.0763 1.0207 0.8553 0.3346 0.3121

0.9777 1.1554 1.0126 0.3686

20% 1.1958 0.3135 0.2963 0.9120 0.1452 0.1362

0.9876 0.3266 1.0078 0.1543

40% 1.2100 0.5640 0.5399 0.8970 0.2505 0.2373

0.9871 0.5917 1.0082 0.2686

0.0 1.0194 0.0257 0.0322 1.0077 0.0632 0.0743

0.9987 0.0260 1.0041 0.0644

1.0 1.0611 0.3296 0.3290 0.9761 0.1650 0.1641

0.9919 0.3336 1.0108 0.1721

2.0 1.0824 0.6398 0.6365 0.9603 0.2782 0.2764

0.9901 0.6507 1.0126 0.2921

20% 1.0259 0.1118 0.1107 0.9680 0.1280 0.1264

0.9978 0.1129 1.0054 0.1323

40% 1.0310 0.2136 0.2115 0.9550 0.2227 0.2189

0.9974 0.2162 1.0061 0.2311
 -22-

Figure Captions

Figure 1: Bias in parameter estimates and rms errors for a 5-point straight line model having true

intercept =1 and slope = 2, x values at 1, 2, 3, 4, and 5, and sy = 0.5. Results obtained

from 105 simulated data sets for each sx, analyzed by minimizing STV (= SEV2 ) and

SEV (OLS). For Dsrms the reference is the covariance-matrix predicted standard error

(the same for all methods). The curves in the lower plot are fits to the function C sx2,

which is the expected dependence for small bias.41

Figure 2: Histograms for estimated values of the slope b in model of Fig. 1 at sx = 0.75. The

predicted SE here is sb = 0.5000; the MC sampling statistics yield sb = 0.448 and

0.812 for the EV (OLS) and TV methods, respectively.

Figure 3: Histograms for b and B (eqn 19) from fits of data having sx = 0.25 and sy = 0.025, for

TV analysis of 105 MC data sets. The illustrated weighted Gaussian fit for B yields a

reasonable value of 28.2 for c2 (31 points); that for b (curve not shown) gives 3160.

The biases for a and b are -0.042 and 0.014, respectively for TV analysis, 0.072 and

-0.024 for EV (OLS) analysis.

Figure 4: Histograms of S/n values obtained in same MC calculations that yielded distributions

for b in Fig. 2. The illustrated curve for TV is a weighted LS fit to the reduced c2

distribution, which yielded n = 3.002 (13) and c2 = 44.2 (47 points).

Figure 5: Monte Carlo results (105 data sets) for bias and precision in the estimation of y = a +

bx for the third model of Table 2 (sx = 0.75, sy = 1.0, a = b = 1, n = 20). rms errors

shown for the uncorrected EV (OLS) results are (sy 2 + bias2)1/2.

Figure 6: Bias and precision in the estimation of x0 in the MC simulations of Fig. 5, with

quantities displayed as functions of the true x0.

Figure 7: Bias and precision in the estimation of y for the model of Fig. 1 (105 MC data sets).

Figure 8: Bias and precision in the estimation of x0 in the MC simulations of Fig. 7.

Figure 9: Bias and precision in the estimation of x0 from 105 MC simulations for the SP1 model

in Table 2 (sx = 0.75, special heteroscedastic y structure).

 -23-

70

60

50

40 a (EV)
s (%)

b (EV)
30 a (TV)
b (TV)
D

20

10

0.4
bias

0.0

-0.4

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

sx

Figure 1: Bias in parameter estimates and rms errors for a 5-point straight line model having true
intercept =1 and slope = 2, x values at 1, 2, 3, 4, and 5, and sy = 0.5. Results obtained
from 105 simulated data sets for each sx, analyzed by minimizing STV (= SEV2 ) and
SEV (OLS). For Dsrms the reference is the covariance-matrix predicted standard error
(the same for all methods). The curves in the lower plot are fits to the function C sx2,
which is the expected dependence for small bias.41
 -24-

14000

12000
EV
TV
10000

8000
count

6000

4000

2000

0
1.0 2.0 3.0 4.0 5.0
b

Figure 2: Histograms for estimated values of the slope b in model of Fig. 1 at sx = 0.75. The
predicted SE here is sb = 0.5000; the MC sampling statistics yield sb = 0.448 and
0.812 for the EV (OLS) and TV methods, respectively.

10000
b
8000 B

6000
Count

4000

2000

0
1.4 1.6 1.8 2.0 2.2 2.4 2.6
b

Figure 3: Histograms for b and B (Eq. 15) from fits of data having sx = 0.25 and sy = 0.025, for
TV analysis of 105 MC data sets. The illustrated weighted Gaussian fit for B yields a
reasonable value of 28.2 for c2 (31 points); that for b (curve not shown) gives 3160.
The biases for a and b are -0.042 and 0.014, respectively for TV analysis, 0.072 and
-0.024 for EV (OLS) analysis.
 -25-

8000

7000 EV
TV
6000

5000
Count
4000

3000

2000

1000

0
0 1 2 3 4
S/n

Figure 4: Histograms of S/n values obtained in same MC calculations that yielded distributions

for b in Fig. 2. The illustrated curve for TV is a weighted LS fit to the reduced c2

distribution, which yielded n = 3.002 (13) and c2 = 44.2 (47 points).

 -26-

1.1

TV
0.9 EVcor
EV
EV - rms
0.7
sy
0.5

a
0.3

0.8
y bias (calc - true)

0.4

0.0

-0.4
b
-0.8
-1 0 1 2 3 4 5 6
x

Figure 5: Monte Carlo results (105 data sets) for bias and precision in the estimation of y = a +

bx for the third model of Table 2 (sx = 0.75, sy = 1.0, a = b = 1, n = 20). rms errors

shown for the uncorrected EV (OLS) results are (sy 2 + bias2)1/2.

 -27-

1.6
TV
EVcor
EV
1.2 EV-rms
sx0
0.8

a
0.4

1.0
b
x0 bias (calc - true)

0.5

0.0

-0.5

-1.0

-1 0 1 2 3 4 5 6
x0

Figure 6: Bias and precision in the estimation of x0 in the MC simulations of Fig. 5, with

quantities displayed as functions of the true x0.

 -28-

3.5

3.0 TV
EVcor
2.5 EV
EV-rms
sy 2.0

1.5

1.0 a

0.8
y bias (calc - true)

b
0.4

0.0

-0.4

-0.8
-1 0 1 2 3 4 5 6
x

Figure 7: Bias and precision in the estimation of y for the model of Fig. 1 (105 MC data sets).
 -29-

1.4
TV
1.2 EVcor
EV
1.0
sx0 EV-rms

0.8

0.6

0.4 a

0.8
b
x0 bias (calc - true)

0.4

0.0

-0.4

-0.8

-1 0 1 2 3 4 5 6
x0

Figure 8: Bias and precision in the estimation of x0 in the MC simulations of Fig. 7.

 -30-

2.4
TV
2.0 EVcor

1.6
sx0
1.2

0.8
a
0.4

0.0
b
x0 bias (calc - true)

-0.1

-0.2

-0.3

-1 0 1 2 3 4 5 6
x0

Figure 9: Bias and precision in the estimation of x0 from 105 MC simulations for the SP1 model

in Table 2 (sx = 0.75, special heteroscedastic y structure).

View publication stats