Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to content

Commit 83c94a8

Browse files
committed
Another try at making numeric MODULO operator produce the right answer.
Although it was now using the right equation, it was making bogus choices of the precision to compute intermediate results to. I'm not sure this is really right even yet, but it's better than before ...
1 parent ca224d2 commit 83c94a8

File tree

1 file changed

+18
-6
lines changed

1 file changed

+18
-6
lines changed

src/backend/utils/adt/numeric.c

Lines changed: 18 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
*
66
* 1998 Jan Wieck
77
*
8-
* $Header: /cvsroot/pgsql/src/backend/utils/adt/numeric.c,v 1.39 2001/03/22 06:16:17 momjian Exp $
8+
* $Header: /cvsroot/pgsql/src/backend/utils/adt/numeric.c,v 1.40 2001/04/14 02:10:57 tgl Exp $
99
*
1010
* ----------
1111
*/
@@ -1134,7 +1134,6 @@ numeric_mod(PG_FUNCTION_ARGS)
11341134

11351135
mod_var(&arg1, &arg2, &result);
11361136

1137-
result.dscale = result.rscale;
11381137
res = make_result(&result);
11391138

11401139
free_var(&result);
@@ -3281,29 +3280,42 @@ mod_var(NumericVar *var1, NumericVar *var2, NumericVar *result)
32813280
{
32823281
NumericVar tmp;
32833282
int save_global_rscale;
3283+
int div_dscale;
32843284

32853285
init_var(&tmp);
32863286

32873287
/* ---------
32883288
* We do this using the equation
32893289
* mod(x,y) = x - trunc(x/y)*y
3290-
* We fiddle a bit with global_rscale to control result precision.
3290+
* We set global_rscale the same way numeric_div and numeric_mul do
3291+
* to get the right answer from the equation. The final result,
3292+
* however, need not be displayed to more precision than the inputs.
32913293
* ----------
32923294
*/
32933295
save_global_rscale = global_rscale;
3294-
global_rscale = var2->rscale + 2;
3296+
3297+
div_dscale = MAX(var1->dscale + var2->dscale, NUMERIC_MIN_DISPLAY_SCALE);
3298+
div_dscale = MIN(div_dscale, NUMERIC_MAX_DISPLAY_SCALE);
3299+
global_rscale = MAX(var1->rscale + var2->rscale,
3300+
NUMERIC_MIN_RESULT_SCALE);
3301+
global_rscale = MAX(global_rscale, div_dscale + 4);
3302+
global_rscale = MIN(global_rscale, NUMERIC_MAX_RESULT_SCALE);
32953303

32963304
div_var(var1, var2, &tmp);
32973305

3306+
tmp.dscale = div_dscale;
3307+
32983308
/* do trunc() by forgetting digits to the right of the decimal point */
32993309
tmp.ndigits = MAX(0, MIN(tmp.ndigits, tmp.weight + 1));
3300-
tmp.rscale = var2->rscale;
33013310

3302-
global_rscale = var2->rscale;
3311+
global_rscale = var2->rscale + tmp.rscale;
3312+
33033313
mul_var(var2, &tmp, &tmp);
33043314

33053315
sub_var(var1, &tmp, result);
33063316

3317+
result->dscale = MAX(var1->dscale, var2->dscale);
3318+
33073319
global_rscale = save_global_rscale;
33083320
free_var(&tmp);
33093321
}

0 commit comments

Comments
 (0)