Guard against incoming rowcount estimate of NaN in cost_mergejoin().

Although rowcount estimates really ought not be NaN, a bug elsewhere
could perhaps result in that, and that would cause Assert failure in
cost_mergejoin, which I believe to be the explanation for bug #5977 from
Anton Kuznetsov.  Seems like a good idea to expend a couple more cycles
to prevent that, even though the real bug is elsewhere.  Not back-patching,
though, because we don't encourage running production systems with
Asserts on.

diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index e200dcf4728c3d9a9b57c53689abd1e5c9a6bba4..d3455228660e43962fe451eab03e310947c66065 100644 (file)
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1704,10 +1704,10 @@ cost_mergejoin(MergePath *path, PlannerInfo *root, SpecialJoinInfo *sjinfo)
                innerendsel;
    Path        sort_path;      /* dummy for result of cost_sort */
 
-   /* Protect some assumptions below that rowcounts aren't zero */
-   if (outer_path_rows <= 0)
+   /* Protect some assumptions below that rowcounts aren't zero or NaN */
+   if (outer_path_rows <= 0 || isnan(outer_path_rows))
        outer_path_rows = 1;
-   if (inner_path_rows <= 0)
+   if (inner_path_rows <= 0 || isnan(inner_path_rows))
        inner_path_rows = 1;
 
    if (!enable_mergejoin)