Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to content

Commit 8e10405

Browse files
committed
Avoid unnecessary out-of-memory errors during encoding conversion.
Encoding conversion uses the very simplistic rule that the output can't be more than 4X longer than the input, and palloc's a buffer of that size. This results in failure to convert any string longer than 1/4 GB, which is becoming an annoying limitation. As a band-aid to improve matters, allow the allocated output buffer size to exceed 1GB. We still insist that the final result fit into MaxAllocSize (1GB), though. Perhaps it'd be safe to relax that restriction, but it'd require close analysis of all callers, which is daunting (not least because external modules might call these functions). For the moment, this should allow a 2X to 4X improvement in the longest string we can convert, which is a useful gain in return for quite a simple patch. Also, once we have successfully converted a long string, repalloc the output down to the actual string length, returning the excess to the malloc pool. This seems worth doing since we can usually expect to give back several MB if we take this path at all. This still leaves much to be desired, most notably that the assumption that MAX_CONVERSION_GROWTH == 4 is very fragile, and yet we have no guard code verifying that the output buffer isn't overrun. Fixing that would require significant changes in the encoding conversion APIs, so it'll have to wait for some other day. The present patch seems safely back-patchable, so patch all supported branches. Alvaro Herrera and Tom Lane Discussion: https://postgr.es/m/20190816181418.GA898@alvherre.pgsql Discussion: https://postgr.es/m/3614.1569359690@sss.pgh.pa.us
1 parent c477f3e commit 8e10405

File tree

1 file changed

+56
-6
lines changed

1 file changed

+56
-6
lines changed

src/backend/utils/mb/mbutils.c

+56-6
Original file line numberDiff line numberDiff line change
@@ -349,23 +349,51 @@ pg_do_encoding_conversion(unsigned char *src, int len,
349349
pg_encoding_to_char(dest_encoding))));
350350

351351
/*
352-
* Allocate space for conversion result, being wary of integer overflow
352+
* Allocate space for conversion result, being wary of integer overflow.
353+
*
354+
* len * MAX_CONVERSION_GROWTH is typically a vast overestimate of the
355+
* required space, so it might exceed MaxAllocSize even though the result
356+
* would actually fit. We do not want to hand back a result string that
357+
* exceeds MaxAllocSize, because callers might not cope gracefully --- but
358+
* if we just allocate more than that, and don't use it, that's fine.
353359
*/
354-
if ((Size) len >= (MaxAllocSize / (Size) MAX_CONVERSION_GROWTH))
360+
if ((Size) len >= (MaxAllocHugeSize / (Size) MAX_CONVERSION_GROWTH))
355361
ereport(ERROR,
356362
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
357363
errmsg("out of memory"),
358364
errdetail("String of %d bytes is too long for encoding conversion.",
359365
len)));
360366

361-
result = palloc(len * MAX_CONVERSION_GROWTH + 1);
367+
result = (unsigned char *)
368+
MemoryContextAllocHuge(CurrentMemoryContext,
369+
(Size) len * MAX_CONVERSION_GROWTH + 1);
362370

363371
OidFunctionCall5(proc,
364372
Int32GetDatum(src_encoding),
365373
Int32GetDatum(dest_encoding),
366374
CStringGetDatum(src),
367375
CStringGetDatum(result),
368376
Int32GetDatum(len));
377+
378+
/*
379+
* If the result is large, it's worth repalloc'ing to release any extra
380+
* space we asked for. The cutoff here is somewhat arbitrary, but we
381+
* *must* check when len * MAX_CONVERSION_GROWTH exceeds MaxAllocSize.
382+
*/
383+
if (len > 1000000)
384+
{
385+
Size resultlen = strlen((char *) result);
386+
387+
if (resultlen >= MaxAllocSize)
388+
ereport(ERROR,
389+
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
390+
errmsg("out of memory"),
391+
errdetail("String of %d bytes is too long for encoding conversion.",
392+
len)));
393+
394+
result = (unsigned char *) repalloc(result, resultlen + 1);
395+
}
396+
369397
return result;
370398
}
371399

@@ -682,23 +710,45 @@ perform_default_encoding_conversion(const char *src, int len,
682710
return unconstify(char *, src);
683711

684712
/*
685-
* Allocate space for conversion result, being wary of integer overflow
713+
* Allocate space for conversion result, being wary of integer overflow.
714+
* See comments in pg_do_encoding_conversion.
686715
*/
687-
if ((Size) len >= (MaxAllocSize / (Size) MAX_CONVERSION_GROWTH))
716+
if ((Size) len >= (MaxAllocHugeSize / (Size) MAX_CONVERSION_GROWTH))
688717
ereport(ERROR,
689718
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
690719
errmsg("out of memory"),
691720
errdetail("String of %d bytes is too long for encoding conversion.",
692721
len)));
693722

694-
result = palloc(len * MAX_CONVERSION_GROWTH + 1);
723+
result = (char *)
724+
MemoryContextAllocHuge(CurrentMemoryContext,
725+
(Size) len * MAX_CONVERSION_GROWTH + 1);
695726

696727
FunctionCall5(flinfo,
697728
Int32GetDatum(src_encoding),
698729
Int32GetDatum(dest_encoding),
699730
CStringGetDatum(src),
700731
CStringGetDatum(result),
701732
Int32GetDatum(len));
733+
734+
/*
735+
* Release extra space if there might be a lot --- see comments in
736+
* pg_do_encoding_conversion.
737+
*/
738+
if (len > 1000000)
739+
{
740+
Size resultlen = strlen(result);
741+
742+
if (resultlen >= MaxAllocSize)
743+
ereport(ERROR,
744+
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
745+
errmsg("out of memory"),
746+
errdetail("String of %d bytes is too long for encoding conversion.",
747+
len)));
748+
749+
result = (char *) repalloc(result, resultlen + 1);
750+
}
751+
702752
return result;
703753
}
704754

0 commit comments

Comments
 (0)