Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to content

Commit 2f55716

Browse files
committed
Avoid buffer bloat in libpq when server is consistently faster than client.
If the server sends a long stream of data, and the server + network are consistently fast enough to force the recv() loop in pqReadData() to iterate until libpq's input buffer is full, then upon processing the last incomplete message in each bufferload we'd usually double the buffer size, due to supposing that we didn't have enough room in the buffer to finish collecting that message. After filling the newly-enlarged buffer, the cycle repeats, eventually resulting in an out-of-memory situation (which would be reported misleadingly as "lost synchronization with server"). Of course, we should not enlarge the buffer unless we still need room after discarding already-processed messages. This bug dates back quite a long time: pqParseInput3 has had the behavior since perhaps 2003, getCopyDataMessage at least since commit 70066eb in 2008. Probably the reason it's not been isolated before is that in common environments the recv() loop would always be faster than the server (if on the same machine) or faster than the network (if not); or at least it wouldn't be slower consistently enough to let the buffer ramp up to a problematic size. The reported cases involve Windows, which perhaps has different timing behavior than other platforms. Per bug #7914 from Shin-ichi Morita, though this is different from his proposed solution. Back-patch to all supported branches.
1 parent be75581 commit 2f55716

File tree

1 file changed

+32
-0
lines changed

1 file changed

+32
-0
lines changed

src/interfaces/libpq/fe-misc.c

+32
Original file line numberDiff line numberDiff line change
@@ -351,6 +351,7 @@ pqCheckOutBufferSpace(size_t bytes_needed, PGconn *conn)
351351
int newsize = conn->outBufSize;
352352
char *newbuf;
353353

354+
/* Quick exit if we have enough space */
354355
if (bytes_needed <= (size_t) newsize)
355356
return 0;
356357

@@ -414,6 +415,37 @@ pqCheckInBufferSpace(size_t bytes_needed, PGconn *conn)
414415
int newsize = conn->inBufSize;
415416
char *newbuf;
416417

418+
/* Quick exit if we have enough space */
419+
if (bytes_needed <= (size_t) newsize)
420+
return 0;
421+
422+
/*
423+
* Before concluding that we need to enlarge the buffer, left-justify
424+
* whatever is in it and recheck. The caller's value of bytes_needed
425+
* includes any data to the left of inStart, but we can delete that in
426+
* preference to enlarging the buffer. It's slightly ugly to have this
427+
* function do this, but it's better than making callers worry about it.
428+
*/
429+
bytes_needed -= conn->inStart;
430+
431+
if (conn->inStart < conn->inEnd)
432+
{
433+
if (conn->inStart > 0)
434+
{
435+
memmove(conn->inBuffer, conn->inBuffer + conn->inStart,
436+
conn->inEnd - conn->inStart);
437+
conn->inEnd -= conn->inStart;
438+
conn->inCursor -= conn->inStart;
439+
conn->inStart = 0;
440+
}
441+
}
442+
else
443+
{
444+
/* buffer is logically empty, reset it */
445+
conn->inStart = conn->inCursor = conn->inEnd = 0;
446+
}
447+
448+
/* Recheck whether we have enough space */
417449
if (bytes_needed <= (size_t) newsize)
418450
return 0;
419451

0 commit comments

Comments
 (0)