summary | shortlog | log | commit | commitdiff | tree
raw | patch | inline | side by side (parent: e2933a6)
raw | patch | inline | side by side (parent: e2933a6)
| author | Tom Lane <tgl@sss.pgh.pa.us> | |
| Mon, 21 Nov 2022 16:59:29 +0000 (11:59 -0500) | ||
| committer | Tom Lane <tgl@sss.pgh.pa.us> | |
| Mon, 21 Nov 2022 16:59:29 +0000 (11:59 -0500) |
The postmaster normally sends SIGQUIT to force-terminate its
child processes after a child crash or immediate-stop request.
If that doesn't result in child exit within a few seconds,
we follow it up with SIGKILL. This patch provides GUC flags
that allow either of these signals to be replaced with SIGABRT.
On typically-configured Unix systems, that will result in a
core dump being produced for each such child. This can be
useful for debugging problems, although it's not something you'd
want to have on in production due to the risk of disk space
bloat from lots of core files.
The old postmaster -T switch, which sent SIGSTOP in place of
SIGQUIT, is changed to be the same as send_abort_for_crash.
As far as I can tell from the code comments, the intent of
that switch was just to block things for long enough to force
core dumps manually, which seems like an unnecessary extra step.
(Maybe at the time, there was no way to get most kernels to
produce core files with per-PID names, requiring manual core
file renaming after each one. But now it's surely the hard way.)
I also took the opportunity to remove the old postmaster -n
(skip shmem reinit) switch, which hasn't actually done anything
in decades, though the documentation still claimed it did.
Discussion: https://postgr.es/m/2251016.1668797294@sss.pgh.pa.us
child processes after a child crash or immediate-stop request.
If that doesn't result in child exit within a few seconds,
we follow it up with SIGKILL. This patch provides GUC flags
that allow either of these signals to be replaced with SIGABRT.
On typically-configured Unix systems, that will result in a
core dump being produced for each such child. This can be
useful for debugging problems, although it's not something you'd
want to have on in production due to the risk of disk space
bloat from lots of core files.
The old postmaster -T switch, which sent SIGSTOP in place of
SIGQUIT, is changed to be the same as send_abort_for_crash.
As far as I can tell from the code comments, the intent of
that switch was just to block things for long enough to force
core dumps manually, which seems like an unnecessary extra step.
(Maybe at the time, there was no way to get most kernels to
produce core files with per-PID names, requiring manual core
file renaming after each one. But now it's surely the hard way.)
I also took the opportunity to remove the old postmaster -n
(skip shmem reinit) switch, which hasn't actually done anything
in decades, though the documentation still claimed it did.
Discussion: https://postgr.es/m/2251016.1668797294@sss.pgh.pa.us
index bd50ea8e480896403250752e1e290130e9f3ce2c..24b1624bad1ae6a6ec213ed78745cc5853c26b18 100644 (file)
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
</listitem>
</varlistentry>
+ <varlistentry id="guc-send-abort-for-crash" xreflabel="send_abort_for_crash">
+ <term><varname>send_abort_for_crash</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>send_abort_for_crash</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ By default, after a backend crash the postmaster will stop remaining
+ child processes by sending them <systemitem>SIGQUIT</systemitem>
+ signals, which permits them to exit more-or-less gracefully. When
+ this option is set to <literal>on</literal>,
+ <systemitem>SIGABRT</systemitem> is sent instead. That normally
+ results in production of a core dump file for each such child
+ process.
+ This can be handy for investigating the states of other processes
+ after a crash. It can also consume lots of disk space in the event
+ of repeated crashes, so do not enable this on systems you are not
+ monitoring carefully.
+ Beware that no support exists for cleaning up the core file(s)
+ automatically.
+ This parameter can only be set in
+ the <filename>postgresql.conf</filename> file or on the server
+ command line.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="guc-send-abort-for-kill" xreflabel="send_abort_for_kill">
+ <term><varname>send_abort_for_kill</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>send_abort_for_kill</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ By default, after attempting to stop a child process with
+ <systemitem>SIGQUIT</systemitem>, the postmaster will wait five
+ seconds and then send <systemitem>SIGKILL</systemitem> to force
+ immediate termination. When this option is set
+ to <literal>on</literal>, <systemitem>SIGABRT</systemitem> is sent
+ instead of <systemitem>SIGKILL</systemitem>. That normally results
+ in production of a core dump file for each such child process.
+ This can be handy for investigating the states
+ of <quote>stuck</quote> child processes. It can also consume lots
+ of disk space in the event of repeated crashes, so do not enable
+ this on systems you are not monitoring carefully.
+ Beware that no support exists for cleaning up the core file(s)
+ automatically.
+ This parameter can only be set in
+ the <filename>postgresql.conf</filename> file or on the server
+ command line.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect1>
<sect1 id="runtime-config-short">
index 55a3f6c69d1f489271b5924f4afb171e3999b392..b13a16a117fd7fac709e02d938c11228f4409702 100644 (file)
</listitem>
</varlistentry>
- <varlistentry>
- <term><option>-n</option></term>
- <listitem>
- <para>
- This option is for debugging problems that cause a server
- process to die abnormally. The ordinary strategy in this
- situation is to notify all other server processes that they
- must terminate and then reinitialize the shared memory and
- semaphores. This is because an errant server process could
- have corrupted some shared state before terminating. This
- option specifies that <command>postgres</command> will
- not reinitialize shared data structures. A knowledgeable
- system programmer can then use a debugger to examine shared
- memory and semaphore state.
- </para>
- </listitem>
- </varlistentry>
-
<varlistentry>
<term><option>-O</option></term>
<listitem>
This option is for debugging problems that cause a server
process to die abnormally. The ordinary strategy in this
situation is to notify all other server processes that they
- must terminate and then reinitialize the shared memory and
- semaphores. This is because an errant server process could
- have corrupted some shared state before terminating. This
- option specifies that <command>postgres</command> will
- stop all other server processes by sending the signal
- <literal>SIGSTOP</literal>, but will not cause them to
- terminate. This permits system programmers to collect core
- dumps from all server processes by hand.
+ must terminate, by sending them <systemitem>SIGQUIT</systemitem>
+ signals. With this option, <systemitem>SIGABRT</systemitem>
+ will be sent instead, resulting in production of core dump files.
</para>
</listitem>
</varlistentry>
index f5da4260a131eb6feb8b089f55d05231768689f3..67f556b4dd81786058217963b28424fd3ea12da4 100644 (file)
--- a/src/backend/main/main.c
+++ b/src/backend/main/main.c
printf(_("\nDeveloper options:\n"));
printf(_(" -f s|i|o|b|t|n|m|h forbid use of some plan types\n"));
- printf(_(" -n do not reinitialize shared memory after abnormal exit\n"));
printf(_(" -O allow system table structure changes\n"));
printf(_(" -P disable system indexes\n"));
printf(_(" -t pa|pl|ex show timings after each query\n"));
- printf(_(" -T send SIGSTOP to all backend processes if one dies\n"));
+ printf(_(" -T send SIGABRT to all backend processes if one dies\n"));
printf(_(" -W NUM wait NUM seconds to allow attach from a debugger\n"));
printf(_("\nOptions for single-user mode:\n"));
index 1da5752047fcac32cfb79e2a47f047823dc899c3..c83cc8cc6cd3e557e7f23a0f344443332148a935 100644 (file)
#define MAXLISTEN 64
static pgsocket ListenSocket[MAXLISTEN];
-/*
- * These globals control the behavior of the postmaster in case some
- * backend dumps core. Normally, it kills all peers of the dead backend
- * and reinitializes shared memory. By specifying -s or -n, we can have
- * the postmaster stop (rather than kill) peers and not reinitialize
- * shared data structures. (Reinit is currently dead code, though.)
- */
-static bool Reinit = true;
-static int SendStop = false;
-
/* still more option variables */
bool EnableSSL = false;
char *bonjour_name;
bool restart_after_crash = true;
bool remove_temp_files_after_crash = true;
+bool send_abort_for_crash = false;
+bool send_abort_for_kill = false;
/* PIDs of special child processes; 0 when not running */
static pid_t StartupPID = 0,
static CAC_state canAcceptConnections(int backend_type);
static bool RandomCancelKey(int32 *cancel_key);
static void signal_child(pid_t pid, int signal);
+static void sigquit_child(pid_t pid);
static bool SignalSomeChildren(int signal, int target);
static void TerminateChildren(int signal);
* tcop/postgres.c (the option sets should not conflict) and with the
* common help() function in main/main.c.
*/
- while ((opt = getopt(argc, argv, "B:bc:C:D:d:EeFf:h:ijk:lN:nOPp:r:S:sTt:W:-:")) != -1)
+ while ((opt = getopt(argc, argv, "B:bc:C:D:d:EeFf:h:ijk:lN:OPp:r:S:sTt:W:-:")) != -1)
{
switch (opt)
{
SetConfigOption("max_connections", optarg, PGC_POSTMASTER, PGC_S_ARGV);
break;
- case 'n':
- /* Don't reinit shared mem after abnormal exit */
- Reinit = false;
- break;
-
case 'O':
SetConfigOption("allow_system_table_mods", "true", PGC_POSTMASTER, PGC_S_ARGV);
break;
case 'T':
/*
- * In the event that some backend dumps core, send SIGSTOP,
- * rather than SIGQUIT, to all its peers. This lets the wily
- * post_hacker collect core dumps from everyone.
+ * This option used to be defined as sending SIGSTOP after a
+ * backend crash, but sending SIGABRT seems more useful.
*/
- SendStop = true;
+ SetConfigOption("send_abort_for_crash", "true", PGC_POSTMASTER, PGC_S_ARGV);
break;
case 't':
/*
* If we already sent SIGQUIT to children and they are slow to shut
- * down, it's time to send them SIGKILL. This doesn't happen
- * normally, but under certain conditions backends can get stuck while
- * shutting down. This is a last measure to get them unwedged.
+ * down, it's time to send them SIGKILL (or SIGABRT if requested).
+ * This doesn't happen normally, but under certain conditions backends
+ * can get stuck while shutting down. This is a last measure to get
+ * them unwedged.
*
* Note we also do this during recovery from a process crash.
*/
- if ((Shutdown >= ImmediateShutdown || (FatalError && !SendStop)) &&
+ if ((Shutdown >= ImmediateShutdown || FatalError) &&
AbortStartTime != 0 &&
(now - AbortStartTime) >= SIGKILL_CHILDREN_AFTER_SECS)
{
/* We were gentle with them before. Not anymore */
ereport(LOG,
- (errmsg("issuing SIGKILL to recalcitrant children")));
- TerminateChildren(SIGKILL);
+ /* translator: %s is SIGKILL or SIGABRT */
+ (errmsg("issuing %s to recalcitrant children",
+ send_abort_for_kill ? "SIGABRT" : "SIGKILL")));
+ TerminateChildren(send_abort_for_kill ? SIGABRT : SIGKILL);
/* reset flag so we don't SIGKILL again */
AbortStartTime = 0;
}
#endif
/* tell children to shut down ASAP */
+ /* (note we don't apply send_abort_for_crash here) */
SetQuitSignalReason(PMQUIT_FOR_STOP);
TerminateChildren(SIGQUIT);
pmState = PM_WAIT_BACKENDS;
/*
* This worker is still alive. Unless we did so already, tell it
* to commit hara-kiri.
- *
- * SIGQUIT is the special signal that says exit without proc_exit
- * and let the user know what's going on. But if SendStop is set
- * (-T on command line), then we send SIGSTOP instead, so that we
- * can get core dumps from all backends by hand.
*/
if (take_action)
- {
- ereport(DEBUG2,
- (errmsg_internal("sending %s to process %d",
- (SendStop ? "SIGSTOP" : "SIGQUIT"),
- (int) rw->rw_pid)));
- signal_child(rw->rw_pid, (SendStop ? SIGSTOP : SIGQUIT));
- }
+ sigquit_child(rw->rw_pid);
}
}
* This backend is still alive. Unless we did so already, tell it
* to commit hara-kiri.
*
- * SIGQUIT is the special signal that says exit without proc_exit
- * and let the user know what's going on. But if SendStop is set
- * (-T on command line), then we send SIGSTOP instead, so that we
- * can get core dumps from all backends by hand.
- *
- * We could exclude dead_end children here, but at least in the
- * SIGSTOP case it seems better to include them.
+ * We could exclude dead_end children here, but at least when
+ * sending SIGABRT it seems better to include them.
*
* Background workers were already processed above; ignore them
* here.
continue;
if (take_action)
- {
- ereport(DEBUG2,
- (errmsg_internal("sending %s to process %d",
- (SendStop ? "SIGSTOP" : "SIGQUIT"),
- (int) bp->pid)));
- signal_child(bp->pid, (SendStop ? SIGSTOP : SIGQUIT));
- }
+ sigquit_child(bp->pid);
}
}
}
else if (StartupPID != 0 && take_action)
{
- ereport(DEBUG2,
- (errmsg_internal("sending %s to process %d",
- (SendStop ? "SIGSTOP" : "SIGQUIT"),
- (int) StartupPID)));
- signal_child(StartupPID, (SendStop ? SIGSTOP : SIGQUIT));
+ sigquit_child(StartupPID);
StartupStatus = STARTUP_SIGNALED;
}
if (pid == BgWriterPID)
BgWriterPID = 0;
else if (BgWriterPID != 0 && take_action)
- {
- ereport(DEBUG2,
- (errmsg_internal("sending %s to process %d",
- (SendStop ? "SIGSTOP" : "SIGQUIT"),
- (int) BgWriterPID)));
- signal_child(BgWriterPID, (SendStop ? SIGSTOP : SIGQUIT));
- }
+ sigquit_child(BgWriterPID);
/* Take care of the checkpointer too */
if (pid == CheckpointerPID)
CheckpointerPID = 0;
else if (CheckpointerPID != 0 && take_action)
- {
- ereport(DEBUG2,
- (errmsg_internal("sending %s to process %d",
- (SendStop ? "SIGSTOP" : "SIGQUIT"),
- (int) CheckpointerPID)));
- signal_child(CheckpointerPID, (SendStop ? SIGSTOP : SIGQUIT));
- }
+ sigquit_child(CheckpointerPID);
/* Take care of the walwriter too */
if (pid == WalWriterPID)
WalWriterPID = 0;
else if (WalWriterPID != 0 && take_action)
- {
- ereport(DEBUG2,
- (errmsg_internal("sending %s to process %d",
- (SendStop ? "SIGSTOP" : "SIGQUIT"),
- (int) WalWriterPID)));
- signal_child(WalWriterPID, (SendStop ? SIGSTOP : SIGQUIT));
- }
+ sigquit_child(WalWriterPID);
/* Take care of the walreceiver too */
if (pid == WalReceiverPID)
WalReceiverPID = 0;
else if (WalReceiverPID != 0 && take_action)
- {
- ereport(DEBUG2,
- (errmsg_internal("sending %s to process %d",
- (SendStop ? "SIGSTOP" : "SIGQUIT"),
- (int) WalReceiverPID)));
- signal_child(WalReceiverPID, (SendStop ? SIGSTOP : SIGQUIT));
- }
+ sigquit_child(WalReceiverPID);
/* Take care of the autovacuum launcher too */
if (pid == AutoVacPID)
AutoVacPID = 0;
else if (AutoVacPID != 0 && take_action)
- {
- ereport(DEBUG2,
- (errmsg_internal("sending %s to process %d",
- (SendStop ? "SIGSTOP" : "SIGQUIT"),
- (int) AutoVacPID)));
- signal_child(AutoVacPID, (SendStop ? SIGSTOP : SIGQUIT));
- }
+ sigquit_child(AutoVacPID);
/* Take care of the archiver too */
if (pid == PgArchPID)
PgArchPID = 0;
else if (PgArchPID != 0 && take_action)
- {
- ereport(DEBUG2,
- (errmsg_internal("sending %s to process %d",
- (SendStop ? "SIGSTOP" : "SIGQUIT"),
- (int) PgArchPID)));
- signal_child(PgArchPID, (SendStop ? SIGSTOP : SIGQUIT));
- }
+ sigquit_child(PgArchPID);
/* We do NOT restart the syslogger */
* Any required cleanup will happen at next restart. We
* set FatalError so that an "abnormal shutdown" message
* gets logged when we exit.
+ *
+ * We don't consult send_abort_for_crash here, as it's
+ * unlikely that dumping cores would illuminate the reason
+ * for checkpointer fork failure.
*/
FatalError = true;
pmState = PM_WAIT_DEAD_END;
case SIGINT:
case SIGTERM:
case SIGQUIT:
- case SIGSTOP:
case SIGKILL:
+ case SIGABRT:
if (kill(-pid, signal) < 0)
elog(DEBUG3, "kill(%ld,%d) failed: %m", (long) (-pid), signal);
break;
#endif
}
+/*
+ * Convenience function for killing a child process after a crash of some
+ * other child process. We log the action at a higher level than we would
+ * otherwise do, and we apply send_abort_for_crash to decide which signal
+ * to send. Normally it's SIGQUIT -- and most other comments in this file
+ * are written on the assumption that it is -- but developers might prefer
+ * to use SIGABRT to collect per-child core dumps.
+ */
+static void
+sigquit_child(pid_t pid)
+{
+ ereport(DEBUG2,
+ (errmsg_internal("sending %s to process %d",
+ (send_abort_for_crash ? "SIGABRT" : "SIGQUIT"),
+ (int) pid)));
+ signal_child(pid, (send_abort_for_crash ? SIGABRT : SIGQUIT));
+}
+
/*
* Send a signal to the targeted children (but NOT special children;
* dead_end children are never signaled, either).
if (StartupPID != 0)
{
signal_child(StartupPID, signal);
- if (signal == SIGQUIT || signal == SIGKILL)
+ if (signal == SIGQUIT || signal == SIGKILL || signal == SIGABRT)
StartupStatus = STARTUP_SIGNALED;
}
if (BgWriterPID != 0)
index 836b49484a15444cf9eee184662a75ba4adb1362..349dd6a5379d41779b5e2c49626f4c037fa66994 100644 (file)
true,
NULL, NULL, NULL
},
+ {
+ {"send_abort_for_crash", PGC_SIGHUP, DEVELOPER_OPTIONS,
+ gettext_noop("Send SIGABRT not SIGQUIT to child processes after backend crash."),
+ NULL,
+ GUC_NOT_IN_SAMPLE
+ },
+ &send_abort_for_crash,
+ false,
+ NULL, NULL, NULL
+ },
+ {
+ {"send_abort_for_kill", PGC_SIGHUP, DEVELOPER_OPTIONS,
+ gettext_noop("Send SIGABRT not SIGKILL to stuck child processes."),
+ NULL,
+ GUC_NOT_IN_SAMPLE
+ },
+ &send_abort_for_kill,
+ false,
+ NULL, NULL, NULL
+ },
{
{"log_duration", PGC_SUSET, LOGGING_WHAT,
index 90e333ccd2097c1d8f57c0b5aff263aeb1ca194d..f078321df241d24b33db3058a768888dae20e1ad 100644 (file)
extern PGDLLIMPORT char *bonjour_name;
extern PGDLLIMPORT bool restart_after_crash;
extern PGDLLIMPORT bool remove_temp_files_after_crash;
+extern PGDLLIMPORT bool send_abort_for_crash;
+extern PGDLLIMPORT bool send_abort_for_kill;
#ifdef WIN32
extern PGDLLIMPORT HANDLE PostmasterHandle;