ProxySQL is a multi-threaded daemon, with one or more threads per module, as already pointed out in ProxySQL Threads.
The most critical threads are the the MySQL Threads.
Although the number of critical sections is limited, a few bugs in the past have caused unexpected locking issues and stalls.
For this reason, ProxySQL 1.4.4 introduces a watchdog that monitors MySQL Threads and MySQL Auxiliary Threads(if enabled).
If one or more worker threads start missing heartbeats for any reason at all, it means they are not serving traffic. The reason for this could be anything, including a bug.
If the watchdog detects that any of these threads are missing heartbeats for an extended period of time for any reason, it assumes they are blocked and it automatically asserts, resulting in a core dump and an automatic restart from the parent process in order to bring everything back to normal.
Basically, the watchdog mechanism detects a (potential) system hang, and attempts to recover it.
Watchdog is enabled by default.
Because each thread should never be blocked for longer than mysql-poll_timeout milliseconds (2000 by default), watchdog checks heartbeats every mysql-poll_timeout+1000 milliseconds (3 seconds by default, 2000ms + 1000ms): if restart_on_missing_heartbeats heartbeats are missing, watchdog triggers an assert.
This means that the whole ProxySQL process crashes, however, by default ProxySQL starts as a process that immediately forks: the parent process (the one that is not responsible for serving traffic) is merely waiting for the child process (the one that is actually responsible for serving traffic) to exit gracefully, and if it does not do that, the parent process restarts it.
If the watchdog asserts and causes a ProxySQL crash, you will see the following lines appear in the error log:
"Watchdog: reached %u missed heartbeats. Aborting!", missed_heartbeats
"Watchdog: see details at https://github.com/sysown/proxysql/wiki/Watchdog"
restart_on_missing_heartbeats can only be configured in the global section of the configuration file, and its default value is 10.
If restart_on_missing_heartbeats is set to 0, watchdogs will still perform checks and generate entries in the error log, but it will not assert and it will not restart proxysql.
See Configuration file for details.
If you do not wish to observe this behavior, you can simply start ProxySQL with the -f option : this will prevent ProxySQL to go into daemon mode – it will not fork, and it will run in the foreground. Please note, however, that in this mode the error log is not going to be written to its default location: error messages are going to be written to stderr only.
Note on ProxySQL and gdb
The watchdog module is extremely simple and lock free, therefore it should never block. Nonetheless, the watchdog module has its own heartbeat, therefore it is able to detect if ProxySQL was intentionally blocked, for example with a SIGSTOP signal or SIGINT signal in gdb: if this happens, it will not assert.