The Monitor Module is responsible for a series of checks against the backends.
It currently supports 4 types of checks:
- connect : it connects to all the backends, and success/failure is logged in table
- ping : it pings to all the backends, and success/failure is logged in table
mysql_server_ping_log . In the case of
mysql-monitor_ping_max_failures missing a heartbeat, it sends a signal to MySQL_Hostgroups_Manager to kill all connections;
- replication lag : it checks
Seconds_Behind_Master to all backends configured with
max_replication_lag greater than 0, and the check is logged in table
max_replication_lag the server is shunned until
- read only : it checks
read_only for all hosts in the hostgroups in table
mysql_replication_hostgroups, and the check is logged in table
mysql_server_read_only_log . If
read_only=1 the host is copied/moved to the
reader_hostgroup, while if
read_only=0 the host is copied/moved to the
Specifies the username that the Monitor module will use to connect to the backend. The user needs only
USAGE privileges to connect, ping and check
read_only. The user needs also
REPLICATION CLIENT privilege if it needs to monitor replication lag.
Password for user mysql-monitor_username
It enables or disables MySQL Monitor. Since MySQL Monitor can interfere with changes applied directly on the Admin interface, this variable allows to temporary disable it.
How frequently a connect check is performed, in milliseconds.
Connection timeout in milliseconds. The current implementation rounds up this value to an integer number of seconds less or equal to the original interval, with 1 second as minimum. This lazy rounding is done because SSL connections are blocking calls.
How frequently a ping check is performed, in milliseconds.
Ping timeout in milliseconds.
If a host misses mysql-monitor_ping_max_failures pings in a row, MySQL_Monitor informs MySQL_Hostgroup_Manager that the node is unreachable and that it should immediately kill all connections.
It is important to note that in case a connection to the backend is not available, MySQL_Monitor will first try to connect in order to ping, therefore the time to detect that a node is down could be one of these two:
- mysql-monitor_ping_max_failures * mysql-monitor_connect_timeout
- mysql-monitor_ping_max_failures * mysql-monitor_ping_timeout
Read only variables:
How frequently a read only check is performed, in milliseconds.
Read only check timeout in milliseconds.
When a node changes its
read_only value from 1 to 0, this variable determines if the node should be present in both hostgroups or not:
- false : node will be moved into
writer_hostgroup and removed from
- true : node will be copied into
writer_hostgroup and stay also in
Replication lag variables:
How frequently a replication lag check is performed, in milliseconds.
Replication lag check timeout in milliseconds.
To prevent log tables from growing without limits, Monitor Module will automatically purge records older than mysql-monitor_history milliseconds. Since ping checks relies on the history table to determine if a node is missing heartbeats, the value of mysql-monitor_history is automatically adjusted to the following if it’s less than that:
- (mysql-monitor_ping_max_failures + 1 ) * mysql-monitor_ping_timeout
The Monitor Module has several internal threads. There are currently 5 main threads:
- Monitor: master thread, responsible for starting and coordinating all the others
- monitor_connect_thread: main thread and scheduler for the connect checks
- monitor_ping_thread: main thread and scheduler for the ping checks
- monitor_read_only_thread: main thread and scheduler for the read only checks
- monitor_replication_lag_thread: main thread and scheduler for the replication lag checks
Up to version v1.2.0 the above threads, excluding Monitor, were also responsible to perform the checks
The implementation in v1.2.0 has a limitation with SSL implementation: with SSL,
connect() is a blocking call, causing the threads to stall while performing the connect phase.
Version v1.2.1 tries to overcome this limitation with a new implementation. Now:
- Monitor initializes a Thread Pool of workers and creates a queue;
- monitor_connect_thread, monitor_ping_thread, monitor_read_only_thread and monitor_replication_lag_thread are producers that generate tasks and send them to the workers using the queue;
- the workers process the tasks and perform the required actions;
- if Monitor detects that the queue is growing too fast, it creates new temporary worker threads
Monitor implements its own connection pool. Connections that are alive for more than 3 *
mysql_thread___monitor_ping_interval milliseconds are automatically purged.
To prevent backends from terminating connections, Monitor module automatically configures
mysql_thread___monitor_ping_interval * 10