Describe the bug
I have a cluster of 20 redis nodes: 10 masters, 10 replicas on 5 servers (all have redis binary 6.2.6). Some of the nodes keep periodically failing after the following line is logged: MASTER <-> REPLICA sync: Flushing old data (5s after it is logged, other nodes mark it as failed). It remains in the failed state for around 15 minutes, then it starts loading data into the memory, once it is done it works for some time and then the process repeats (the entire cycle takes less then 1h).
We have several clusters (all use the same binary on every machine), but this issue persists only on one cluster (one with most reads, writes and memory used). I don’t know how to reliably replicate this behaviour (I think some nodes entered this state after one machine crashed and then restarted), but it keeps coming up.
Note that simply removing failing nodes from the cluster, clearing directories of these nodes and adding “empty” as replicas back to the cluster does not solve the problem. What worked for me was performing a full wipeout of the cluster (clearing node directories, recreating the cluster) and manually adding the data back (which is not a good option).
Logs from one of failing nodes (all the nodes with this behaviour have pretty much the same logs):
25980:S 22 Dec 2021 18:37:35.180 * MASTER <-> REPLICA sync: Flushing old data
25980:S 22 Dec 2021 18:53:52.257 * MASTER <-> REPLICA sync: Loading DB in memory
25980:S 22 Dec 2021 18:53:52.257 * Loading RDB produced by version 6.2.6
25980:S 22 Dec 2021 18:53:52.257 * RDB age 1697 seconds
25980:S 22 Dec 2021 18:53:52.257 * RDB memory usage when created 49151.62 Mb
25980:S 22 Dec 2021 18:53:52.835 # Address updated for node 709085b91dd9f51482e92cd691428fa8cef55adc, now ?:7202
25980:S 22 Dec 2021 18:53:52.836 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now ?:7200
25980:S 22 Dec 2021 18:53:52.836 # Address updated for node d1c28ff5b93388a3691a676305754a1764395789, now ?:7201
25980:S 22 Dec 2021 18:53:52.854 * Clear FAIL state for node 73e7fe7d003eba5c2d3b1b68727d22ac30ced54a: replica is reachable again.
25980:S 22 Dec 2021 18:53:52.855 # Address updated for node 709085b91dd9f51482e92cd691428fa8cef55adc, now 192.168.2.4:7202
25980:S 22 Dec 2021 18:53:52.855 # Address updated for node d1c28ff5b93388a3691a676305754a1764395789, now 192.168.2.2:7201
25980:S 22 Dec 2021 18:53:52.855 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now 192.168.2.4:7200
25980:S 22 Dec 2021 18:53:52.855 # Address updated for node 3936af310758df5b61fa9d2fcd14cbad632ae693, now ?:7203
25980:S 22 Dec 2021 18:53:52.855 # Address updated for node ef6c57e407866c6f320a450de98d7366a3fe54fa, now ?:7202
25980:S 22 Dec 2021 18:53:52.855 # Address updated for node 3936af310758df5b61fa9d2fcd14cbad632ae693, now 192.168.2.3:7203
25980:S 22 Dec 2021 18:53:52.855 # Address updated for node ef6c57e407866c6f320a450de98d7366a3fe54fa, now 192.168.2.2:7202
25980:S 22 Dec 2021 18:53:52.855 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now ?:7200
25980:S 22 Dec 2021 18:53:52.856 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now 192.168.2.4:7200
25980:S 22 Dec 2021 18:53:52.856 # Address updated for node b4e322353f41f3ac4622e8cfb8f11fb31bc704ff, now ?:7201
25980:S 22 Dec 2021 18:53:52.856 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now ?:7201
25980:S 22 Dec 2021 18:53:52.856 # Address updated for node b4e322353f41f3ac4622e8cfb8f11fb31bc704ff, now 192.168.2.4:7201
25980:S 22 Dec 2021 18:53:52.856 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now 192.168.2.3:7201
25980:S 22 Dec 2021 18:53:52.856 # Address updated for node 709085b91dd9f51482e92cd691428fa8cef55adc, now ?:7202
25980:S 22 Dec 2021 18:53:52.857 # Address updated for node 709085b91dd9f51482e92cd691428fa8cef55adc, now 192.168.2.4:7202
25980:S 22 Dec 2021 18:53:52.857 # Address updated for node 158c66d6e6157df99af303d7455575cfab3f4492, now ?:7203
25980:S 22 Dec 2021 18:53:52.857 # Address updated for node 6ab1ec00d594e6a14eea7f5975b465037eadbeb5, now ?:7200
25980:S 22 Dec 2021 18:53:52.857 # Address updated for node 158c66d6e6157df99af303d7455575cfab3f4492, now 192.168.2.5:7203
25980:S 22 Dec 2021 18:53:52.857 # Address updated for node 6ab1ec00d594e6a14eea7f5975b465037eadbeb5, now 192.168.2.5:7200
25980:S 22 Dec 2021 18:53:52.857 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now ?:7201
25980:S 22 Dec 2021 18:53:52.857 # Address updated for node 9908aef4fc3a151e85a201744dddd77eedf6131c, now ?:7202
25980:S 22 Dec 2021 18:53:52.857 # Address updated for node 3936af310758df5b61fa9d2fcd14cbad632ae693, now ?:7203
25980:S 22 Dec 2021 18:53:52.857 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now 192.168.2.3:7201
25980:S 22 Dec 2021 18:53:52.858 # Address updated for node 9908aef4fc3a151e85a201744dddd77eedf6131c, now 192.168.2.5:7202
25980:S 22 Dec 2021 18:53:52.858 # Address updated for node 3936af310758df5b61fa9d2fcd14cbad632ae693, now 192.168.2.3:7203
25980:S 22 Dec 2021 18:53:52.858 # Address updated for node 56f4d7cfc24f1cd40a88851d5986ad6f353f3738, now ?:7203
25980:S 22 Dec 2021 18:53:52.858 * Connecting to MASTER ?:7203
25980:S 22 Dec 2021 18:53:52.859 # Unable to connect to MASTER: Invalid argument
25980:S 22 Dec 2021 18:53:52.859 # Address updated for node b4e322353f41f3ac4622e8cfb8f11fb31bc704ff, now ?:7201
25980:S 22 Dec 2021 18:53:52.859 # Address updated for node 56f4d7cfc24f1cd40a88851d5986ad6f353f3738, now 192.168.2.2:7203
25980:S 22 Dec 2021 18:53:52.859 * Connecting to MASTER 192.168.2.2:7203
25980:S 22 Dec 2021 18:53:52.859 * MASTER <-> REPLICA sync started
25980:S 22 Dec 2021 18:53:52.859 # Address updated for node 06cf5bde0be7c7133b7b51574402eece64e10d2e, now ?:7200
25980:S 22 Dec 2021 18:53:52.859 # Address updated for node 03d34824c22e49d9d6902413ae308fdaed66594f, now ?:7203
25980:S 22 Dec 2021 18:53:52.859 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now ?:7201
25980:S 22 Dec 2021 18:53:52.859 # Address updated for node b4e322353f41f3ac4622e8cfb8f11fb31bc704ff, now 192.168.2.4:7201
25980:S 22 Dec 2021 18:53:52.859 # Address updated for node 06cf5bde0be7c7133b7b51574402eece64e10d2e, now 192.168.2.3:7200
25980:S 22 Dec 2021 18:53:52.859 # Address updated for node 434f7bf3d1f90af3a9e8d434220b824dcb89f9aa, now ?:7203
25980:S 22 Dec 2021 18:53:52.859 # Address updated for node 03d34824c22e49d9d6902413ae308fdaed66594f, now 192.168.2.4:7203
25980:S 22 Dec 2021 18:53:52.859 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now 192.168.2.3:7201
25980:S 22 Dec 2021 18:53:52.860 # Address updated for node 434f7bf3d1f90af3a9e8d434220b824dcb89f9aa, now 192.168.2.1:7203
25980:S 22 Dec 2021 18:53:52.860 # Address updated for node 03d34824c22e49d9d6902413ae308fdaed66594f, now ?:7203
25980:S 22 Dec 2021 18:53:52.860 * FAIL message received from b4e322353f41f3ac4622e8cfb8f11fb31bc704ff about 434f7bf3d1f90af3a9e8d434220b824dcb89f9aa
25980:S 22 Dec 2021 18:53:52.860 # Address updated for node 03d34824c22e49d9d6902413ae308fdaed66594f, now 192.168.2.4:7203
25980:S 22 Dec 2021 18:53:52.860 # Address updated for node b4e322353f41f3ac4622e8cfb8f11fb31bc704ff, now ?:7201
25980:S 22 Dec 2021 18:53:52.860 # Address updated for node 3936af310758df5b61fa9d2fcd14cbad632ae693, now ?:7203
25980:S 22 Dec 2021 18:53:52.860 # Address updated for node b4e322353f41f3ac4622e8cfb8f11fb31bc704ff, now 192.168.2.4:7201
25980:S 22 Dec 2021 18:53:52.860 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now ?:7201
25980:S 22 Dec 2021 18:53:52.860 # Address updated for node 3936af310758df5b61fa9d2fcd14cbad632ae693, now 192.168.2.3:7203
25980:S 22 Dec 2021 18:53:52.860 # Address updated for node 8db389db45fd20805fdbfe7503cb5583cf4d069f, now ?:7202
25980:S 22 Dec 2021 18:53:52.861 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now 192.168.2.3:7201
25980:S 22 Dec 2021 18:53:52.861 # Address updated for node 06cf5bde0be7c7133b7b51574402eece64e10d2e, now ?:7200
25980:S 22 Dec 2021 18:53:52.861 # Address updated for node 6ab1ec00d594e6a14eea7f5975b465037eadbeb5, now ?:7200
25980:S 22 Dec 2021 18:53:52.861 # Address updated for node 3936af310758df5b61fa9d2fcd14cbad632ae693, now ?:7203
25980:S 22 Dec 2021 18:53:52.861 # Address updated for node 8db389db45fd20805fdbfe7503cb5583cf4d069f, now 192.168.2.3:7202
25980:S 22 Dec 2021 18:53:52.861 # Address updated for node 06cf5bde0be7c7133b7b51574402eece64e10d2e, now 192.168.2.3:7200
25980:S 22 Dec 2021 18:53:52.861 # Address updated for node 31d9fb936e598795004652654b31ea7329bb8c0a, now ?:7201
25980:S 22 Dec 2021 18:53:52.861 # Address updated for node 6ab1ec00d594e6a14eea7f5975b465037eadbeb5, now 192.168.2.5:7200
25980:S 22 Dec 2021 18:53:52.861 # Address updated for node 3936af310758df5b61fa9d2fcd14cbad632ae693, now 192.168.2.3:7203
25980:S 22 Dec 2021 18:53:52.861 # Address updated for node 31d9fb936e598795004652654b31ea7329bb8c0a, now 192.168.2.5:7201
25980:S 22 Dec 2021 18:53:52.861 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now ?:7200
25980:S 22 Dec 2021 18:53:52.861 # Address updated for node 06cf5bde0be7c7133b7b51574402eece64e10d2e, now ?:7200
25980:S 22 Dec 2021 18:53:52.861 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now 192.168.2.4:7200
25980:S 22 Dec 2021 18:53:52.862 # Address updated for node 06cf5bde0be7c7133b7b51574402eece64e10d2e, now 192.168.2.3:7200
25980:S 22 Dec 2021 18:53:52.862 # Address updated for node 31d9fb936e598795004652654b31ea7329bb8c0a, now ?:7201
25980:S 22 Dec 2021 18:53:52.862 # Address updated for node 709085b91dd9f51482e92cd691428fa8cef55adc, now ?:7202
25980:S 22 Dec 2021 18:53:52.862 # Address updated for node 31d9fb936e598795004652654b31ea7329bb8c0a, now 192.168.2.5:7201
25980:S 22 Dec 2021 18:53:52.862 # Address updated for node 3936af310758df5b61fa9d2fcd14cbad632ae693, now ?:7203
25980:S 22 Dec 2021 18:53:52.862 # Address updated for node 709085b91dd9f51482e92cd691428fa8cef55adc, now 192.168.2.4:7202
25980:S 22 Dec 2021 18:53:52.862 # Address updated for node 3936af310758df5b61fa9d2fcd14cbad632ae693, now 192.168.2.3:7203
25980:S 22 Dec 2021 18:53:52.862 # Address updated for node ef6c57e407866c6f320a450de98d7366a3fe54fa, now ?:7202
25980:S 22 Dec 2021 18:53:52.862 # Address updated for node 56f4d7cfc24f1cd40a88851d5986ad6f353f3738, now ?:7203
25980:S 22 Dec 2021 18:53:52.862 * Connecting to MASTER ?:7203
25980:S 22 Dec 2021 18:53:52.862 # Unable to connect to MASTER: Invalid argument
25980:S 22 Dec 2021 18:53:52.862 # Address updated for node ef6c57e407866c6f320a450de98d7366a3fe54fa, now 192.168.2.2:7202
25980:S 22 Dec 2021 18:53:52.863 # Address updated for node 06cf5bde0be7c7133b7b51574402eece64e10d2e, now ?:7200
25980:S 22 Dec 2021 18:53:52.863 # Address updated for node 709085b91dd9f51482e92cd691428fa8cef55adc, now ?:7202
25980:S 22 Dec 2021 18:53:52.863 # Address updated for node 03d34824c22e49d9d6902413ae308fdaed66594f, now ?:7203
25980:S 22 Dec 2021 18:53:52.863 # Address updated for node 56f4d7cfc24f1cd40a88851d5986ad6f353f3738, now 192.168.2.2:7203
25980:S 22 Dec 2021 18:53:52.863 * Connecting to MASTER 192.168.2.2:7203
25980:S 22 Dec 2021 18:53:52.863 * MASTER <-> REPLICA sync started
25980:S 22 Dec 2021 18:53:52.863 # Address updated for node 8db389db45fd20805fdbfe7503cb5583cf4d069f, now ?:7202
25980:S 22 Dec 2021 18:53:52.863 # Address updated for node 06cf5bde0be7c7133b7b51574402eece64e10d2e, now 192.168.2.3:7200
25980:S 22 Dec 2021 18:53:52.863 # Address updated for node 709085b91dd9f51482e92cd691428fa8cef55adc, now 192.168.2.4:7202
25980:S 22 Dec 2021 18:53:52.863 # Address updated for node 3936af310758df5b61fa9d2fcd14cbad632ae693, now ?:7203
25980:S 22 Dec 2021 18:53:52.863 # Address updated for node 03d34824c22e49d9d6902413ae308fdaed66594f, now 192.168.2.4:7203
25980:S 22 Dec 2021 18:53:52.863 # Address updated for node 8db389db45fd20805fdbfe7503cb5583cf4d069f, now 192.168.2.3:7202
25980:S 22 Dec 2021 18:53:52.863 * Non blocking connect for SYNC fired the event.
25980:S 22 Dec 2021 18:53:53.075 * Master replied to PING, replication can continue...
25980:S 22 Dec 2021 18:53:53.075 * Partial resynchronization not possible (no cached master)
25980:S 22 Dec 2021 18:53:54.801 # Address updated for node 3936af310758df5b61fa9d2fcd14cbad632ae693, now 192.168.2.3:7203
25980:S 22 Dec 2021 18:57:12.992 * Full resync from master: e99540f2ea8c51635b3c324c23950a909c3d2fda:253200996575
25980:S 22 Dec 2021 18:59:11.649 * FAIL message received from e147b5afbc1e1c3f1984b3cbcb6882221f48d029 about d8bc99f25507740d4b068cc4fc35fad6694f0a8c
25980:S 22 Dec 2021 19:07:13.072 * MASTER <-> REPLICA sync: receiving 17922512181 bytes from master to disk
25980:S 22 Dec 2021 19:11:28.122 # Done loading RDB, keys loaded: 286802974, keys expired: 0.
25980:S 22 Dec 2021 19:11:28.122 * MASTER <-> REPLICA sync: Finished with success
25980:S 22 Dec 2021 19:11:28.974 * Background append only file rewriting started by pid 8989
25980:S 22 Dec 2021 19:11:28.976 # == CRITICAL == This replica is sending an error to its master: 'Protocol error: too big inline request' after processing the command '<unknown>'
25980:S 22 Dec 2021 19:11:28.976 # Protocol error (too big inline request) from client: id=5059 addr=192.168.2.2:7203 laddr=192.168.2.1:33785 fd=7 name= age=1 idle=0 flags=M db=0 sub=0 psub=0 multi=-1 qbuf=81920 qbuf-free=32758 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=135168 events=r cmd=NULL user=(superuser) redir=-1. Query buffer during protocol error: '.%:.}]Q....x.V{..' (... more 81792 bytes ...) ''
25980:S 22 Dec 2021 19:11:28.998 * Clear FAIL state for node 434f7bf3d1f90af3a9e8d434220b824dcb89f9aa: replica is reachable again.
25980:S 22 Dec 2021 19:11:29.913 # Connection with master lost.
25980:S 22 Dec 2021 19:11:29.913 * Reconnecting to MASTER 192.168.2.2:7203
25980:S 22 Dec 2021 19:11:29.913 * MASTER <-> REPLICA sync started
25980:S 22 Dec 2021 19:11:29.913 * Non blocking connect for SYNC fired the event.
25980:S 22 Dec 2021 19:11:29.913 * Master replied to PING, replication can continue...
25980:S 22 Dec 2021 19:11:29.913 * Partial resynchronization not possible (no cached master)
25980:S 22 Dec 2021 19:16:22.085 * Clear FAIL state for node d8bc99f25507740d4b068cc4fc35fad6694f0a8c: replica is reachable again.
25980:S 22 Dec 2021 19:17:32.302 * Full resync from master: e99540f2ea8c51635b3c324c23950a909c3d2fda:253306928978
25980:S 22 Dec 2021 19:18:45.524 * AOF rewrite child asks to stop sending diffs.
8989:C 22 Dec 2021 19:18:45.524 * Parent agreed to stop sending diffs. Finalizing AOF...
8989:C 22 Dec 2021 19:18:45.524 * Concatenating 0.00 MB of AOF diff received from parent.
8989:C 22 Dec 2021 19:18:45.524 * SYNC append only file rewrite performed
8989:C 22 Dec 2021 19:18:46.358 * AOF rewrite: 4 MB of memory used by copy-on-write
25980:S 22 Dec 2021 19:18:47.338 * Background AOF rewrite terminated with success
25980:S 22 Dec 2021 19:18:47.338 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
25980:S 22 Dec 2021 19:18:47.338 * Background AOF rewrite finished successfully
25980:S 22 Dec 2021 19:27:12.607 * MASTER <-> REPLICA sync: receiving 17922554688 bytes from master to disk
25980:S 22 Dec 2021 19:28:31.988 * MASTER <-> REPLICA sync: Flushing old data
25980:S 22 Dec 2021 19:42:13.131 * MASTER <-> REPLICA sync: Loading DB in memory
25980:S 22 Dec 2021 19:42:13.131 * Loading RDB produced by version 6.2.6
25980:S 22 Dec 2021 19:42:13.131 * RDB age 1481 seconds
25980:S 22 Dec 2021 19:42:13.131 * RDB memory usage when created 49150.99 Mb
25980:S 22 Dec 2021 19:42:13.857 # Address updated for node 3936af310758df5b61fa9d2fcd14cbad632ae693, now ?:7203
25980:S 22 Dec 2021 19:42:13.857 # Address updated for node 73e7fe7d003eba5c2d3b1b68727d22ac30ced54a, now ?:7200
25980:S 22 Dec 2021 19:42:14.732 # Address updated for node 3936af310758df5b61fa9d2fcd14cbad632ae693, now 192.168.2.3:7203
25980:S 22 Dec 2021 19:42:14.732 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now ?:7201
25980:S 22 Dec 2021 19:42:14.732 # Address updated for node 73e7fe7d003eba5c2d3b1b68727d22ac30ced54a, now 192.168.2.2:7200
25980:S 22 Dec 2021 19:42:14.733 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now 192.168.2.3:7201
25980:S 22 Dec 2021 19:42:14.733 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now ?:7200
25980:S 22 Dec 2021 19:42:14.733 # Address updated for node 06cf5bde0be7c7133b7b51574402eece64e10d2e, now ?:7200
25980:S 22 Dec 2021 19:42:14.734 # Address updated for node 8db389db45fd20805fdbfe7503cb5583cf4d069f, now ?:7202
25980:S 22 Dec 2021 19:42:14.734 # Address updated for node 31d9fb936e598795004652654b31ea7329bb8c0a, now ?:7201
25980:S 22 Dec 2021 19:42:14.734 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now 192.168.2.4:7200
25980:S 22 Dec 2021 19:42:14.734 # Address updated for node 06cf5bde0be7c7133b7b51574402eece64e10d2e, now 192.168.2.3:7200
25980:S 22 Dec 2021 19:42:14.734 # Address updated for node 8db389db45fd20805fdbfe7503cb5583cf4d069f, now 192.168.2.3:7202
25980:S 22 Dec 2021 19:42:14.734 # Address updated for node 31d9fb936e598795004652654b31ea7329bb8c0a, now 192.168.2.5:7201
25980:S 22 Dec 2021 19:42:14.734 # Address updated for node 434f7bf3d1f90af3a9e8d434220b824dcb89f9aa, now ?:7203
25980:S 22 Dec 2021 19:42:14.734 * FAIL message received from 6ab1ec00d594e6a14eea7f5975b465037eadbeb5 about 73e7fe7d003eba5c2d3b1b68727d22ac30ced54a
25980:S 22 Dec 2021 19:42:14.734 # Address updated for node 434f7bf3d1f90af3a9e8d434220b824dcb89f9aa, now 192.168.2.1:7203
25980:S 22 Dec 2021 19:42:14.735 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now ?:7200
25980:S 22 Dec 2021 19:42:14.735 # Address updated for node 03d34824c22e49d9d6902413ae308fdaed66594f, now ?:7203
25980:S 22 Dec 2021 19:42:14.735 # Address updated for node 8db389db45fd20805fdbfe7503cb5583cf4d069f, now ?:7202
25980:S 22 Dec 2021 19:42:14.735 # Address updated for node 709085b91dd9f51482e92cd691428fa8cef55adc, now ?:7202
25980:S 22 Dec 2021 19:42:14.735 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now 192.168.2.4:7200
25980:S 22 Dec 2021 19:42:14.735 # Address updated for node 8db389db45fd20805fdbfe7503cb5583cf4d069f, now 192.168.2.3:7202
25980:S 22 Dec 2021 19:42:14.735 # Address updated for node 434f7bf3d1f90af3a9e8d434220b824dcb89f9aa, now ?:7203
25980:S 22 Dec 2021 19:42:14.736 # Address updated for node 03d34824c22e49d9d6902413ae308fdaed66594f, now 192.168.2.4:7203
25980:S 22 Dec 2021 19:42:14.736 # Address updated for node 31d9fb936e598795004652654b31ea7329bb8c0a, now ?:7201
25980:S 22 Dec 2021 19:42:14.736 # Address updated for node 434f7bf3d1f90af3a9e8d434220b824dcb89f9aa, now 192.168.2.1:7203
25980:S 22 Dec 2021 19:42:14.736 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now ?:7201
25980:S 22 Dec 2021 19:42:14.736 # Address updated for node b4e322353f41f3ac4622e8cfb8f11fb31bc704ff, now ?:7201
25980:S 22 Dec 2021 19:42:14.736 # Address updated for node 31d9fb936e598795004652654b31ea7329bb8c0a, now 192.168.2.5:7201
25980:S 22 Dec 2021 19:42:14.736 # Address updated for node 709085b91dd9f51482e92cd691428fa8cef55adc, now 192.168.2.4:7202
25980:S 22 Dec 2021 19:42:14.736 # Address updated for node b4e322353f41f3ac4622e8cfb8f11fb31bc704ff, now 192.168.2.4:7201
25980:S 22 Dec 2021 19:42:14.737 # Address updated for node 434f7bf3d1f90af3a9e8d434220b824dcb89f9aa, now ?:7203
25980:S 22 Dec 2021 19:42:14.737 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now 192.168.2.3:7201
25980:S 22 Dec 2021 19:42:14.737 # Address updated for node 434f7bf3d1f90af3a9e8d434220b824dcb89f9aa, now 192.168.2.1:7203
25980:S 22 Dec 2021 19:42:14.737 # Address updated for node 8db389db45fd20805fdbfe7503cb5583cf4d069f, now ?:7202
25980:S 22 Dec 2021 19:42:14.737 # Address updated for node d1c28ff5b93388a3691a676305754a1764395789, now ?:7201
25980:S 22 Dec 2021 19:42:14.737 # Address updated for node 9908aef4fc3a151e85a201744dddd77eedf6131c, now ?:7202
25980:S 22 Dec 2021 19:42:14.738 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now ?:7200
25980:S 22 Dec 2021 19:42:14.738 # Address updated for node 709085b91dd9f51482e92cd691428fa8cef55adc, now ?:7202
25980:S 22 Dec 2021 19:42:14.738 # Address updated for node 8db389db45fd20805fdbfe7503cb5583cf4d069f, now 192.168.2.3:7202
25980:S 22 Dec 2021 19:42:14.738 # Address updated for node d1c28ff5b93388a3691a676305754a1764395789, now 192.168.2.2:7201
25980:S 22 Dec 2021 19:42:14.738 # Address updated for node 9908aef4fc3a151e85a201744dddd77eedf6131c, now 192.168.2.5:7202
25980:S 22 Dec 2021 19:42:14.738 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now 192.168.2.4:7200
25980:S 22 Dec 2021 19:42:14.738 # Address updated for node 709085b91dd9f51482e92cd691428fa8cef55adc, now 192.168.2.4:7202
25980:S 22 Dec 2021 19:42:14.739 # Address updated for node 31d9fb936e598795004652654b31ea7329bb8c0a, now ?:7201
25980:S 22 Dec 2021 19:42:14.740 # Address updated for node 31d9fb936e598795004652654b31ea7329bb8c0a, now 192.168.2.5:7201
25980:S 22 Dec 2021 19:42:14.740 # Address updated for node 56f4d7cfc24f1cd40a88851d5986ad6f353f3738, now ?:7203
25980:S 22 Dec 2021 19:42:14.741 * Connecting to MASTER ?:7203
25980:S 22 Dec 2021 19:42:14.741 # Unable to connect to MASTER: Invalid argument
25980:S 22 Dec 2021 19:42:14.741 # Address updated for node 56f4d7cfc24f1cd40a88851d5986ad6f353f3738, now 192.168.2.2:7203
25980:S 22 Dec 2021 19:42:14.741 * Connecting to MASTER 192.168.2.2:7203
25980:S 22 Dec 2021 19:42:14.741 * MASTER <-> REPLICA sync started
25980:S 22 Dec 2021 19:42:14.741 # Address updated for node 709085b91dd9f51482e92cd691428fa8cef55adc, now ?:7202
25980:S 22 Dec 2021 19:42:14.742 # Address updated for node 709085b91dd9f51482e92cd691428fa8cef55adc, now 192.168.2.4:7202
25980:S 22 Dec 2021 19:42:14.742 # Address updated for node 158c66d6e6157df99af303d7455575cfab3f4492, now ?:7203
25980:S 22 Dec 2021 19:42:14.742 # Address updated for node 158c66d6e6157df99af303d7455575cfab3f4492, now 192.168.2.5:7203
25980:S 22 Dec 2021 19:42:14.743 * FAIL message received from 3936af310758df5b61fa9d2fcd14cbad632ae693 about 434f7bf3d1f90af3a9e8d434220b824dcb89f9aa
25980:S 22 Dec 2021 19:42:14.743 # Address updated for node 709085b91dd9f51482e92cd691428fa8cef55adc, now ?:7202
25980:S 22 Dec 2021 19:42:14.743 # Address updated for node 03d34824c22e49d9d6902413ae308fdaed66594f, now ?:7203
25980:S 22 Dec 2021 19:42:14.743 # Address updated for node 9908aef4fc3a151e85a201744dddd77eedf6131c, now ?:7202
25980:S 22 Dec 2021 19:42:14.743 # Address updated for node 709085b91dd9f51482e92cd691428fa8cef55adc, now 192.168.2.4:7202
25980:S 22 Dec 2021 19:42:14.743 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now ?:7200
25980:S 22 Dec 2021 19:42:14.743 # Address updated for node 03d34824c22e49d9d6902413ae308fdaed66594f, now 192.168.2.4:7203
25980:S 22 Dec 2021 19:42:14.743 # Address updated for node 9908aef4fc3a151e85a201744dddd77eedf6131c, now 192.168.2.5:7202
25980:S 22 Dec 2021 19:42:14.743 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now 192.168.2.4:7200
25980:S 22 Dec 2021 19:42:14.743 # Address updated for node ef6c57e407866c6f320a450de98d7366a3fe54fa, now ?:7202
25980:S 22 Dec 2021 19:42:14.744 # Address updated for node 158c66d6e6157df99af303d7455575cfab3f4492, now ?:7203
25980:S 22 Dec 2021 19:42:14.744 # Address updated for node ef6c57e407866c6f320a450de98d7366a3fe54fa, now 192.168.2.2:7202
25980:S 22 Dec 2021 19:42:14.744 # Address updated for node 158c66d6e6157df99af303d7455575cfab3f4492, now 192.168.2.5:7203
25980:S 22 Dec 2021 19:42:14.744 # Address updated for node 06cf5bde0be7c7133b7b51574402eece64e10d2e, now ?:7200
25980:S 22 Dec 2021 19:42:14.744 # Address updated for node ef6c57e407866c6f320a450de98d7366a3fe54fa, now ?:7202
25980:S 22 Dec 2021 19:42:14.744 # Address updated for node 06cf5bde0be7c7133b7b51574402eece64e10d2e, now 192.168.2.3:7200
25980:S 22 Dec 2021 19:42:14.744 # Address updated for node ef6c57e407866c6f320a450de98d7366a3fe54fa, now 192.168.2.2:7202
25980:S 22 Dec 2021 19:42:14.745 # Address updated for node 31d9fb936e598795004652654b31ea7329bb8c0a, now ?:7201
25980:S 22 Dec 2021 19:42:14.745 # Address updated for node 9908aef4fc3a151e85a201744dddd77eedf6131c, now ?:7202
25980:S 22 Dec 2021 19:42:14.745 # Address updated for node b4e322353f41f3ac4622e8cfb8f11fb31bc704ff, now ?:7201
25980:S 22 Dec 2021 19:42:14.745 * Non blocking connect for SYNC fired the event.
25980:S 22 Dec 2021 19:42:14.745 # Address updated for node b4e322353f41f3ac4622e8cfb8f11fb31bc704ff, now 192.168.2.4:7201
25980:S 22 Dec 2021 19:42:14.745 # Address updated for node 9908aef4fc3a151e85a201744dddd77eedf6131c, now 192.168.2.5:7202
25980:S 22 Dec 2021 19:42:14.947 * Master replied to PING, replication can continue...
25980:S 22 Dec 2021 19:42:15.158 * Partial resynchronization not possible (no cached master)
25980:S 22 Dec 2021 19:42:15.158 # Address updated for node 31d9fb936e598795004652654b31ea7329bb8c0a, now 192.168.2.5:7201
25980:S 22 Dec 2021 19:44:21.570 * FAIL message received from 6ab1ec00d594e6a14eea7f5975b465037eadbeb5 about 7e701d55a7e397c593e2be367bef083de8bcbc88
25980:S 22 Dec 2021 19:46:49.278 * FAIL message received from e147b5afbc1e1c3f1984b3cbcb6882221f48d029 about d8bc99f25507740d4b068cc4fc35fad6694f0a8c
25980:S 22 Dec 2021 19:48:19.629 * Full resync from master: e99540f2ea8c51635b3c324c23950a909c3d2fda:253462349608
25980:S 22 Dec 2021 19:57:06.412 * MASTER <-> REPLICA sync: receiving 17923026057 bytes from master to disk
25980:S 22 Dec 2021 19:57:59.949 # Done loading RDB, keys loaded: 286793373, keys expired: 0.
25980:S 22 Dec 2021 19:57:59.949 * MASTER <-> REPLICA sync: Finished with success
25980:S 22 Dec 2021 19:58:00.857 * Background append only file rewriting started by pid 13655
25980:S 22 Dec 2021 19:58:00.860 # == CRITICAL == This replica is sending an error to its master: 'Protocol error: too big inline request' after processing the command '<unknown>'
25980:S 22 Dec 2021 19:58:00.860 # Protocol error (too big inline request) from client: id=5227 addr=192.168.2.2:7203 laddr=192.168.2.1:43137 fd=7 name= age=1 idle=0 flags=M db=0 sub=0 psub=0 multi=-1 qbuf=81920 qbuf-free=32758 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=135168 events=r cmd=NULL user=(superuser) redir=-1. Query buffer during protocol error: '.' (... more 81792 bytes ...) '51A01639638645908048DF4728360A49..'
25980:S 22 Dec 2021 19:58:00.913 * Clear FAIL state for node 73e7fe7d003eba5c2d3b1b68727d22ac30ced54a: replica is reachable again.
25980:S 22 Dec 2021 19:58:00.947 * Clear FAIL state for node 434f7bf3d1f90af3a9e8d434220b824dcb89f9aa: replica is reachable again.
25980:S 22 Dec 2021 19:58:01.789 # Connection with master lost.
25980:S 22 Dec 2021 19:58:01.789 * Reconnecting to MASTER 192.168.2.2:7203
25980:S 22 Dec 2021 19:58:01.789 * MASTER <-> REPLICA sync started
25980:S 22 Dec 2021 19:58:01.789 * Non blocking connect for SYNC fired the event.
25980:S 22 Dec 2021 19:58:01.789 * Master replied to PING, replication can continue...
25980:S 22 Dec 2021 19:58:01.790 * Partial resynchronization not possible (no cached master)
25980:S 22 Dec 2021 19:58:03.022 * Full resync from master: e99540f2ea8c51635b3c324c23950a909c3d2fda:253510818973
25980:S 22 Dec 2021 19:59:05.140 * Clear FAIL state for node 7e701d55a7e397c593e2be367bef083de8bcbc88: replica is reachable again.
25980:S 22 Dec 2021 20:04:00.439 * Clear FAIL state for node d8bc99f25507740d4b068cc4fc35fad6694f0a8c: replica is reachable again.
25980:S 22 Dec 2021 20:05:31.437 * AOF rewrite child asks to stop sending diffs.
13655:C 22 Dec 2021 20:05:31.437 * Parent agreed to stop sending diffs. Finalizing AOF...
13655:C 22 Dec 2021 20:05:31.437 * Concatenating 0.00 MB of AOF diff received from parent.
13655:C 22 Dec 2021 20:05:31.437 * SYNC append only file rewrite performed
13655:C 22 Dec 2021 20:05:32.332 * AOF rewrite: 4 MB of memory used by copy-on-write
25980:S 22 Dec 2021 20:05:33.351 * Background AOF rewrite terminated with success
25980:S 22 Dec 2021 20:05:33.351 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
25980:S 22 Dec 2021 20:05:33.351 * Background AOF rewrite finished successfully
25980:S 22 Dec 2021 20:06:53.703 * MASTER <-> REPLICA sync: receiving 17923253708 bytes from master to disk
25980:S 22 Dec 2021 20:08:18.650 * MASTER <-> REPLICA sync: Flushing old data
25980:S 22 Dec 2021 20:22:23.592 * MASTER <-> REPLICA sync: Loading DB in memory
25980:S 22 Dec 2021 20:22:23.592 * Loading RDB produced by version 6.2.6
25980:S 22 Dec 2021 20:22:23.592 * RDB age 1461 seconds
25980:S 22 Dec 2021 20:22:23.592 * RDB memory usage when created 49198.24 Mb
25980:S 22 Dec 2021 20:22:24.749 # Address updated for node 158c66d6e6157df99af303d7455575cfab3f4492, now ?:7203
25980:S 22 Dec 2021 20:22:24.749 # Address updated for node 709085b91dd9f51482e92cd691428fa8cef55adc, now ?:7202
25980:S 22 Dec 2021 20:22:24.750 # Address updated for node 158c66d6e6157df99af303d7455575cfab3f4492, now 192.168.2.5:7203
25980:S 22 Dec 2021 20:22:24.750 # Address updated for node 709085b91dd9f51482e92cd691428fa8cef55adc, now 192.168.2.4:7202
25980:S 22 Dec 2021 20:22:24.750 # Address updated for node 6ab1ec00d594e6a14eea7f5975b465037eadbeb5, now ?:7200
25980:S 22 Dec 2021 20:22:24.750 # Address updated for node ef6c57e407866c6f320a450de98d7366a3fe54fa, now ?:7202
25980:S 22 Dec 2021 20:22:24.750 # Address updated for node 6ab1ec00d594e6a14eea7f5975b465037eadbeb5, now 192.168.2.5:7200
25980:S 22 Dec 2021 20:22:24.750 # Address updated for node 73e7fe7d003eba5c2d3b1b68727d22ac30ced54a, now ?:7200
25980:S 22 Dec 2021 20:22:24.750 # Address updated for node ef6c57e407866c6f320a450de98d7366a3fe54fa, now 192.168.2.2:7202
25980:S 22 Dec 2021 20:22:24.751 # Address updated for node 9908aef4fc3a151e85a201744dddd77eedf6131c, now ?:7202
25980:S 22 Dec 2021 20:22:24.751 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now ?:7201
25980:S 22 Dec 2021 20:22:24.751 # Address updated for node 73e7fe7d003eba5c2d3b1b68727d22ac30ced54a, now 192.168.2.2:7200
25980:S 22 Dec 2021 20:22:24.751 # Address updated for node 31d9fb936e598795004652654b31ea7329bb8c0a, now ?:7201
25980:S 22 Dec 2021 20:22:24.751 # Address updated for node 9908aef4fc3a151e85a201744dddd77eedf6131c, now 192.168.2.5:7202
25980:S 22 Dec 2021 20:22:24.751 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now 192.168.2.3:7201
25980:S 22 Dec 2021 20:22:24.751 # Address updated for node 31d9fb936e598795004652654b31ea7329bb8c0a, now 192.168.2.5:7201
25980:S 22 Dec 2021 20:22:24.751 # Address updated for node 56f4d7cfc24f1cd40a88851d5986ad6f353f3738, now ?:7203
25980:S 22 Dec 2021 20:22:24.752 * Connecting to MASTER ?:7203
25980:S 22 Dec 2021 20:22:24.752 # Unable to connect to MASTER: Invalid argument
25980:S 22 Dec 2021 20:22:24.752 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now ?:7200
25980:S 22 Dec 2021 20:22:24.752 # Address updated for node 158c66d6e6157df99af303d7455575cfab3f4492, now ?:7203
25980:S 22 Dec 2021 20:22:24.752 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now ?:7201
25980:S 22 Dec 2021 20:22:24.752 # Address updated for node 03d34824c22e49d9d6902413ae308fdaed66594f, now ?:7203
25980:S 22 Dec 2021 20:22:24.752 # Address updated for node 56f4d7cfc24f1cd40a88851d5986ad6f353f3738, now 192.168.2.2:7203
25980:S 22 Dec 2021 20:22:24.752 * Connecting to MASTER 192.168.2.2:7203
25980:S 22 Dec 2021 20:22:24.752 * MASTER <-> REPLICA sync started
25980:S 22 Dec 2021 20:22:24.752 # Address updated for node 06cf5bde0be7c7133b7b51574402eece64e10d2e, now ?:7200
25980:S 22 Dec 2021 20:22:24.752 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now 192.168.2.4:7200
25980:S 22 Dec 2021 20:22:24.752 # Address updated for node 158c66d6e6157df99af303d7455575cfab3f4492, now 192.168.2.5:7203
25980:S 22 Dec 2021 20:22:24.752 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now 192.168.2.3:7201
25980:S 22 Dec 2021 20:22:24.753 # Address updated for node 03d34824c22e49d9d6902413ae308fdaed66594f, now 192.168.2.4:7203
25980:S 22 Dec 2021 20:22:24.753 # Address updated for node 06cf5bde0be7c7133b7b51574402eece64e10d2e, now 192.168.2.3:7200
25980:S 22 Dec 2021 20:22:24.753 # Address updated for node 9908aef4fc3a151e85a201744dddd77eedf6131c, now ?:7202
25980:S 22 Dec 2021 20:22:24.753 # Address updated for node 03d34824c22e49d9d6902413ae308fdaed66594f, now ?:7203
25980:S 22 Dec 2021 20:22:24.753 # Address updated for node 9908aef4fc3a151e85a201744dddd77eedf6131c, now 192.168.2.5:7202
25980:S 22 Dec 2021 20:22:24.753 # Address updated for node 03d34824c22e49d9d6902413ae308fdaed66594f, now 192.168.2.4:7203
25980:S 22 Dec 2021 20:22:24.753 # Address updated for node 73e7fe7d003eba5c2d3b1b68727d22ac30ced54a, now ?:7200
25980:S 22 Dec 2021 20:22:24.753 # Address updated for node 9908aef4fc3a151e85a201744dddd77eedf6131c, now ?:7202
25980:S 22 Dec 2021 20:22:24.753 # Address updated for node 56f4d7cfc24f1cd40a88851d5986ad6f353f3738, now ?:7203
25980:S 22 Dec 2021 20:22:24.753 * Connecting to MASTER ?:7203
25980:S 22 Dec 2021 20:22:24.753 # Unable to connect to MASTER: Invalid argument
25980:S 22 Dec 2021 20:22:24.753 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now ?:7201
25980:S 22 Dec 2021 20:22:24.754 # Address updated for node 73e7fe7d003eba5c2d3b1b68727d22ac30ced54a, now 192.168.2.2:7200
25980:S 22 Dec 2021 20:22:24.754 # Address updated for node 9908aef4fc3a151e85a201744dddd77eedf6131c, now 192.168.2.5:7202
25980:S 22 Dec 2021 20:22:24.754 # Address updated for node ef6c57e407866c6f320a450de98d7366a3fe54fa, now ?:7202
25980:S 22 Dec 2021 20:22:24.754 # Address updated for node 56f4d7cfc24f1cd40a88851d5986ad6f353f3738, now 192.168.2.2:7203
25980:S 22 Dec 2021 20:22:24.754 * Connecting to MASTER 192.168.2.2:7203
25980:S 22 Dec 2021 20:22:24.754 * MASTER <-> REPLICA sync started
25980:S 22 Dec 2021 20:22:24.754 # Address updated for node 6ab1ec00d594e6a14eea7f5975b465037eadbeb5, now ?:7200
25980:S 22 Dec 2021 20:22:24.754 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now ?:7200
25980:S 22 Dec 2021 20:22:24.754 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now 192.168.2.3:7201
25980:S 22 Dec 2021 20:22:24.754 # Address updated for node 73e7fe7d003eba5c2d3b1b68727d22ac30ced54a, now ?:7200
25980:S 22 Dec 2021 20:22:24.754 # Address updated for node ef6c57e407866c6f320a450de98d7366a3fe54fa, now 192.168.2.2:7202
25980:S 22 Dec 2021 20:22:24.754 # Address updated for node 8db389db45fd20805fdbfe7503cb5583cf4d069f, now ?:7202
25980:S 22 Dec 2021 20:22:24.754 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now 192.168.2.4:7200
25980:S 22 Dec 2021 20:22:24.754 # Address updated for node 73e7fe7d003eba5c2d3b1b68727d22ac30ced54a, now 192.168.2.2:7200
25980:S 22 Dec 2021 20:22:24.754 # Address updated for node 8db389db45fd20805fdbfe7503cb5583cf4d069f, now 192.168.2.3:7202
25980:S 22 Dec 2021 20:22:24.754 # Address updated for node 6ab1ec00d594e6a14eea7f5975b465037eadbeb5, now 192.168.2.5:7200
25980:S 22 Dec 2021 20:22:24.755 # Address updated for node 73e7fe7d003eba5c2d3b1b68727d22ac30ced54a, now ?:7200
25980:S 22 Dec 2021 20:22:24.755 # Address updated for node 31d9fb936e598795004652654b31ea7329bb8c0a, now ?:7201
25980:S 22 Dec 2021 20:22:24.755 # Address updated for node 73e7fe7d003eba5c2d3b1b68727d22ac30ced54a, now 192.168.2.2:7200
25980:S 22 Dec 2021 20:22:24.755 # Address updated for node 31d9fb936e598795004652654b31ea7329bb8c0a, now 192.168.2.5:7201
25980:S 22 Dec 2021 20:22:24.755 # Address updated for node 3936af310758df5b61fa9d2fcd14cbad632ae693, now ?:7203
25980:S 22 Dec 2021 20:22:24.755 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now ?:7200
25980:S 22 Dec 2021 20:22:24.755 # Address updated for node 6ab1ec00d594e6a14eea7f5975b465037eadbeb5, now ?:7200
25980:S 22 Dec 2021 20:22:24.755 # Address updated for node 3936af310758df5b61fa9d2fcd14cbad632ae693, now 192.168.2.3:7203
25980:S 22 Dec 2021 20:22:24.756 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now 192.168.2.4:7200
25980:S 22 Dec 2021 20:22:24.756 # Address updated for node 31d9fb936e598795004652654b31ea7329bb8c0a, now ?:7201
25980:S 22 Dec 2021 20:22:24.756 # Address updated for node 6ab1ec00d594e6a14eea7f5975b465037eadbeb5, now 192.168.2.5:7200
25980:S 22 Dec 2021 20:22:24.756 # Address updated for node 06cf5bde0be7c7133b7b51574402eece64e10d2e, now ?:7200
25980:S 22 Dec 2021 20:22:24.756 # Address updated for node d1c28ff5b93388a3691a676305754a1764395789, now ?:7201
25980:S 22 Dec 2021 20:22:24.756 # Address updated for node 31d9fb936e598795004652654b31ea7329bb8c0a, now 192.168.2.5:7201
25980:S 22 Dec 2021 20:22:24.756 # Address updated for node 709085b91dd9f51482e92cd691428fa8cef55adc, now ?:7202
25980:S 22 Dec 2021 20:22:24.756 # Address updated for node 56f4d7cfc24f1cd40a88851d5986ad6f353f3738, now ?:7203
25980:S 22 Dec 2021 20:22:24.756 * Connecting to MASTER ?:7203
25980:S 22 Dec 2021 20:22:24.756 # Unable to connect to MASTER: Invalid argument
25980:S 22 Dec 2021 20:22:24.756 # Address updated for node 06cf5bde0be7c7133b7b51574402eece64e10d2e, now 192.168.2.3:7200
25980:S 22 Dec 2021 20:22:24.756 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now ?:7201
25980:S 22 Dec 2021 20:22:24.756 # Address updated for node d1c28ff5b93388a3691a676305754a1764395789, now 192.168.2.2:7201
25980:S 22 Dec 2021 20:22:24.756 # Address updated for node 709085b91dd9f51482e92cd691428fa8cef55adc, now 192.168.2.4:7202
25980:S 22 Dec 2021 20:22:24.756 # Address updated for node 9908aef4fc3a151e85a201744dddd77eedf6131c, now ?:7202
25980:S 22 Dec 2021 20:22:24.756 # Address updated for node ef6c57e407866c6f320a450de98d7366a3fe54fa, now ?:7202
25980:S 22 Dec 2021 20:22:24.756 # Address updated for node 56f4d7cfc24f1cd40a88851d5986ad6f353f3738, now 192.168.2.2:7203
25980:S 22 Dec 2021 20:22:24.756 * Connecting to MASTER 192.168.2.2:7203
25980:S 22 Dec 2021 20:22:24.756 * MASTER <-> REPLICA sync started
25980:S 22 Dec 2021 20:22:24.756 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now 192.168.2.3:7201
25980:S 22 Dec 2021 20:22:24.756 # Address updated for node 9908aef4fc3a151e85a201744dddd77eedf6131c, now 192.168.2.5:7202
25980:S 22 Dec 2021 20:22:24.757 # Address updated for node ef6c57e407866c6f320a450de98d7366a3fe54fa, now 192.168.2.2:7202
25980:S 22 Dec 2021 20:22:24.757 # Address updated for node 6ab1ec00d594e6a14eea7f5975b465037eadbeb5, now ?:7200
25980:S 22 Dec 2021 20:22:24.757 # Address updated for node 3936af310758df5b61fa9d2fcd14cbad632ae693, now ?:7203
25980:S 22 Dec 2021 20:22:24.757 # Address updated for node 6ab1ec00d594e6a14eea7f5975b465037eadbeb5, now 192.168.2.5:7200
25980:S 22 Dec 2021 20:22:24.757 # Address updated for node 3936af310758df5b61fa9d2fcd14cbad632ae693, now 192.168.2.3:7203
25980:S 22 Dec 2021 20:22:24.758 # Address updated for node 6ab1ec00d594e6a14eea7f5975b465037eadbeb5, now ?:7200
25980:S 22 Dec 2021 20:22:24.758 # Address updated for node 6ab1ec00d594e6a14eea7f5975b465037eadbeb5, now 192.168.2.5:7200
25980:S 22 Dec 2021 20:22:24.758 # Address updated for node 73e7fe7d003eba5c2d3b1b68727d22ac30ced54a, now ?:7200
25980:S 22 Dec 2021 20:22:24.758 * Non blocking connect for SYNC fired the event.
25980:S 22 Dec 2021 20:22:24.985 * Master replied to PING, replication can continue...
25980:S 22 Dec 2021 20:22:24.985 * Partial resynchronization not possible (no cached master)
25980:S 22 Dec 2021 20:26:26.564 * Full resync from master: e99540f2ea8c51635b3c324c23950a909c3d2fda:253647088587
25980:S 22 Dec 2021 20:26:52.930 * FAIL message received from 31d9fb936e598795004652654b31ea7329bb8c0a about 7e701d55a7e397c593e2be367bef083de8bcbc88
25980:S 22 Dec 2021 20:33:54.610 * FAIL message received from 6ab1ec00d594e6a14eea7f5975b465037eadbeb5 about d8bc99f25507740d4b068cc4fc35fad6694f0a8c
25980:S 22 Dec 2021 20:36:21.842 * MASTER <-> REPLICA sync: receiving 17910905234 bytes from master to disk
25980:S 22 Dec 2021 20:38:51.388 # Done loading RDB, keys loaded: 286792616, keys expired: 0.
25980:S 22 Dec 2021 20:38:51.388 * MASTER <-> REPLICA sync: Finished with success
25980:S 22 Dec 2021 20:38:52.281 * Background append only file rewriting started by pid 17988
25980:S 22 Dec 2021 20:38:52.284 # == CRITICAL == This replica is sending an error to its master: 'Protocol error: too big inline request' after processing the command '<unknown>'
25980:S 22 Dec 2021 20:38:52.284 # Protocol error (too big inline request) from client: id=5368 addr=192.168.2.2:7203 laddr=192.168.2.1:37355 fd=7 name= age=1 idle=0 flags=M db=0 sub=0 psub=0 multi=-1 qbuf=81920 qbuf-free=32758 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=135168 events=r cmd=NULL user=(superuser) redir=-1. Query buffer during protocol error: '.' (... more 81792 bytes ...) 'tcTaMK8so0XXw6l..'
25980:S 22 Dec 2021 20:38:53.470 # Connection with master lost.
25980:S 22 Dec 2021 20:38:53.470 * Reconnecting to MASTER 192.168.2.2:7203
25980:S 22 Dec 2021 20:38:53.471 * MASTER <-> REPLICA sync started
25980:S 22 Dec 2021 20:38:53.471 * Non blocking connect for SYNC fired the event.
25980:S 22 Dec 2021 20:38:53.471 * Master replied to PING, replication can continue...
25980:S 22 Dec 2021 20:38:53.471 * Partial resynchronization not possible (no cached master)
25980:S 22 Dec 2021 20:40:43.660 * Clear FAIL state for node 7e701d55a7e397c593e2be367bef083de8bcbc88: replica is reachable again.
25980:S 22 Dec 2021 20:45:49.031 * AOF rewrite child asks to stop sending diffs.
17988:C 22 Dec 2021 20:45:49.031 * Parent agreed to stop sending diffs. Finalizing AOF...
17988:C 22 Dec 2021 20:45:49.031 * Concatenating 0.00 MB of AOF diff received from parent.
17988:C 22 Dec 2021 20:45:49.031 * SYNC append only file rewrite performed
17988:C 22 Dec 2021 20:45:50.031 * AOF rewrite: 4 MB of memory used by copy-on-write
25980:S 22 Dec 2021 20:45:51.098 * Background AOF rewrite terminated with success
25980:S 22 Dec 2021 20:45:51.098 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
25980:S 22 Dec 2021 20:45:51.098 * Background AOF rewrite finished successfully
25980:S 22 Dec 2021 20:47:09.646 * Full resync from master: e99540f2ea8c51635b3c324c23950a909c3d2fda:253737120761
25980:S 22 Dec 2021 20:51:36.999 * Clear FAIL state for node d8bc99f25507740d4b068cc4fc35fad6694f0a8c: replica is reachable again.
25980:S 22 Dec 2021 20:55:48.174 * MASTER <-> REPLICA sync: receiving 17917537607 bytes from master to disk
25980:S 22 Dec 2021 20:57:05.975 * MASTER <-> REPLICA sync: Flushing old data
25980:S 22 Dec 2021 21:11:29.437 * MASTER <-> REPLICA sync: Loading DB in memory
25980:S 22 Dec 2021 21:11:29.437 * Loading RDB produced by version 6.2.6
25980:S 22 Dec 2021 21:11:29.437 * RDB age 1460 seconds
25980:S 22 Dec 2021 21:11:29.437 * RDB memory usage when created 49147.87 Mb
25980:S 22 Dec 2021 21:11:30.514 # Address updated for node 6ab1ec00d594e6a14eea7f5975b465037eadbeb5, now ?:7200
25980:S 22 Dec 2021 21:11:30.515 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now ?:7201
25980:S 22 Dec 2021 21:11:30.543 # Address updated for node 6ab1ec00d594e6a14eea7f5975b465037eadbeb5, now 192.168.2.5:7200
25980:S 22 Dec 2021 21:11:30.543 # Address updated for node ef6c57e407866c6f320a450de98d7366a3fe54fa, now ?:7202
25980:S 22 Dec 2021 21:11:30.543 # Address updated for node 158c66d6e6157df99af303d7455575cfab3f4492, now ?:7203
25980:S 22 Dec 2021 21:11:30.543 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now 192.168.2.3:7201
25980:S 22 Dec 2021 21:11:30.543 # Address updated for node 03d34824c22e49d9d6902413ae308fdaed66594f, now ?:7203
25980:S 22 Dec 2021 21:11:30.543 # Address updated for node ef6c57e407866c6f320a450de98d7366a3fe54fa, now 192.168.2.2:7202
25980:S 22 Dec 2021 21:11:30.543 # Address updated for node 158c66d6e6157df99af303d7455575cfab3f4492, now 192.168.2.5:7203
25980:S 22 Dec 2021 21:11:30.543 # Address updated for node 03d34824c22e49d9d6902413ae308fdaed66594f, now 192.168.2.4:7203
25980:S 22 Dec 2021 21:11:30.544 # Address updated for node 56f4d7cfc24f1cd40a88851d5986ad6f353f3738, now ?:7203
25980:S 22 Dec 2021 21:11:30.545 * Connecting to MASTER ?:7203
25980:S 22 Dec 2021 21:11:30.545 # Unable to connect to MASTER: Invalid argument
25980:S 22 Dec 2021 21:11:30.545 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now ?:7200
25980:S 22 Dec 2021 21:11:30.545 # Address updated for node 56f4d7cfc24f1cd40a88851d5986ad6f353f3738, now 192.168.2.2:7203
25980:S 22 Dec 2021 21:11:30.545 * Connecting to MASTER 192.168.2.2:7203
25980:S 22 Dec 2021 21:11:30.545 * MASTER <-> REPLICA sync started
25980:S 22 Dec 2021 21:11:30.545 # Address updated for node 73e7fe7d003eba5c2d3b1b68727d22ac30ced54a, now ?:7200
25980:S 22 Dec 2021 21:11:30.546 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now 192.168.2.4:7200
25980:S 22 Dec 2021 21:11:30.546 # Address updated for node ef6c57e407866c6f320a450de98d7366a3fe54fa, now ?:7202
25980:S 22 Dec 2021 21:11:30.546 # Address updated for node 73e7fe7d003eba5c2d3b1b68727d22ac30ced54a, now 192.168.2.2:7200
25980:S 22 Dec 2021 21:11:30.546 # Address updated for node ef6c57e407866c6f320a450de98d7366a3fe54fa, now 192.168.2.2:7202
25980:S 22 Dec 2021 21:11:30.546 # Address updated for node 8db389db45fd20805fdbfe7503cb5583cf4d069f, now ?:7202
25980:S 22 Dec 2021 21:11:30.546 # Address updated for node 3936af310758df5b61fa9d2fcd14cbad632ae693, now ?:7203
25980:S 22 Dec 2021 21:11:30.546 # Address updated for node 709085b91dd9f51482e92cd691428fa8cef55adc, now ?:7202
25980:S 22 Dec 2021 21:11:30.547 # Address updated for node 3936af310758df5b61fa9d2fcd14cbad632ae693, now 192.168.2.3:7203
25980:S 22 Dec 2021 21:11:30.547 # Address updated for node 9908aef4fc3a151e85a201744dddd77eedf6131c, now ?:7202
25980:S 22 Dec 2021 21:11:30.547 # Address updated for node 709085b91dd9f51482e92cd691428fa8cef55adc, now 192.168.2.4:7202
25980:S 22 Dec 2021 21:11:30.547 # Address updated for node 8db389db45fd20805fdbfe7503cb5583cf4d069f, now 192.168.2.3:7202
25980:S 22 Dec 2021 21:11:30.547 # Address updated for node 9908aef4fc3a151e85a201744dddd77eedf6131c, now 192.168.2.5:7202
25980:S 22 Dec 2021 21:11:30.548 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now ?:7201
25980:S 22 Dec 2021 21:11:30.548 # Address updated for node 06cf5bde0be7c7133b7b51574402eece64e10d2e, now ?:7200
25980:S 22 Dec 2021 21:11:30.548 # Address updated for node 8db389db45fd20805fdbfe7503cb5583cf4d069f, now ?:7202
25980:S 22 Dec 2021 21:11:30.548 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now 192.168.2.3:7201
25980:S 22 Dec 2021 21:11:30.548 # Address updated for node 06cf5bde0be7c7133b7b51574402eece64e10d2e, now 192.168.2.3:7200
25980:S 22 Dec 2021 21:11:30.548 # Address updated for node 8db389db45fd20805fdbfe7503cb5583cf4d069f, now 192.168.2.3:7202
25980:S 22 Dec 2021 21:11:30.549 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now ?:7200
25980:S 22 Dec 2021 21:11:30.549 # Address updated for node 3936af310758df5b61fa9d2fcd14cbad632ae693, now ?:7203
25980:S 22 Dec 2021 21:11:30.549 # Address updated for node 56f4d7cfc24f1cd40a88851d5986ad6f353f3738, now ?:7203
25980:S 22 Dec 2021 21:11:30.549 * Connecting to MASTER ?:7203
25980:S 22 Dec 2021 21:11:30.549 # Unable to connect to MASTER: Invalid argument
25980:S 22 Dec 2021 21:11:30.549 # Address updated for node b4e322353f41f3ac4622e8cfb8f11fb31bc704ff, now ?:7201
25980:S 22 Dec 2021 21:11:30.549 # Address updated for node 3936af310758df5b61fa9d2fcd14cbad632ae693, now 192.168.2.3:7203
25980:S 22 Dec 2021 21:11:30.549 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now 192.168.2.4:7200
25980:S 22 Dec 2021 21:11:30.549 # Address updated for node 56f4d7cfc24f1cd40a88851d5986ad6f353f3738, now 192.168.2.2:7203
25980:S 22 Dec 2021 21:11:30.549 * Connecting to MASTER 192.168.2.2:7203
25980:S 22 Dec 2021 21:11:30.549 * MASTER <-> REPLICA sync started
25980:S 22 Dec 2021 21:11:30.549 # Address updated for node b4e322353f41f3ac4622e8cfb8f11fb31bc704ff, now 192.168.2.4:7201
25980:S 22 Dec 2021 21:11:30.549 # Address updated for node 8db389db45fd20805fdbfe7503cb5583cf4d069f, now ?:7202
25980:S 22 Dec 2021 21:11:30.549 # Address updated for node ef6c57e407866c6f320a450de98d7366a3fe54fa, now ?:7202
25980:S 22 Dec 2021 21:11:30.549 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now ?:7200
25980:S 22 Dec 2021 21:11:30.550 # Address updated for node b4e322353f41f3ac4622e8cfb8f11fb31bc704ff, now ?:7201
25980:S 22 Dec 2021 21:11:30.550 # Address updated for node 31d9fb936e598795004652654b31ea7329bb8c0a, now ?:7201
25980:S 22 Dec 2021 21:11:30.550 # Address updated for node 8db389db45fd20805fdbfe7503cb5583cf4d069f, now 192.168.2.3:7202
25980:S 22 Dec 2021 21:11:30.550 # Address updated for node 6ab1ec00d594e6a14eea7f5975b465037eadbeb5, now ?:7200
25980:S 22 Dec 2021 21:11:30.550 # Address updated for node ef6c57e407866c6f320a450de98d7366a3fe54fa, now 192.168.2.2:7202
25980:S 22 Dec 2021 21:11:30.550 # Address updated for node 5f32067dd8ac9a3c9c0fd919010614ce218195fa, now 192.168.2.4:7200
25980:S 22 Dec 2021 21:11:30.550 * FAIL message received from 5f32067dd8ac9a3c9c0fd919010614ce218195fa about 434f7bf3d1f90af3a9e8d434220b824dcb89f9aa
25980:S 22 Dec 2021 21:11:30.550 # Address updated for node b4e322353f41f3ac4622e8cfb8f11fb31bc704ff, now 192.168.2.4:7201
25980:S 22 Dec 2021 21:11:30.550 # Address updated for node 31d9fb936e598795004652654b31ea7329bb8c0a, now 192.168.2.5:7201
25980:S 22 Dec 2021 21:11:30.550 # Address updated for node 6ab1ec00d594e6a14eea7f5975b465037eadbeb5, now 192.168.2.5:7200
25980:S 22 Dec 2021 21:11:30.551 # Address updated for node 8db389db45fd20805fdbfe7503cb5583cf4d069f, now ?:7202
25980:S 22 Dec 2021 21:11:30.551 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now ?:7201
25980:S 22 Dec 2021 21:11:30.551 # Address updated for node b4e322353f41f3ac4622e8cfb8f11fb31bc704ff, now ?:7201
25980:S 22 Dec 2021 21:11:30.551 # Address updated for node 8db389db45fd20805fdbfe7503cb5583cf4d069f, now 192.168.2.3:7202
25980:S 22 Dec 2021 21:11:30.551 # Address updated for node e147b5afbc1e1c3f1984b3cbcb6882221f48d029, now 192.168.2.3:7201
25980:S 22 Dec 2021 21:11:30.551 # Address updated for node b4e322353f41f3ac4622e8cfb8f11fb31bc704ff, now 192.168.2.4:7201
25980:S 22 Dec 2021 21:11:30.551 # Address updated for node ef6c57e407866c6f320a450de98d7366a3fe54fa, now ?:7202
25980:S 22 Dec 2021 21:11:30.552 # Address updated for node ef6c57e407866c6f320a450de98d7366a3fe54fa, now 192.168.2.2:7202
25980:S 22 Dec 2021 21:11:30.552 # Address updated for node 9908aef4fc3a151e85a201744dddd77eedf6131c, now ?:7202
25980:S 22 Dec 2021 21:11:30.552 # Address updated for node 56f4d7cfc24f1cd40a88851d5986ad6f353f3738, now ?:7203
25980:S 22 Dec 2021 21:11:30.552 * Connecting to MASTER ?:7203
25980:S 22 Dec 2021 21:11:30.552 # Unable to connect to MASTER: Invalid argument
25980:S 22 Dec 2021 21:11:30.552 # Address updated for node 56f4d7cfc24f1cd40a88851d5986ad6f353f3738, now 192.168.2.2:7203
25980:S 22 Dec 2021 21:11:30.552 * Connecting to MASTER 192.168.2.2:7203
25980:S 22 Dec 2021 21:11:30.552 * MASTER <-> REPLICA sync started
25980:S 22 Dec 2021 21:11:30.553 # Address updated for node 9908aef4fc3a151e85a201744dddd77eedf6131c, now 192.168.2.5:7202
25980:S 22 Dec 2021 21:11:30.553 # Address updated for node 56f4d7cfc24f1cd40a88851d5986ad6f353f3738, now ?:7203
25980:S 22 Dec 2021 21:11:30.553 * Connecting to MASTER ?:7203
25980:S 22 Dec 2021 21:11:30.553 # Unable to connect to MASTER: Invalid argument
25980:S 22 Dec 2021 21:11:30.553 # Address updated for node 56f4d7cfc24f1cd40a88851d5986ad6f353f3738, now 192.168.2.2:7203
25980:S 22 Dec 2021 21:11:30.553 * Connecting to MASTER 192.168.2.2:7203
25980:S 22 Dec 2021 21:11:30.553 * MASTER <-> REPLICA sync started
25980:S 22 Dec 2021 21:11:30.554 # Address updated for node 158c66d6e6157df99af303d7455575cfab3f4492, now ?:7203
25980:S 22 Dec 2021 21:11:30.554 # Address updated for node 9908aef4fc3a151e85a201744dddd77eedf6131c, now ?:7202
25980:S 22 Dec 2021 21:11:30.554 # Address updated for node 06cf5bde0be7c7133b7b51574402eece64e10d2e, now ?:7200
25980:S 22 Dec 2021 21:11:30.554 * Non blocking connect for SYNC fired the event.
25980:S 22 Dec 2021 21:11:30.816 * Master replied to PING, replication can continue...
25980:S 22 Dec 2021 21:11:31.065 * Partial resynchronization not possible (no cached master)
25980:S 22 Dec 2021 21:11:31.306 # Address updated for node 06cf5bde0be7c7133b7b51574402eece64e10d2e, now 192.168.2.3:7200
25980:S 22 Dec 2021 21:11:32.185 # Address updated for node 9908aef4fc3a151e85a201744dddd77eedf6131c, now 192.168.2.5:7202
25980:S 22 Dec 2021 21:11:32.401 # Address updated for node 158c66d6e6157df99af303d7455575cfab3f4492, now 192.168.2.5:7203
25980:S 22 Dec 2021 21:16:04.632 * Full resync from master: e99540f2ea8c51635b3c324c23950a909c3d2fda:253875101840
25980:S 22 Dec 2021 21:23:36.776 * FAIL message received from 3936af310758df5b61fa9d2fcd14cbad632ae693 about d8bc99f25507740d4b068cc4fc35fad6694f0a8c
25980:S 22 Dec 2021 21:25:57.769 * MASTER <-> REPLICA sync: receiving 17909014873 bytes from master to disk
25980:S 22 Dec 2021 21:27:07.256 # Done loading RDB, keys loaded: 286739068, keys expired: 0.
25980:S 22 Dec 2021 21:27:07.256 * MASTER <-> REPLICA sync: Finished with success
25980:S 22 Dec 2021 21:27:08.141 * Background append only file rewriting started by pid 23847
25980:S 22 Dec 2021 21:27:08.143 # == CRITICAL == This replica is sending an error to its master: 'Protocol error: too big inline request' after processing the command '<unknown>'
25980:S 22 Dec 2021 21:27:08.143 # Protocol error (too big inline request) from client: id=5542 addr=192.168.2.2:7203 laddr=192.168.2.1:36667 fd=7 name= age=1 idle=0 flags=M db=0 sub=0 psub=0 multi=-1 qbuf=81920 qbuf-free=32758 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=135168 events=r cmd=NULL user=(superuser) redir=-1. Query buffer during protocol error: '....' (... more 81792 bytes ...) '14|0D5C08EB-007D-482B-B2DB-4DE203BB173B..'
25980:S 22 Dec 2021 21:27:08.186 * Clear FAIL state for node 434f7bf3d1f90af3a9e8d434220b824dcb89f9aa: replica is reachable again.
25980:S 22 Dec 2021 21:27:08.760 # Connection with master lost.
25980:S 22 Dec 2021 21:27:08.760 * Reconnecting to MASTER 192.168.2.2:7203
25980:S 22 Dec 2021 21:27:08.760 * MASTER <-> REPLICA sync started
25980:S 22 Dec 2021 21:27:08.760 * Non blocking connect for SYNC fired the event.
25980:S 22 Dec 2021 21:27:08.761 * Master replied to PING, replication can continue...
25980:S 22 Dec 2021 21:27:08.761 * Partial resynchronization not possible (no cached master)
25980:S 22 Dec 2021 21:29:33.992 # User requested shutdown...
25980:S 22 Dec 2021 21:29:33.992 # Writing initial AOF, can't exit.
25980:signal-handler (1640204973) Received SIGTERM scheduling shutdown...
23847:signal-handler (1640204973) Received SIGTERM scheduling shutdown...
25980:S 22 Dec 2021 21:29:34.026 # User requested shutdown...
25980:S 22 Dec 2021 21:29:34.026 # Writing initial AOF, can't exit.
25980:S 22 Dec 2021 21:29:34.026 # SIGTERM received but errors trying to shut down the server, check the logs for more information
25980:S 22 Dec 2021 21:29:41.509 * FAIL message received from 6ab1ec00d594e6a14eea7f5975b465037eadbeb5 about 434f7bf3d1f90af3a9e8d434220b824dcb89f9aa
25980:S 22 Dec 2021 21:29:41.512 * FAIL message received from 5f32067dd8ac9a3c9c0fd919010614ce218195fa about 7e701d55a7e397c593e2be367bef083de8bcbc88
25980:S 22 Dec 2021 21:29:58.506 * FAIL message received from e147b5afbc1e1c3f1984b3cbcb6882221f48d029 about 73e7fe7d003eba5c2d3b1b68727d22ac30ced54a
25980:S 22 Dec 2021 21:33:43.549 * AOF rewrite child asks to stop sending diffs.
23847:C 22 Dec 2021 21:33:43.549 * Parent agreed to stop sending diffs. Finalizing AOF...
23847:C 22 Dec 2021 21:33:43.549 * Concatenating 0.00 MB of AOF diff received from parent.
23847:C 22 Dec 2021 21:33:43.549 * SYNC append only file rewrite performed
23847:C 22 Dec 2021 21:33:44.342 * AOF rewrite: 4 MB of memory used by copy-on-write
25980:S 22 Dec 2021 21:33:45.323 * Background AOF rewrite terminated with success
25980:S 22 Dec 2021 21:33:45.323 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
25980:S 22 Dec 2021 21:33:45.323 * Background AOF rewrite finished successfully
25980:S 22 Dec 2021 21:36:47.832 * Full resync from master: e99540f2ea8c51635b3c324c23950a909c3d2fda:253957567999
25980:S 22 Dec 2021 21:45:50.436 * MASTER <-> REPLICA sync: receiving 17920908458 bytes from master to disk
25980:S 22 Dec 2021 21:47:08.242 * MASTER <-> REPLICA sync: Flushing old data
25980:S 22 Dec 2021 22:00:05.219 * MASTER <-> REPLICA sync: Loading DB in memory
Here you can see this process visualized, the two trendlines are "used_memory" values of two failing nodes over time (flushing old data log is the moment when data stops):
This is the result of calling INFO on of the nodes (all have the same configuration):
# Server
redis_version:6.2.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:efe04f4e448f5ad6
redis_mode:cluster
os:Linux 4.19.0-9-amd64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:8.3.0
process_id:18571
process_supervised:systemd
run_id:78ef4b60da98c6f0170407765adbd7a55d9ff715
tcp_port:7200
server_time_usec:1641571997903170
uptime_in_seconds:17282
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:14182045
executable:/opt/redis_binaries/redis-server
config_file:/etc/redis/node_conf_files/cookiematching_7200.conf
io_threads_active:0
# Clients
connected_clients:89
cluster_connections:38
maxclients:10000
client_recent_max_input_buffer:32
client_recent_max_output_buffer:20504
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
# Memory
used_memory:20503788256
used_memory_human:19.10G
used_memory_rss:20937285632
used_memory_rss_human:19.50G
used_memory_peak:20503803936
used_memory_peak_human:19.10G
used_memory_peak_perc:100.00%
used_memory_overhead:6115615288
used_memory_startup:1468400
used_memory_dataset:14388172968
used_memory_dataset_perc:70.18%
allocator_allocated:20503895224
allocator_active:20524175360
allocator_resident:20945936384
total_system_memory:404403412992
total_system_memory_human:376.63G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:51539607552
maxmemory_human:48.00G
maxmemory_policy:allkeys-lru
allocator_frag_ratio:1.00
allocator_frag_bytes:20280136
allocator_rss_ratio:1.02
allocator_rss_bytes:421761024
rss_overhead_ratio:1.00
rss_overhead_bytes:-8650752
mem_fragmentation_ratio:1.02
mem_fragmentation_bytes:433566344
mem_not_counted_for_evict:3578
mem_replication_backlog:1048576
mem_clients_slaves:20512
mem_clients_normal:1804672
mem_aof_buffer:3584
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
current_cow_size:5025792
current_cow_size_age:42
current_fork_perc:19.48
current_save_keys_processed:24523777
current_save_keys_total:125902966
rdb_changes_since_last_save:324321
rdb_bgsave_in_progress:1
rdb_last_save_time:1641571894
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:196
rdb_current_bgsave_time_sec:42
rdb_last_cow_size:1492475904
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:147
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:7279239168
module_fork_in_progress:0
module_fork_last_cow_size:0
aof_current_size:7478824066
aof_base_size:5239960581
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:13
# Stats
total_connections_received:2660
total_commands_processed:340017703
instantaneous_ops_per_sec:5601
total_net_input_bytes:21620506521
total_net_output_bytes:13534313433
instantaneous_input_kbps:324.87
instantaneous_output_kbps:137.62
rejected_connections:0
sync_full:1
sync_partial_ok:0
sync_partial_err:1
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:207
evicted_keys:0
keyspace_hits:25531002
keyspace_misses:165544986
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:381093
total_forks:116
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:5
dump_payload_sanitizations:0
total_reads_processed:340020279
total_writes_processed:439113011
io_threaded_reads_processed:0
io_threaded_writes_processed:0
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.2.3,port=7202,state=online,offset=10471944058,lag=0
master_failover_state:no-failover
master_replid:3ddbbc74d205cde9848fe962903fc613ae5af9dd
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:10471960047
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:10470911472
repl_backlog_histlen:1048576
# CPU
used_cpu_sys:4784.654426
used_cpu_user:2786.454085
used_cpu_sys_children:959.697741
used_cpu_user_children:9008.792155
used_cpu_sys_main_thread:4776.362818
used_cpu_user_main_thread:2783.488297
# Modules
# Errorstats
errorstat_CLUSTERDOWN:count=1
errorstat_MOVED:count=4
# Cluster
cluster_enabled:1
# Keyspace
db0:keys=125938193,expires=0,avg_ttl=0
Note on how I can help Today I did a fresh restart of the entire cluster, so for now I cannot replicate this issue. But if it comes up in the future I could for example run a modified redis binary and provide its logs.
Comment From: Jeremy-Run
repl_backlog_size:1048576 (1MB, Total size in bytes of the replication backlog buffer)
repl_backlog_histlen:1048576 (1MB, Size in bytes of the data in the replication backlog buffer)
You said "this issue persists only on one cluster (one with most reads, writes and memory used)" .
My guess is that the replication buffer is too small, causing constant full synchronization.
You can adjust the size of the repl_backlog_size parameter to see if this problem occurs again
Comment From: Joe2k
@Mikolaj6 Hi! Did you find any solution to this issue?