Notice! - If a Redis module was involved, please open an issue in the module's repo instead! - If you're using docker on Apple M1, please make sure the image you're using was compiled for ARM!
Crash report
Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.
8:M 18 Oct 2023 06:05:11.814 * Background saving started by pid 55
55:C 18 Oct 2023 06:05:11.815 * DB saved on disk
55:C 18 Oct 2023 06:05:11.816 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
8:M 18 Oct 2023 06:05:11.895 * Background saving terminated with success
8:M 18 Oct 2023 06:05:11.895 * Synchronization with replica 21.65.101.20:6379 succeeded
8:M 18 Oct 2023 06:05:13.068 * Replica 21.65.101.20:6380 asks for synchronization
8:M 18 Oct 2023 06:05:13.068 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '20c9caafdab1359e6b6a6a46d0cc2ade7b841b8c', my replication IDs are 'f8688aa73f3f0d25629d02841c64b827f9a07f32' and 'af79ffc721b085897157146603d7d80c61b7c4c5')
8:M 18 Oct 2023 06:05:13.068 * Starting BGSAVE for SYNC with target: disk
8:M 18 Oct 2023 06:05:13.069 * Background saving started by pid 56
56:C 18 Oct 2023 06:05:13.070 * DB saved on disk
56:C 18 Oct 2023 06:05:13.070 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
8:M 18 Oct 2023 06:05:13.107 * Background saving terminated with success
8:M 18 Oct 2023 06:05:13.107 * Synchronization with replica 21.65.101.20:6380 succeeded
8:M 18 Oct 2023 06:10:22.358 # Connection with replica 21.65.101.20:6379 lost.
8:M 18 Oct 2023 06:10:22.364 # Configuration change detected. Reconfiguring myself as a replica of 44caa8ecdd58bff792ee975e064972fd3e7baf86
8:S 18 Oct 2023 06:10:22.364 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
8:S 18 Oct 2023 06:10:22.364 * Connecting to MASTER 21.65.101.20:6379
8:S 18 Oct 2023 06:10:22.365 * MASTER <-> REPLICA sync started
8:S 18 Oct 2023 06:10:22.365 # Connection with replica 21.65.101.10:6380 lost.
8:S 18 Oct 2023 06:10:22.365 # Connection with replica 21.65.101.10:6381 lost.
8:S 18 Oct 2023 06:10:22.365 * Non blocking connect for SYNC fired the event.
8:S 18 Oct 2023 06:10:22.365 * Master replied to PING, replication can continue...
8:S 18 Oct 2023 06:10:22.366 * Trying a partial resynchronization (request f8688aa73f3f0d25629d02841c64b827f9a07f32:981).
8:S 18 Oct 2023 06:10:22.366 * Successful partial resynchronization with master.
8:S 18 Oct 2023 06:10:22.366 # Master replication ID changed to 436807e2e4395e2f680f9ac1290dabcaafd3f440
8:S 18 Oct 2023 06:10:22.366 # Connection with replica 21.65.101.20:6380 lost.
8:S 18 Oct 2023 06:10:22.366 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.
8:S 18 Oct 2023 06:10:39.168 # Address updated for node dd12c8159ff72c7a76f0451c984e48d2d3796efb, now 10.128.2.1:6379
8:S 18 Oct 2023 06:10:39.548 # Address updated for node ff1bba77c4b1d72c05d70787b9bf07e39f551141, now 10.128.0.1:6379
8:S 18 Oct 2023 06:10:54.074 * FAIL message received from 1a84630e0e6b9ab7b3864d11428706c9e4731727 about 11dfc8f566ae13de42bda013cbcea7bdacf7259d
8:S 18 Oct 2023 06:10:54.075 * FAIL message received from 571a4e7140e8fbe12f5edd77f4eaecd4bde43dde about 49aa9a8d3d318704746752f01658d6e13cb8987d
8:S 18 Oct 2023 06:10:54.144 * Marking node fe96157d7d5401441066bb5598778e46a331ba09 as failing (quorum reached).
8:S 18 Oct 2023 06:10:58.048 * FAIL message received from 7cb853b89475c87e40275fcc4bcfa05ce3170de7 about ff1bba77c4b1d72c05d70787b9bf07e39f551141
8:S 18 Oct 2023 06:11:00.945 * FAIL message received from 44caa8ecdd58bff792ee975e064972fd3e7baf86 about dd12c8159ff72c7a76f0451c984e48d2d3796efb
8:S 18 Oct 2023 06:11:10.064 * 10 changes in 300 seconds. Saving...
8:S 18 Oct 2023 06:11:10.065 * Background saving started by pid 82
82:C 18 Oct 2023 06:11:10.066 * DB saved on disk
82:C 18 Oct 2023 06:11:10.066 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
8:S 18 Oct 2023 06:11:10.166 * Background saving terminated with success
8:S 18 Oct 2023 06:11:22.305 # Address updated for node 94e96da8962e6f1c335e8384e5acd1fe51a0e5af, now 10.130.0.1:6379
8:S 18 Oct 2023 06:11:23.017 # Address updated for node 94e96da8962e6f1c335e8384e5acd1fe51a0e5af, now 21.65.101.20:6381
8:S 18 Oct 2023 06:11:24.974 # Address updated for node 8e33916db10659af47a9ed706203fc83db9664ba, now 10.128.0.1:6379
8:S 18 Oct 2023 06:11:26.285 # Address updated for node 8e33916db10659af47a9ed706203fc83db9664ba, now 21.65.101.20:6384
8:S 18 Oct 2023 06:15:34.730 # Taking over the master (user request).
8:S 18 Oct 2023 06:15:34.730 # New configEpoch set to 68
8:M 18 Oct 2023 06:15:34.730 # Connection with master lost.
8:M 18 Oct 2023 06:15:34.730 * Caching the disconnected master state.
8:M 18 Oct 2023 06:15:34.730 * Discarding previously cached master state.
8:M 18 Oct 2023 06:15:34.730 # Setting secondary replication ID to 436807e2e4395e2f680f9ac1290dabcaafd3f440, valid up to offset: 49987. New replication ID is 94417bffb38e16e1586d417b7302e3b0fa7f108a
8:M 18 Oct 2023 06:15:34.735 * Replica 21.65.101.10:6380 asks for synchronization
8:M 18 Oct 2023 06:15:34.735 * Partial resynchronization request from 21.65.101.10:6380 accepted. Sending 0 bytes of backlog starting from offset 49987.
8:M 18 Oct 2023 06:15:34.737 * Replica 21.65.101.20:6380 asks for synchronization
8:M 18 Oct 2023 06:15:34.737 * Partial resynchronization request from 21.65.101.20:6380 accepted. Sending 0 bytes of backlog starting from offset 49987.
8:M 18 Oct 2023 06:15:34.738 * Replica 21.65.101.20:6381 asks for synchronization
8:M 18 Oct 2023 06:15:34.738 * Partial resynchronization request from 21.65.101.20:6381 accepted. Sending 0 bytes of backlog starting from offset 49987.
8:M 18 Oct 2023 06:15:34.742 * Replica 21.65.101.20:6379 asks for synchronization
8:M 18 Oct 2023 06:15:34.742 * Partial resynchronization request from 21.65.101.20:6379 accepted. Sending 0 bytes of backlog starting from offset 49987.
8:M 18 Oct 2023 06:15:47.684 # Address updated for node b8edfce3d4b6d2ec95d01bbbbd58499ef8184138, now 10.130.1.26:6379
8:M 18 Oct 2023 06:15:49.028 * Replica 21.65.101.10:6381 asks for synchronization
8:M 18 Oct 2023 06:15:49.028 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '8f715698d8ba536edd3afb032bcc5950bb9a93c7', my replication IDs are '94417bffb38e16e1586d417b7302e3b0fa7f108a' and '436807e2e4395e2f680f9ac1290dabcaafd3f440')
8:M 18 Oct 2023 06:15:49.028 * Starting BGSAVE for SYNC with target: disk
8:M 18 Oct 2023 06:15:49.029 * Background saving started by pid 83
83:C 18 Oct 2023 06:15:49.030 * DB saved on disk
83:C 18 Oct 2023 06:15:49.030 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
8:M 18 Oct 2023 06:15:49.072 * Background saving terminated with success
8:M 18 Oct 2023 06:15:49.072 * Synchronization with replica 21.65.101.10:6381 succeeded
8:M 18 Oct 2023 06:15:50.005 # Address updated for node b8edfce3d4b6d2ec95d01bbbbd58499ef8184138, now 21.65.101.10:6381
8:M 18 Oct 2023 06:15:55.050 # Address updated for node dd54fdd8b574a9077445b97e6802a23172e1b6be, now 21.65.101.10:6384
8:M 18 Oct 2023 06:16:04.798 # Connection with replica 21.65.101.20:6381 lost.
8:M 18 Oct 2023 06:16:07.133 * Marking node 559593007dc23685fe00e56bdb3e78951421a746 as failing (quorum reached).
8:M 18 Oct 2023 06:16:07.169 * Marking node 94e96da8962e6f1c335e8384e5acd1fe51a0e5af as failing (quorum reached).
8:M 18 Oct 2023 06:16:07.170 * FAIL message received from dd54fdd8b574a9077445b97e6802a23172e1b6be about 8e33916db10659af47a9ed706203fc83db9664ba
8:M 18 Oct 2023 06:20:50.034 * 10 changes in 300 seconds. Saving...
8:M 18 Oct 2023 06:20:50.034 * Background saving started by pid 144
144:C 18 Oct 2023 06:20:50.036 * DB saved on disk
144:C 18 Oct 2023 06:20:50.036 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
8:M 18 Oct 2023 06:20:50.136 * Background saving terminated with success
8:M 18 Oct 2023 06:20:54.168 # Disconnecting timedout replica (streaming sync): 21.65.101.20:6380
8:M 18 Oct 2023 06:20:54.168 # Connection with replica 21.65.101.20:6380 lost.
8:M 18 Oct 2023 06:20:54.168 # Disconnecting timedout replica (streaming sync): 21.65.101.20:6379
8:M 18 Oct 2023 06:20:54.168 # Connection with replica 21.65.101.20:6379 lost.
8:M 18 Oct 2023 06:25:51.065 * 10 changes in 300 seconds. Saving...
8:M 18 Oct 2023 06:25:51.065 * Background saving started by pid 164
164:C 18 Oct 2023 06:25:51.067 * DB saved on disk
164:C 18 Oct 2023 06:25:51.068 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 0 MB
8:M 18 Oct 2023 06:25:51.166 * Background saving terminated with success
8:M 18 Oct 2023 06:30:52.092 * 10 changes in 300 seconds. Saving...
8:M 18 Oct 2023 06:30:52.093 * Background saving started by pid 165
165:C 18 Oct 2023 06:30:52.095 * DB saved on disk
165:C 18 Oct 2023 06:30:52.095 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
8:M 18 Oct 2023 06:30:52.194 * Background saving terminated with success
8:M 18 Oct 2023 06:35:53.085 * 10 changes in 300 seconds. Saving...
8:M 18 Oct 2023 06:35:53.085 * Background saving started by pid 172
172:C 18 Oct 2023 06:35:53.087 * DB saved on disk
172:C 18 Oct 2023 06:35:53.087 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
8:M 18 Oct 2023 06:35:53.186 * Background saving terminated with success
8:M 18 Oct 2023 06:40:54.050 * 10 changes in 300 seconds. Saving...
8:M 18 Oct 2023 06:40:54.050 * Background saving started by pid 173
173:C 18 Oct 2023 06:40:54.052 * DB saved on disk
173:C 18 Oct 2023 06:40:54.052 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
8:M 18 Oct 2023 06:40:54.152 * Background saving terminated with success
8:M 18 Oct 2023 06:45:55.097 * 10 changes in 300 seconds. Saving...
8:M 18 Oct 2023 06:45:55.097 * Background saving started by pid 174
174:C 18 Oct 2023 06:45:55.099 * DB saved on disk
174:C 18 Oct 2023 06:45:55.099 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
8:M 18 Oct 2023 06:45:55.199 * Background saving terminated with success
8:M 18 Oct 2023 06:50:56.063 * 10 changes in 300 seconds. Saving...
8:M 18 Oct 2023 06:50:56.063 * Background saving started by pid 175
175:C 18 Oct 2023 06:50:56.065 * DB saved on disk
175:C 18 Oct 2023 06:50:56.065 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 0 MB
8:M 18 Oct 2023 06:50:56.164 * Background saving terminated with success
8:M 18 Oct 2023 06:55:57.000 * 10 changes in 300 seconds. Saving...
8:M 18 Oct 2023 06:55:57.000 * Background saving started by pid 176
176:C 18 Oct 2023 06:55:57.002 * DB saved on disk
176:C 18 Oct 2023 06:55:57.002 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
8:M 18 Oct 2023 06:55:57.101 * Background saving terminated with success
8:M 18 Oct 2023 07:00:58.023 * 10 changes in 300 seconds. Saving...
8:M 18 Oct 2023 07:00:58.023 * Background saving started by pid 190
190:C 18 Oct 2023 07:00:58.025 * DB saved on disk
190:C 18 Oct 2023 07:00:58.026 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
8:M 18 Oct 2023 07:00:58.125 * Background saving terminated with success
8:M 18 Oct 2023 07:05:59.078 * 10 changes in 300 seconds. Saving...
8:M 18 Oct 2023 07:05:59.078 * Background saving started by pid 191
191:C 18 Oct 2023 07:05:59.081 * DB saved on disk
191:C 18 Oct 2023 07:05:59.081 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
8:M 18 Oct 2023 07:05:59.180 * Background saving terminated with success
8:M 18 Oct 2023 07:07:48.982 * 10000 changes in 60 seconds. Saving...
8:M 18 Oct 2023 07:07:48.982 * Background saving started by pid 192
192:C 18 Oct 2023 07:07:48.996 * DB saved on disk
192:C 18 Oct 2023 07:07:48.997 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 0 MB
8:M 18 Oct 2023 07:07:49.083 * Background saving terminated with success
8:M 18 Oct 2023 07:08:50.075 * 10000 changes in 60 seconds. Saving...
8:M 18 Oct 2023 07:08:50.075 * Background saving started by pid 193
193:C 18 Oct 2023 07:08:50.121 * DB saved on disk
193:C 18 Oct 2023 07:08:50.122 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
8:M 18 Oct 2023 07:08:50.177 * Background saving terminated with success
8:M 18 Oct 2023 07:09:51.077 * 10000 changes in 60 seconds. Saving...
8:M 18 Oct 2023 07:09:51.078 * Background saving started by pid 194
194:C 18 Oct 2023 07:09:51.167 * DB saved on disk
194:C 18 Oct 2023 07:09:51.168 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
8:M 18 Oct 2023 07:09:51.179 * Background saving terminated with success
8:M 18 Oct 2023 07:10:52.083 * 10000 changes in 60 seconds. Saving...
8:M 18 Oct 2023 07:10:52.084 * Background saving started by pid 201
201:C 18 Oct 2023 07:10:52.222 * DB saved on disk
201:C 18 Oct 2023 07:10:52.224 * Fork CoW for RDB: current 3 MB, peak 3 MB, average 2 MB
8:M 18 Oct 2023 07:10:52.287 * Background saving terminated with success
8:M 18 Oct 2023 07:11:53.092 * 10000 changes in 60 seconds. Saving...
8:M 18 Oct 2023 07:11:53.093 * Background saving started by pid 202
202:C 18 Oct 2023 07:11:53.244 * DB saved on disk
202:C 18 Oct 2023 07:11:53.245 * Fork CoW for RDB: current 2 MB, peak 2 MB, average 1 MB
8:M 18 Oct 2023 07:11:53.296 * Background saving terminated with success
8:M 18 Oct 2023 07:12:44.673 * Replica 21.65.101.20:6379 asks for synchronization
8:M 18 Oct 2023 07:12:44.673 * Unable to partial resync with replica 21.65.101.20:6379 for lack of backlog (Replica request was: 164515).
8:M 18 Oct 2023 07:12:44.673 * Starting BGSAVE for SYNC with target: disk
8:M 18 Oct 2023 07:12:44.674 * Background saving started by pid 215
215:C 18 Oct 2023 07:12:44.945 * DB saved on disk
215:C 18 Oct 2023 07:12:44.947 * Fork CoW for RDB: current 3 MB, peak 3 MB, average 2 MB
8:M 18 Oct 2023 07:12:45.332 * Background saving terminated with success
Killed
According to monitor, the memory was up to 1.13GB, but the rdb file is only 3MB. redis_version:7.0.13. Additional information
- OS distribution and version : OpenShift 4.8.29
- Steps to reproduce (if any): (1) deploy redis cluster with redis_version:7.0.13 on 2 OpenShift . (2) breakdown network between leader and follower on switch. (3)write data to leader. (4) recovery network.
Comment From: yossigo
You should use the INFO command to get better visibility into how much memory is consumed by Redis and in what way.