I am setting up a redis cluster with 6 containers(Redis 7.2.5). After scanning with the scan command along with various test cases, I delete the keys I obtained. I'm using RedisPlusPlus. But suddenly the redis instances are shutting down. I don't know how a client can cause a server to shut down.
my cluster.conf is
cluster-enabled yes cluster-config-file nodes.conf databases 16 timeout 300 tcp-keepalive 60 maxmemory 2gb maxmemory-policy allkeys-lru cluster-node-timeout 15000 cluster-require-full-coverage yes protected-mode no dbfilename dump-rdb dir /data save 10 1 loglevel debug client-output-buffer-limit normal 0 0 0 client-output-buffer-limit replica 256mb 64mb 60 client-output-buffer-limit pubsub 32mb 8mb 60 When I examine the logs, I saw the interesting log below: I/O error reading from node link: connection closed I/O error reading from node link: connection closed Connection with Node ... at ... failed: Connection refused.
Those are the latest logs.
I'm working on a virtual machine. I could not find the cause of the problem.
OS: RHEL 8.5 Compiler: gcc 8.5 hiredis version: 1.2.0 redis-plus-plus version: 1.3.11(Written in cmakelists.txt)
When I connect to redis with cli to see monitor command output, I saw this messages, Error: Server closed the connection. At dmesg command output, I saw the listed messages; Code: Unable to access opcode bytes at RIP 0xfffffffffffffd6 app[4130076]: seg fault at 0 ip 00000000000000 sp 0007ff... error 14 in app
Comment From: sundb
@selcukaltinay can you share the fully crash log?
Comment From: selcukaltinay
Services run on docker. The container log output with debug log level: GOSSIP ... master GOSSIP ... slave GOSSIP ... slave I/O error reading from node link: connection closed I/O error reading from node link: connection closed Pinging node c51cffb... Connection with Node ...... at 192.168.1.113:39271 failed: Connection refused --- Processing packet of type pong, 2616 bytes GOSSIP ... master GOSSIP ... slave GOSSIP ... slave
redis-cli monitor output: "SCAN" "0" "MATCH" "mydomain" "COUNT" "10000" "GET" "mydomain1" "SCAN" "0" "MATCH" "another" "COUNT" "10000" "DEL" "another1" "DEL" "another2" "DEL" "another3" "DEL" "another4" "DEL" "another5" "GET" "another1" "GET" "another2" "SCAN" "0" "MATCH" "another2" "COUNT" "10000" "DEL" "another1" "DEL" "another2" Error: Server closed the connection
At dmesg command output: Code: Unable to access opcode bytes at RIP 0xfffffffffffffd6 app[4130076]: seg fault at 0 ip 00000000000000 sp 0007ff... error 14 in app
Regards
Comment From: selcukaltinay
I have compiled redis 7.0.11 instead of using redis 7.2.5 container. There is no problem at the moment. The problem is probably related to Docker's resource usage. But I haven't been able to figure out exactly what's causing the problem yet.
Regards
Comment From: sundb
@selcukaltinay i can't see any clue from your log. it feels like it's related to docker, do you have reproduction steps?