Redis very slow master-slave replication

I made replication like this master -> slave1-> slave2. Slave 1 is master to slave2. Master ->slave1 replication work very fast. But slave1->slave2 replication is slow. Ethernet speed between slave1 and slave2 server is about 200Mb/s but replication speed is about 1Mb/s. at same time redis server use about 300%Cpu(according to top command).

Database has 16Gb of data

redis was upgraded from 6.0.6 (default redis version for ubuntu main repository) to 7.0.8 from official redis ppa It was a problem with starting redis after upgrade . I fix this by editing systemd startup file(add TimeoutSec parameter )

all redis instances are running as standlone apps (no docker)

Master:
    host Ubuntu 20.04.5 LTS
    redis-server- 7.08
    cpu Intel(R) Xeon(R) CPU E5-2620
    redis-config
            bind 0.0.0.0 
            protected-mode no
            daemonize no
            supervised systemd
            pidfile /var/run/redis/redis-server.pid
            save 900 1
            save 300 10
            #save 60 10000
            rdbcompression no
            repl-diskless-sync no
            repl-backlog-size 2gb
             maxmemory 50gb
             io-threads 6
            client-output-buffer-limit replica 0 0 0
            client-output-buffer-limit pubsub 0mb 0mb 0
            client-output-buffer-limit slave  0 0 0

Slave1
    host Ubuntu 20.04.5 LTS
    redis-server- 7.08
    cpu - Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
    redis-config:
            bind 0.0.0.0 -::1
            protected-mode no
            daemonize no
            supervised systemd
            #pidfile /run/redis/redis-server.pid
            save 3600 1 300 100 
            rdbcompression no
            replicaof xxx.xxx.xxx.xxx 6379
            io-threads 6
            io-threads-do-reads yes

Slave2
    host Ubuntu 20.04.5 LTS (digital ocean virtual machine)
    redis-server- 7.08
    cpu - Intel(R) Xeon(R) Platinum 8358 CPU 
    redis-config : 
        daemonize no
        supervised systemd
        save 900 1
        save 300 10
        #save 60 10000
        rdbcompression no
        replicaof xxx.xxx.xxx.xxx 6379
        repl-diskless-sync no
        repl-timeout 300
        repl-backlog-size 2gb
        io-threads 3

Comment From: gappymate

Maybe this can help tracking down the issue: https://redis.io/docs/management/optimization/cpu-profiling/ I'd suggest downgrading to an older version if possible, if you cannot find a solution.

Best, gappymate braum-lulu@pptp.de

Comment From: wenglingmei

I tried to reproduce it within the same machine to exclude the impact of the network. With an 11G db, both slaves took about 4mins. Does this issue only happen when the master and slave are in different machines? @polearnik

Comment From: polearnik

In my case it was on different machines. I dont know what i need to check for debugging

Comment From: wenglingmei

In my case it was on different machines. I dont know what i need to check for debugging

Would you mind checking if the problem still exists when slave2 runs on a native machine, instead of a virtual machine? On the other hand, since the CPU usage is 300%, maybe profiling redis server with perf could help.

When I ran the three redis servers in the same machine, the CPU usage are all around 30%-40%.