I want to process all the data received by master for which I am using PSYNC command to receive all the commands received by master.
<serverIP>:6379> psync 76c9130f7880f33c8323d894dc2491bf8e57f29a 113690954
Entering replica output mode... (press Ctrl-C to quit)
SYNC with master, discarding bytes of bulk transfer until EOF marker...
Error reading RDB payload while SYNCing
I am getting this error when running PSYNC in redis cli. How can I read the bytes transferred by PSYNC command and will it also keep receiving more updates or every time I will have to pull the updates with new offset using PSYNC command? How can I read and process data received by this command? Can anyone please help me with this?
Log from Master
24387:M 07 Sep 2022 08:29:09.685 * Partial resynchronization request from 127.0.0.1:<unknown-replica-port> accepted. Sending 4556 bytes of backlog starting from offset 113690954.
24387:M 07 Sep 2022 08:30:04.023 # Connection with replica 127.0.0.1:<unknown-replica-port> lost.
24387:M 07 Sep 2022 08:30:54.212 * Replica 127.0.0.1:<unknown-replica-port> asks for synchronization
24387:M 07 Sep 2022 08:30:54.212 * Partial resynchronization request from 127.0.0.1:<unknown-replica-port> accepted. Sending 4710 bytes of backlog starting from offset 113690954.
24387:M 07 Sep 2022 08:31:55.870 # Disconnecting timedout replica (streaming sync): 127.0.0.1:<unknown-replica-port>
24387:M 07 Sep 2022 08:31:55.870 # Connection with replica 127.0.0.1:<unknown-replica-port> lost.
Version - Redis server v=7.0.4
Comment From: oranagra
please state which version of redis you're using, and also check the server log file (post the relevant lines from that time)
Comment From: NirmalP29
@oranagra I have updated the description.
Comment From: oranagra
i can confirm there's seem to be a problem with latest redis-cli (older one works, even with latest redis). i don't currently have time to debug it. maybe someone else can step in.
i did also notice two other things:
* doing redis-cli --slave does work, but for some reason it takes longer than it used to take with older redis-cli
* using this psync <id> <offset> does result in full-sync anyway (i think you intended to ask for partial sync and skip generating an rdb file)
Comment From: enjoy-binbin
i tested it in 5.0.14 (both cli and server), it work and then it looks like it's broken since 6.0 (redis-cli) i can take a look when i have times
Comment From: enjoy-binbin
@NirmalP29 for now, you can try to use sync instead of psync if needed, i suppose it will work.
something goes wrong with psync, related to psync2 and the FULLRESYNC reply, needed some times to figure it out, i think redis-cli now dosen't support psync
Comment From: NirmalP29
@enjoy-binbin
psync 76c9130f7880f33c8323d894dc2491bf8e57f29a 113793594
Entering replica output mode... (press Ctrl-C to quit)
SYNC with master, discarding 1197 bytes of bulk transfer...
SYNC done. Logging commands from master.
"ping"
"SELECT","0"
"set","c","key"
For redis-cli v 5.0.14 I am able to get the incremental load but how do I get the existing data? Why this is discarding bulk transfer? Is there any client which I can use to implement redis replication protocol to process data?
Comment From: enjoy-binbin
the existing data is a RDB file, you need to find a tool that can parse this. and this is the discarded data (RDB payload). as far as i can tell, now redis-cli psync or redis-cli sync only support the incremental data, so it will discard the RDB data
Is there any client which I can use to implement redis replication protocol to process data?
sorry, i don't have much experience and practice here. @chenyang8094 maybe you can chime in? and maybe know the tool (RedisShake?) that can help it
Comment From: chenyang8094
@NirmalP29 I don't know the actual scenario you use, but if you want to monitor the data changes of redis for a long time (the monitor and redis do not run on the same machine, otherwise you can directly monitor the update of the AOF file). Then I think the redis-cli tool is not suitable for doing this, compared to what you may need is a CDC (Change Data Capture) tool for sending AOF updates to the remote monitors.
RedisShake is essentially a data synchronization tool. It simulates a complete redis replica to replicate data from the source redis to the target redis you specify. This is not the same as CDC. CDC may not care about the data processing on the target, But RedisShake will force quit when the target redis execution fails. So, if you want to use RedisShake to monitor redis changes, you need to make correct response to RedisShake in your monitor program.
Comment From: NirmalP29
I have implemented socket to capture any change using PSYNC command. Will it impact any performance? I will also try reading from AOF but it might be slow as the CDC service is running on different server. The sequence of commands in PSYNC will be same as master?
Comment From: chenyang8094
I have implemented socket to capture any change using PSYNC command. Will it impact any performance?
Redis uses asynchronous replication by default, so I think this has a minimal impact on performance.
The sequence of commands in PSYNC will be same as master?
The master will rewrite some commands, so the command received by the replica is not necessarily the original command.
Comment From: igxlin
Hi @enjoy-binbin , I created a PR https://github.com/redis/redis/pull/11644 to fix this issue. Could you help review it?
This PR only makes PSYNC fall back to SYNC logic. I am not sure if we are willing to support PSYNC in redis-cli. I will update the PR to support it if so.
Thanks.