Hi I want to store a snapshot, a .rdb file on another server. Here are my steps 1. change redis config file to appendonly no 2. copy over the .rdb file 3. start redis 4. run redis-cli -p CONFIG SET appendonly yes

I see the following in the log

Starting BGSAVE for SYNC with target: disk
Background saving started by pid 31083
DB saved on disk
RDB: 0 MB of memory used by copy-on-write
Background saving terminated with success
Synchronization with replica 100.10.10.2:35221 succeeded
Background append only file rewriting started by pid 31115
AOF rewrite child asks to stop sending diffs.
Parent agreed to stop sending diffs. Finalizing AOF...
Concatenating 0.00 MB of AOF diff received from parent.
SYNC append only file rewrite performed
AOF rewrite: 0 MB of memory used by copy-on-write

Usually the normal cases, I would see the following after "AOF rewrite: 0 MB of memory used by copy-on-write"

Background AOF rewrite terminated with success
Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
Background AOF rewrite finished successfully

I would like to know 1. how to find out what redis is doing? 2. how to further debug to see what get stuck or what caused the slowness?
3. Is there a way to check the rewrite status? 4. Is there a way to block other process/access until aof rewrite finishes? 5. is there known issue in this area? (I searched but i don't see people have experience such issue)

Thanks Agnes

Comment From: oranagra

@wei0313 it appears as if the main process isn't detecting that the child process terminated.

  • can you please tell us which redis version are you using?
  • can you please check if the child process existed or is it still running? you can do that by searching for a process named redis-aof-rewrite.
  • what do you see when you do INFO persistence?
  • what's the size of mem_aof_buffer in INFO memory?

to try and answer your questions:

  1. if the server is completely unresponsive, maybe using pstack you can see what it's dong.
  2. the above question may help us understand what's going on.
  3. INFO persistence shows aof_rewrite_in_progress that's the only thing we can see.
  4. not sure what you mean
  5. i can't recall any known issue around that area

Comment From: wei0313

Hi @oranagra

redis version : Redis server v=5.0.5 sha=00000000:0 malloc=jemalloc-5.1.0 bits=64 build=5ec67c1f2dc30f72

I checked the child process but i don't see redis-aof-rewrite running. 

[root@dev-vcap1 redis]# bin/redis-cli -p <port>  info memory
# Memory
used_memory:886000
used_memory_human:865.23K
used_memory_rss:4726784
used_memory_rss_human:4.51M
used_memory_peak:926880
used_memory_peak_human:905.16K
used_memory_peak_perc:95.59%
used_memory_overhead:858312
used_memory_startup:791624
used_memory_dataset:27688
used_memory_dataset_perc:29.34%
allocator_allocated:1130056
allocator_active:1437696
allocator_resident:12685312
total_system_memory:33449402368
total_system_memory_human:31.15G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.27
allocator_frag_bytes:307640
allocator_rss_ratio:8.82
allocator_rss_bytes:11247616
rss_overhead_ratio:0.37
rss_overhead_bytes:-7958528
mem_fragmentation_ratio:5.74
mem_fragmentation_bytes:3902712
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:66616
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0

[root@dev-vcap1 redis]# bin/redis-cli  -p <port>  INFO persistence
# Persistence
loading:0
rdb_changes_since_last_save:545
rdb_bgsave_in_progress:0
rdb_last_save_time:1592425796
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0

for 1. server is responsive, I can set value, but the backup was not loaded for 4. I am just curious if there is a way to make redis wait until the backup .rdb is successfully loaded to the database before accepting any other requests

Comment From: wei0313

Hi @oranagra, I found the issue. Apparently, there is another thread killed the redis the same second as it was doing the aof rewrite. Hence, it was never display the terminated with success message. Thanks for looking into it.

Comment From: oranagra

I'm not really sure what you're trying to achieve and how. You say you want to snapshot on another server, are the steps you mention at the top executed on the destination server? Redis does percent writes while loading. I'm not sure how you transfer the rdb file to the destination, and on which server you want to refuse writes?

Comment From: wei0313

I don't need to worry about #4 now. I thought it was some other issues that redis started before the restore snapshot finished. However, it was our code which caused the issue, not redis. thanks @oranagra

Comment From: oranagra

problem was found (outside redis). closing.