Hi,

I'm running 3 Redis servers with Sentinel support (3 distinct Vps with SSD Hard disks and 64Gb Ram - Ubuntu 20.04) and use AOF and Rdb persistence mechanisms I use a lot of Redis streams (storing gps data)

Frequently I get errors like "Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis"

When this error happens, it seems the master goes down and a slave becomes the new master. During this process, from my client App, I get error like "READONLY You can‘t write against a read only replica."

It's probably a matter of tuning but I have no idea how to do it.

Some settings of my sentinel.conf sentinel down-after-milliseconds mymaster 5000 sentinel failover-timeout mymaster 60000

Some settings of my redis.conf maxmemory 55gb replica-read-only yes appendfsync everysec no-appendfsync-on-rewrite no auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb

I read some posts where they said to modify vm.dirty_bytes parameters to 33554432 but don't if it's a good point.

Any advice?

Comment From: moticless

Hi @Stan92 - Please verify you didn't disabled aof-rewrite-incremental-fsync. - If you can, please see if this PR #9404 for 7.0 makes the difference? - Your server sometimes overloaded for a few seconds. As such, setting Sentinel with down-after-milliseconds mymaster to 5000msec is too short to trigger often failover. Please try use the default not to abuse failover mechanism.

Comment From: Stan92

@moticless I thank you for your support.. the aof-rewrite-incremental-fsync was not disabled. I increased the down-after-milliseconds to 10000msec (when you try to use default, did u mean 30 sec?) and also reduced the maxmemory to 32gb About the PR #9404 I think I'm too newbie to understand what to do with it.. I'll take a look to see if I get better results. Thanks

Comment From: moticless

My bad, I gave the wrong PR. It is #9409. It is part of 7.0-rc1. Thanks.

Comment From: Jeremy-Run

Is it possible that the disk is busy? Take a look at the aof.c file, make an assumption that if the appendfsync parameter is set to no, this problem is bypassed.

Comment From: moticless

Is it possible that the disk is busy?

Yes. Writing to disk is slower than writing to RAM. fsync() can take some time in a loaded server.

Take a look at the aof.c file, make an assumption that if the appendfsync parameter is set to no, this problem is bypassed.

sure, it might help, as long as you understand the implication. Quote from redis.conf:

The default is "everysec", as that's usually the right compromise between
speed and data safety. It's up to you to understand if you can relax this to
"no" that will let the operating system flush the output buffer when
it wants, for better performances (but if you can live with the idea of
some data loss consider the default persistence mode that's snapshotting),
or on the contrary, use "always" that's very slow but a bit safer than
everysec.

(If selected no, then data will be committed to disk every 30sec for most linux systems)

Comment From: Stan92

@moticless just for the feedback, it seems that the settings do the job.. A quick question, what do you mean by "Please try use the default not to abuse failover mechanism." ? What would be the "correct/max" value for down-after-milliseconds ? And thanks once again for the support

Comment From: moticless

Cool.

Since your setup experiencing unclear latency (and resources?!) issue from the beginning, expecting that all the requests will have round-trip delay no higher than 5000msec (=down-after-milliseconds), might be too harsh which in turn might invoke false-and-often failovers. I would expect to see 10000msec, or less, for steady setups that has no latency (or resources) issues in their network. In case of new setups (and your case) I would use the default (30000msec) or more.