Dear all,

I have 2 enviroments which have the same business loads, ENV A redis writing is 100ms, while ENV B writeing same data costs 800ms, could you please help to give me some clues and healp me the questions at the end?

Env information

  1. ENV A:
  2. OS: Centos7.9, 3.10.0-1160.el7.x86_64
  3. Server: HPE DL388 Gen10
  4. CPU: Intel Xeon-Silver 4114 (2.2GHz/10-core/85W)*2
  5. Memory: DDR4 32GB X 4（dual）
  6. Disk: SAS 900G 10K X 4, RAID 5
  7. vm.overcommit_memory default
  8. sys/kernel/mm/transparent_hugepage/enabled default
  9. There is no warnning message in /var/log/messages when redis-server starts
  10. redis configuations: appendfsync everysec, repl-backlog-size 1mb, others are the sames with ENV B
  11. redis version：5.0.5
  12. ENV B:
  13. OS: Centos7.9, 3.10.0-957.el7.x86_64
  14. CPU: Silver 4210 (10-Core, 2.2 GHz, 85W) *2
  15. Memory: HPE 32GB Rank x4 DDR4-2933 *4
  16. Disk: HPE 960GB SATA 6G Read Intensive SFF *5, RAID5
  17. vm.overcommit_memory default
  18. sys/kernel/mm/transparent_hugepage/enabled default
  19. There are following 2 warnning messages in /var/log/messages when redis-server starts
  20. Sep 3 10:19:40 localhost redis-server: 6082:M 03 Sep 2022 10:19:40.949 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
  21. Sep 3 10:19:40 localhost redis-server: 6082:M 03 Sep 2022 10:19:40.949 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
  22. redis configuations: appendfsync everysec, repl-backlog-size 10mb, others are the sames with ENV B
  23. redis version：5.0.5

questions:

  1. Is the a seperated subprocess of the AOF persistence while the main process writes the data to memory?
  2. What's the root cause of the slow writings in ENV B, memory issue or disk issue?
  3. It is necessay to set the 'vm.overcommit_memory = 1' and disable the THP? After they are set, how they take efforts?
  4. Why ENV A does not have the warrings?

Thanks in advance. Simon

Comment From: oranagra

@SimonLiuhd a few quick tips: * i don't think you mentioned which version of redis you're using. depending on the version maybe these log warnings are false. * when redis persists. either generates a snapshot (aka BGSAVE), or performs an AOF rewrite (compaction of AOF file), it creates a child process to do that. * overcommit_memory may be necessary in order for that child process to be created. e.g. if redis consumes nearly half the memory of the system. * THP can cause latency when that child process is created.

Comment From: SimonLiuhd

@oranagra Thanks for your reply.

  • The redis version is 5.0.5
  • I understood that the RDB and the AOF rewite are done by a child process. For calling redis and AOF, I think it has 3 steps (exluding the rewrite) :
    1. call (such as set key1 value1)
    1. append the aof_buf
    1. fsync Are the above step b and step c child processes? And where the response for client is generated, after the step a or after step b?

Thanks in advance

Comment From: oranagra

what you described (writing to AOF, not an AOF Rewrite), is don'e without a child process. the appendfsync config controls how the fsync is done, by by default it's set to everysec, in which case the fsync is done by a background thread.

Comment From: SimonLiuhd

@oranagra Thank you very much

A relationed qustion: If the fsyc (appendfsync is configured as everysec) is slow (since the disk is SATA, and has a slow IOPS), why and how does it lead to the redis slow invocation(such as many clients calls "set keyX valueX" at high taffics)?

Comment From: oranagra

that's done on purpose, in order to avoid a case where redis is consuming many writes and is unable to really persist them, it could have eventually create a huge gap between writes that are accepted and what's actually safe on disk. so while the writes don't suffer from the disk latency, they do need to still be bound to the disk throughput.