I setup my redis cluster on top of kubernetes. And the kuberneters's nodes are AWS's EC2. Recently i have been found that one of my cluster's used_cpu_user has been keep increasing. I used redis-cli's monitor feature check ongoing traffic, but there are no traffic except client's ping. My used_cpu_user is around 20582 without any traffic. Here is my redis cluster's config: Please take a look and give me some clue.
cluster-enabled yes cluster-require-full-coverage no cluster-node-timeout 15000 cluster-replica-validity-factor 0 cluster-config-file /data/nodes.conf cluster-migration-barrier 1 cluster-allow-reads-when-down yes appendonly no no-appendfsync-on-rewrite yes auto-aof-rewrite-min-size 1024mb save 300 100 protected-mode no maxmemory 30gb maxmemory-policy allkeys-lru io-threads 2 logfile "redis.log" repl-backlog-size 512mb activedefrag yes active-defrag-threshold-lower 5 active-defrag-threshold-upper 15
Comment From: smartnews-weitao
Seems related to activedefrag
Comment From: oranagra
@smartnews-weitao please state which version of redis are you using, and maybe also post two snapshots of the INFO ALL output taken one minute apart so we can see the changes in statistics and metrics.
You can try disabling activedefrag and see if it solves the CPU utilization problem, then when we know that, we can look into the fragmentation, figure out where it comes from and if/why active defrag can't fix it.
Maybe the 5% threshold is too low, and the existing >5% fragmentation is located somewhere where the defragger isn't able to fix it, in which case it'll keep working forever and never get anything done. if that's true, increasing the threshold to above the constant non-defraggable value is the right thing to do (that's why the default is 10%, a value that's too low is risky).