We are using Redis 6.0.0. After OOM, I see this log. What is RDB memory usage? 3489 MB is very close to max memory that we have. Does it indicate that we are storing a lot of data in Redis ? Or its just being caused by RDB overhead.

1666:M 01 Jun 2022 19:23:32.268 # Server initialized
1666:M 01 Jun 2022 19:23:32.270 * Loading RDB produced by version 6.0.6
1666:M 01 Jun 2022 19:23:32.270 * RDB age 339 seconds
1666:M 01 Jun 2022 19:23:32.270 * RDB memory usage when created **3489.20 Mb**

So we can we rule out fragmentation being the reason for OOM? Because loading the data from disk itself takes almost max memory?

Comment From: itamarhaber

Hello @vineelyalamarthy

This is the amount of RAM that the server was using when the RDB was created. If your server's RAM is close to 4GB, that would explain why the OOM had been triggered.

Comment From: vineelyalamarthy

so we can

Hello @vineelyalamarthy

This is the amount of RAM that the server was using when the RDB was created. If your server's RAM is close to 4GB, that would explain why the OOM had been triggered.

so can we rule out fragmentation given that just loading the dataset from RDB itself is taking almost max memory?

Comment From: oranagra

What do you mean by "following an OOM"? Did the kernel OOM killer killed the process? 3489mb is what redis was using at the time it saved the RDB file, NOT including (external) fragmentation. if that's close to your maxmemory or not, doesn't mean a lot about physical RAM limit and kernel OOM kill.

Comment From: vineelyalamarthy

close

What do you mean by "following an OOM"? Did the kernel OOM killer killed the process? 3489mb is what redis was using at the time it saved the RDB file, NOT including (external) fragmentation. if that's close to your maxmemory or not, doesn't mean a lot about physical RAM limit and kernel OOM kill.

  • Redis pods are getting rebooted due to OOM in Kubernetes environment.
  • After one of the OOM restarts I saw the log line I mentioned (RDB RDB memory usage when created **3489.20 Mb)
  • Since this is immediately after restart ,fragmentation wouldn't have happened. I am under the impression that fragmentation happens over a period of time.
  • So is it fair to conclude that we got OOM due to real data occupying the whole memory (allocated to the container) and not due to fragmentation.

Comment From: oranagra

the memory usage of 3.4gb wasn't when the node crashed, it was on the last snapshot that was saved. it's also not immediately after restart since that print show the memory usage of the server that saved the rdb, not the one that loads it. either way, this is the used_memory metric of redis, which is how much memory redis allocated (not including external fragmentation). you mentioned that the 3.4gb is close to the maxmemory limit, but i'm not sure if it's also close to the provisioned memory limit in k8s. i suppose you better monitor the RSS metrics to see if fragmentation indeed rises, and then consider either adding more memory, or using active-defrag.

note: keep in mind that not all RSS memory overheads are fragmentation, there are others. so if in fact you see there's an RSS growth, the following metrics may help realizing where the RSS overhead comes from:

            "allocator_frag_ratio:%.2f\r\n"
            "allocator_frag_bytes:%zu\r\n"
            "allocator_rss_ratio:%.2f\r\n"
            "allocator_rss_bytes:%zd\r\n"
            "rss_overhead_ratio:%.2f\r\n"
            "rss_overhead_bytes:%zd\r\n"
            "mem_fragmentation_ratio:%.2f\r\n"
            "mem_fragmentation_bytes:%zd\r\n"