Hi all,
I am running Redis in a Kubernetes environment and see incredibly high memory fragmentation percentages. I’m interested in tuning the jemalloc narenas option but unsure of how to do this for Redis. It doesn’t seem to respect the MALLOC_CONF env var.
curious if others have experience tuning the allocator and whether this specific option has helped address fragmentation as it has in other systems (eg JVM-based applications)
Comment From: oranagra
why do you think narenas will make a difference?
please share some metrics from your redis and we can figure out what the problem is. and maybe enabling activedefrag will help.
please run:
INFO MEMORY
MEMORY MALLOC-STATS
Comment From: lyveng
@oranagra We are seeing a similar issue with redis 5.0.14 which uses jemalloc 5.1.0. We had migrated the application from VMs to k8s. The redis container memory limit was 100G while the maxmemory was configured to be 70G. Over time(about a week) the memory_fragmentation_ratio keeps increasing in k8s and it touches ~1.28 when the redis process gets OOM Killed. Upon debugging, we noticed that the opt.narenas in the output of MEMORY MALLOC-STATS was 384 in k8s while it was 64 in VM. The jemalloc docs describes high value of narenas as a potential reason for higher memory fragmentation. Quoting it here
By default jemalloc uses multiple arenas to reduce internal lock contention. However high arena count may also increase overall memory fragmentation, since arenas manage memory independently. When high degree of parallelism is not expected at the allocator level, lower number of arenas often improves memory usage.
While going through the code of jemalloc, I noticed that the value of opt.narenas is initialized as the number of cpus multiplied by 4(ref). The number of cpus is calculated in malloc_ncpus function as the number of online processors(ref) unless JEMALLOC_GLIBC_MALLOC_HOOK is defined.
In case of VM, it was a 16 core vm. While in k8s, the underlying node was 96 core. This explained why the value of opt.narenas in the output of MEMORY MALLOC-STATS on the VM is 64 while the same is 384. Upon going through the code of jemalloc, I noticed the relevant change was there in this commit done as part of PR jemalloc/jemalloc#1676. The older version of jemalloc seems to initialise the number of cpus as the number of online processors, while this commit changes that to derive the number of cpus from the cpu affinity of the redis process.
For now, I've just rebuilt redis 5.0.14(which internally builds jemalloc) by explicitly setting narenas to 4 in this line
config_malloc_conf="narenas:4"
I'll try deploying this build and validate whether the memory_fragmentation_ratio has reduced now. But the ideal thing would be to use a more recent version of redis.
Comment From: oranagra
it's true that if you allocate from just one arena, you'll get lower fragmentation compared to allocating from multiple arenas, and true the jemalloc created multiple arenas when there are many processors, but the default is to allocate from the same arena on each thread, and thus the first thread only allocates from the first arena.
since redis is mostly single threaded, the other arenas are suppose to be completely idle, and will not cause fragmentation.
(that is, unless you're using some fancy module).
please have a look at your MEMORY MALLOC-STATS and see if that's wrong.
Comment From: lyveng
@oranagra Thanks for pointing out. The output of MEMORY MALLOC-STATS shows the details of only one arena and there is only 1 thread assigned to that arena. I'll dig into why the memory fragmentation ratio is different for the same application deployed on vm vs k8s.
Comment From: lyveng
The issue got resolved once we upgraded redis to 7.2.4. Now the memory fragmentation ratio is under 1.07 for all nodes even after 20 days. It usually used to breach the memory limit within 10 days and get OOMKilled with redis 5.0.14. We didn't spend time debugging why the memory fragmentation ratio was spiking in redis 5.0.14 though.
Comment From: lyveng
@oranagra I was curious about why the memory fragmentation was increasing in 5.0.14. Here is the output of MEMORY MALLOC-STATS on redis 5.0.14.
___ Begin jemalloc statistics ___
Version: "5.1.0-0-g0"
Build-time option settings
config.cache_oblivious: true
config.debug: false
config.fill: true
config.lazy_lock: false
config.malloc_conf: ""
config.prof: false
config.prof_libgcc: false
config.prof_libunwind: false
config.stats: true
config.utrace: false
config.xmalloc: false
Run-time option settings
opt.abort: false
opt.abort_conf: false
opt.retain: true
opt.dss: "secondary"
opt.narenas: 384
opt.percpu_arena: "disabled"
opt.metadata_thp: "disabled"
opt.background_thread: false (background_thread: false)
opt.dirty_decay_ms: 10000 (arenas.dirty_decay_ms: 10000)
opt.muzzy_decay_ms: 10000 (arenas.muzzy_decay_ms: 10000)
opt.junk: "false"
opt.zero: false
opt.tcache: true
opt.lg_tcache_max: 15
opt.thp: "default"
opt.stats_print: false
opt.stats_print_opts: ""
Arenas: 384
Quantum size: 8
Page size: 4096
Maximum thread-cached size class: 32768
Number of bin size classes: 39
Number of thread-cache bin size classes: 44
Number of large size classes: 196
Allocated: 75160124736, active: 78794145792, metadata: 1134788568 (n_thp 0), resident: 79934369792, mapped: 80007569408, retained: 31126806528
Background threads: 0, num_runs: 0, run_interval: 0 ns
n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds
background_thread 24059595 0 0 1 0 0 0
ctl 48145684 0 0 1 0 0 0
prof 0 0 0 0 0 0 0
arenas[0]:
assigned threads: 1
uptime: 1203963185353285
dss allocation precedence: "secondary"
decaying: time npages sweeps madvises purged
dirty: 10000 1291 951936 12820353 138350529
muzzy: 10000 43 894721 11496114 96109794
allocated nmalloc ndalloc nrequests
small: 74262240576 2024728454 1855032042 45857623461
large: 897884160 54081490 54067932 54081490
total: 75160124736 2078809944 1909099974 45911704951
active: 78794145792
mapped: 80007569408
retained: 31126806528
base: 1134755792
internal: 32776
metadata_thp: 0
tcache_bytes: 178544
resident: 79934369792
n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds
large 20815897 0 0 1 0 0 0
extent_avail 72402503 0 0 3 0 0 0
extents_dirty 220262109 0 0 3 0 0 0
extents_muzzy 50956746 0 0 3 0 0 0
extents_retained 49800611 0 0 3 0 0 0
decay_dirty 15153198 0 0 1 0 0 0
decay_muzzy 15095983 0 0 1 0 0 0
base 31361030 0 0 3 0 0 0
tcache_list 12029796 0 0 1 0 0 0
bins: size ind allocated nmalloc ndalloc nrequests curregs curslabs regs pgs util nfills nflushes nslabs nreslabs n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds
8 0 757144 19651593 19556950 27046735 94643 185 512 1 0.999 6026876 4389277 194 1922127 22446152 0 0 1 0 0 0
16 1 677825552 426245351 383881254 3647237036 42364097 181209 256 1 0.913 24400271 6995219 188349 331397863 43620774 0 0 1 0 0 0
24 2 1024452216 431707257 389021748 14077310298 42685509 91118 512 3 0.914 24456554 7043040 94857 332773553 43627985 0 0 1 0 0 0
32 3 197024 6736879 6730722 3554630150 6157 140 128 1 0.343 4184922 3778690 1384 4954102 19996035 0 0 1 0 0 0
40 4 761165480 197041266 178012129 453101270 19029137 38452 512 5 0.966 14367651 5749385 42885 178058207 32194149 0 0 1 0 0 0
48 5 1077313872 273660604 251216565 542982270 22444039 109462 256 3 0.800 16604963 6309899 121018 190246062 35077231 0 0 1 0 0 0
56 6 37464728 27659617 26990604 11557265244 669013 1558 512 7 0.838 11605292 5163307 2884 23777050 28802604 0 0 1 0 0 0
64 7 2501952 22450248 22411155 8401149452 39093 746 64 1 0.818 14796791 5049946 3035 19142071 31881856 0 0 1 0 0 0
80 8 1910720 5957817 5933933 347845174 23884 496 256 5 0.188 3527965 3517888 1336 3985351 19077824 0 0 1 0 0 0
96 9 55968 3046740 3046157 61404181 583 27 128 3 0.168 2391637 2265209 164 1030470 16686942 0 0 1 0 0 0
112 10 2352 4552763 4552742 220489479 21 1 256 7 0.082 3089941 2987129 2 0 18106868 0 0 1 0 0 0
128 11 896 3764241 3764234 132521561 7 1 32 1 0.218 2799338 2772728 1968 207532 17605796 0 0 1 0 0 0
160 12 1280 3716712 3716704 90689300 8 1 128 5 0.062 2765360 2736441 3584 32169 17538763 0 0 1 0 0 0
192 13 576 2251826 2251823 34064079 3 1 64 3 0.046 1659236 1716157 13595 6701 15432377 0 0 1 0 0 0
224 14 224 2546995 2546994 15165532 1 1 128 7 0.007 1970013 1993875 285658 0 16564998 0 0 1 0 0 0
256 15 0 1273996 1273996 4896926 0 0 16 1 1 1250849 1249856 868662 1 16267824 0 0 1 0 0 0
320 16 2541207680 140161060 132219786 237990372 7941274 214367 64 5 0.578 11945622 6132768 260145 78011906 30414108 0 0 1 0 0 0
384 17 110415360 9973661 9686121 13827109 287540 11089 32 3 0.810 3604192 4077175 30767 7558947 19761607 0 0 1 0 0 0
448 18 332795904 15901929 15159081 45531711 742848 12298 64 7 0.943 4565188 4523248 25776 13106541 21157485 0 0 1 0 0 0
512 19 710684160 25048767 23660712 54567056 1388055 173548 8 1 0.999 6485382 5401911 376917 20079447 24497374 0 0 1 0 0 0
640 20 53843200 3787998 3703868 3815547 84130 2657 32 5 0.989 2086207 2488400 9273 3284211 16620291 0 0 1 0 0 0
768 21 837496320 19325655 18235165 73235938 1090490 69811 16 3 0.976 5037220 4785908 94073 19677532 21971258 0 0 1 0 0 0
896 22 3534926080 51663757 47718527 272692029 3945230 127436 32 7 0.967 7903752 5424347 148880 50240272 25528218 0 0 1 0 0 0
1024 23 2008666112 25422214 23460626 135340620 1961588 490414 4 1 0.999 6696415 5341296 610282 25679267 24797656 0 0 1 0 0 0
1280 24 6121985280 50085842 45303041 355064231 4782801 303120 16 5 0.986 9365824 5977340 321788 44226761 27713415 0 0 1 0 0 0
1536 25 7838636544 49989084 44885805 403433906 5103279 644895 8 3 0.989 10820795 6913810 671043 46389506 30461591 0 0 1 0 0 0
1792 26 1953880320 17760102 16669767 88951693 1090335 71243 16 7 0.956 4879109 4691599 90613 18429759 21710486 0 0 1 0 0 0
2048 27 1543837696 13309516 12555689 60844604 753827 377094 2 1 0.999 4363869 4411330 911597 13634593 22251094 0 0 1 0 0 0
2560 28 10922954240 51852793 47586014 256353369 4266779 534241 8 5 0.998 10634651 7113463 668030 42132675 30579728 0 0 1 0 0 0
3072 29 13934662656 48403558 43867535 311146221 4536023 1134060 4 3 0.999 10350432 6786811 1209784 46775289 30452546 0 0 1 0 0 0
3584 30 9138393600 29892511 27342736 199832200 2549775 342047 8 7 0.931 7386254 5656479 376885 27865015 25484251 0 0 1 0 0 0
4096 31 3887562752 14003839 13054727 86302701 949112 949112 1 1 1 4659880 4485137 14003839 0 48233378 0 0 1 0 0 0
5120 32 3058544640 9968436 9371064 54603247 597372 154882 4 5 0.964 3934752 3985028 267809 10004375 20330311 0 0 1 0 0 0
6144 33 651055104 4255547 4149581 12461627 105966 53699 2 3 0.986 2838141 3192292 226334 4003515 18459197 0 0 1 0 0 0
7168 34 384735232 3236884 3183210 7226830 53674 13442 4 7 0.998 2469579 2685221 60293 3063864 17291739 0 0 1 0 0 0
8192 35 318324736 2724462 2685604 5647294 38858 38858 1 2 1 2220583 2358873 2724462 0 22019317 0 0 1 0 0 0
10240 36 481136640 2901558 2854572 7206881 46986 23548 2 5 0.997 2391922 2541369 120095 2738669 17179728 0 0 1 0 0 0
12288 37 210960384 1813798 1796630 2710749 17168 17168 1 3 1 1555140 1611284 1813798 0 18806647 0 0 1 0 0 0
14336 38 101885952 981578 974471 1038839 7107 3561 2 7 0.997 781889 815454 41535 920681 13706647 0 0 1 0 0 0
large: size ind allocated nmalloc ndalloc nrequests curlextents
16384 39 79626240 479755 474895 581989 4860
20480 40 94187520 8450414 8445815 942463333 4599
24576 41 37699584 1893369 1891835 3047674 1534
28672 42 16830464 1439052 1438465 1999548 587
32768 43 11010048 1147960 1147624 1471082 336
40960 44 16343040 27502154 27501755 27502154 399
49152 45 13516800 2130032 2129757 2130032 275
57344 46 15253504 960635 960369 960635 266
65536 47 5570560 911346 911261 911346 85
81920 48 25804800 1758874 1758559 1758874 315
98304 49 10616832 2932593 2932485 2932593 108
114688 50 6766592 980747 980688 980747 59
131072 51 4718592 524015 523979 524015 36
163840 52 7536640 1242625 1242579 1242625 46
196608 53 3145728 589062 589046 589062 16
229376 54 3211264 380956 380942 380956 14
262144 55 2097152 325394 325386 325394 8
327680 56 1638400 63286 63281 63286 5
393216 57 393216 42490 42489 42490 1
458752 58 458752 31248 31247 31248 1
524288 59 2097152 89739 89735 89739 4
655360 60 655360 18669 18668 18669 1
786432 61 786432 180963 180962 180963 1
917504 62 0 460 460 460 0
1048576 63 1048576 2667 2666 2667 1
1310720 64 0 1993 1993 1993 0
1572864 65 0 938 938 938 0
---
2097152 67 0 1 1 1 0
2621440 68 0 51 51 51 0
---
268435456 95 0 1 1 1 0
---
536870912 99 536870912 1 0 1 1
---
--- End jemalloc statistics ---
I noticed that you mentioned that the Retained value being high is due to THP being enabled here. In our case as well we have THP enabled and the Retained is ~28G which is close to the mem_fragmentation_bytes. We have not set the config disable-thp in redis. Is there some indicator in the output of MEMORY MALLOC-STATS that indicate that thp is the problem that is causing the fragmentation?
If so, is there anything that can explain why the fragmentation issue got fixed when we upgraded to redis 7.2.4. One thing I noticed is that config.cache_oblivious is set to true in 5.0.14 while it is set to false in 7.2.4. This is done as part of PR #12315 .
There is no need to debug this. It is just that curiosity is killing me.
Comment From: sundb
@lyveng could you also provide the output of MEMORY MALLOC-STATS in 7.2.4?
at the same time i saw most of fragmentation is in size class 320(~1G).
Comment From: lyveng
Here is the output from 7.2.4 redis container which is running for ~21 days
Defaulted container "redis" out of: redis, zulu-service, mtl-sidecar, cfgsvc-dns-proxy
I have no name!@zulu-service-hyd-prod-vb4-7f5c6d6f86-sgjzz:/data$ redis-cli
127.0.0.1:6379> MEMORY MALLOC-STATS
___ Begin jemalloc statistics ___
Version: "5.3.0-0-g0"
Build-time option settings
config.cache_oblivious: false
config.debug: false
config.fill: true
config.lazy_lock: false
config.malloc_conf: ""
config.opt_safety_checks: false
config.prof: false
config.prof_libgcc: false
config.prof_libunwind: false
config.stats: true
config.utrace: false
config.xmalloc: false
Run-time option settings
opt.abort: false
opt.abort_conf: false
opt.cache_oblivious: false
opt.confirm_conf: false
opt.retain: true
opt.dss: "secondary"
opt.narenas: 1
opt.percpu_arena: "disabled"
opt.oversize_threshold: 8388608
opt.hpa: false
opt.hpa_slab_max_alloc: 65536
opt.hpa_hugification_threshold: 1992294
opt.hpa_hugify_delay_ms: 10000
opt.hpa_min_purge_interval_ms: 5000
opt.hpa_dirty_mult: "0.25"
opt.hpa_sec_nshards: 4
opt.hpa_sec_max_alloc: 32768
opt.hpa_sec_max_bytes: 262144
opt.hpa_sec_bytes_after_flush: 131072
opt.hpa_sec_batch_fill_extra: 0
opt.metadata_thp: "disabled"
opt.mutex_max_spin: 600
opt.background_thread: false (background_thread: true)
opt.dirty_decay_ms: 10000 (arenas.dirty_decay_ms: 10000)
opt.muzzy_decay_ms: 0 (arenas.muzzy_decay_ms: 0)
opt.lg_extent_max_active_fit: 6
opt.junk: "false"
opt.zero: false
opt.tcache: true
opt.tcache_max: 32768
opt.tcache_nslots_small_min: 20
opt.tcache_nslots_small_max: 200
opt.tcache_nslots_large: 20
opt.lg_tcache_nslots_mul: 1
opt.tcache_gc_incr_bytes: 65536
opt.tcache_gc_delay_bytes: 0
opt.lg_tcache_flush_small_div: 1
opt.lg_tcache_flush_large_div: 1
opt.thp: "default"
opt.stats_print: false
opt.stats_print_opts: ""
opt.stats_print: false
opt.stats_print_opts: ""
opt.stats_interval: -1
opt.stats_interval_opts: ""
opt.zero_realloc: "free"
Arenas: 2
Quantum size: 8
Page size: 4096
Maximum thread-cached size class: 32768
Number of bin size classes: 39
Number of thread-cache bin size classes: 44
Number of large size classes: 196
Allocated: 75162186624, active: 77501292544, metadata: 1064106080 (n_thp 0), resident: 78567763968, mapped: 78713835520, retained: 17779273728
Count of realloc(non-null-ptr, 0) calls: 0
Background threads: 1, num_runs: 196544, run_interval: 9899234542 ns
n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
background_thread 38727764 19 0 0 0 0 1 0 0 0 0 0
max_per_bg_thd 21145133 10 0 0 0 0 3 0 0 0 0 0
ctl 77473615 39 0 0 0 0 1 0 0 0 0 0
prof 0 0 0 0 0 0 0 0 0 0 0 0
prof_thds_data 0 0 0 0 0 0 0 0 0 0 0 0
prof_dump 0 0 0 0 0 0 0 0 0 0 0 0
prof_recent_alloc 0 0 0 0 0 0 0 0 0 0 0 0
prof_recent_dump 0 0 0 0 0 0 0 0 0 0 0 0
prof_stats 0 0 0 0 0 0 0 0 0 0 0 0
Merged arenas stats:
assigned threads: 2
uptime: 1945734017456208
dss allocation precedence: "N/A"
decaying: time npages sweeps madvises purged
dirty: N/A 607 192661 9379296 55909212
muzzy: N/A 0 0 0 0
allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nfill (#/sec) nflush (#/sec)
small: 74245817216 1661799210 854 1485679625 763 46140933538 23713 445707602 229 273172957 140
large: 916369408 42480472 21 42466070 21 1331169672 684 42480472 21 14733811 7
total: 75162186624 1704279682 875 1528145695 785 47472103210 24398 488188074 250 287906768 147
active: 77501292544
mapped: 78713835520
retained: 17779273728
base: 1063975008
internal: 131072
metadata_thp: 0
tcache_bytes: 163008
tcache_stashed_bytes: 0
resident: 78567763968
abandoned_vm: 0
extent_avail: 39624
n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
large 38727761 19 0 0 0 0 2 0 0 0 0 0
extent_avail 104838589 53 0 0 0 0 382754 0 0 0 0 0
extents_dirty 190799546 98 16 0 0 0 385326 0 0 0 0 1
extents_muzzy 43081659 22 0 0 0 0 2 0 0 0 0 0
extents_retained 58775889 30 1 0 0 0 385324 0 0 0 0 1
decay_dirty 43180189 22 0 0 0 0 392222 0 0 0 0 0
decay_muzzy 38924305 20 0 0 0 0 392200 0 0 0 0 0
base 84437910 43 0 0 0 0 2 0 0 0 0 0
tcache_list 38727763 19 0 0 0 0 4 0 0 0 0 0
hpa_shard 0 0 0 0 0 0 0 0 0 0 0 0
hpa_shard_grow 0 0 0 0 0 0 0 0 0 0 0 0
hpa_sec 0 0 0 0 0 0 0 0 0 0 0 0
bins: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nshards curregs curslabs nonfull_slabs regs pgs util nfills (#/sec) nflushes (#/sec) nslabs nreslabs (#/sec) n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
8 0 5968 8004467 4 8003721 4 6831505 3 1 746 2 0 512 1 0.728 2260498 1 2084667 1 2 105498 0 43072928 22 0 0 0 0 2 0 0 0 0 0
16 1 704653296 310663384 159 266622553 137 4587981766 2357 1 44040831 177793 22944 256 1 0.967 47999067 24 12400148 6 190222 212177452 109 99317198 51 0 0 0 0 2 0 0 0 0 0
24 2 1061954064 311515107 160 267267021 137 10207508943 5246 1 44248086 89195 9899 512 3 0.968 47629668 24 12399418 6 95312 212004987 108 98852159 50 0 0 0 0 2 0 0 0 0 0
32 3 179776 1570031 0 1564413 0 4581096974 2354 1 5618 58 37 128 1 0.756 888995 0 895423 0 804 658355 0 40512983 20 0 0 0 0 2 0 0 0 0 0
40 4 797116720 144153970 74 124226052 63 416415671 214 1 19927918 39437 576 512 5 0.986 26656163 13 12137641 6 45714 104000144 53 77567279 39 0 0 0 0 2 0 0 0 0 0
48 5 1116796848 209997671 107 186731070 95 527494267 271 1 23266601 99508 33900 256 3 0.913 32894033 16 12366418 6 117940 110619489 56 84106152 43 0 0 0 0 2 0 0 0 0 0
56 6 33528656 50520573 25 49921847 25 12660342748 6506 1 598726 1323 365 512 7 0.883 29165287 14 11354514 5 2396 49726595 25 79249958 40 0 0 0 0 2 0 0 0 0 0
64 7 26688 49005203 25 49004786 25 8829489031 4537 1 417 76 75 64 1 0.085 33169982 17 11125958 5 2485 34938894 17 83026186 42 0 0 0 0 2 0 0 0 0 0
80 8 280880 1312021 0 1308510 0 368445388 189 1 3511 20 16 256 5 0.685 856192 0 1094115 0 469 922880 0 40678537 20 0 0 0 0 2 0 0 0 0 0
96 9 1632 1560626 0 1560609 0 61411941 31 1 17 1 0 128 3 0.132 1559234 0 1559793 0 1 0 0 41846789 21 0 0 0 0 2 0 0 0 0 0
112 10 605360 41868 0 36463 0 236051428 121 1 5405 22 2 256 7 0.959 19707 0 25165 0 69 8518 0 38772702 19 0 0 0 0 2 0 0 0 0 0
128 11 28800 68877 0 68652 0 142144650 73 1 225 8 1 32 1 0.878 68576 0 68626 0 9 2 0 38864972 19 0 0 0 0 2 0 0 0 0 0
160 12 1760 426125 0 426114 0 92230017 47 1 11 1 0 128 5 0.085 424551 0 425204 0 1 0 0 39577517 20 0 0 0 0 2 0 0 0 0 0
192 13 1158528 3420523 1 3414489 1 37453508 19 1 6034 95 4 64 3 0.992 3366653 1 3393725 1 354 149252 0 45488493 23 0 0 0 0 2 0 0 0 0 0
224 14 672 5387839 2 5387836 2 15330206 7 1 3 1 0 128 7 0.023 5387318 2 5387355 2 1 0 0 49502435 25 0 0 0 0 2 0 0 0 0 0
256 15 768 1567446 0 1567443 0 6438643 3 1 3 1 0 16 1 0.187 1567210 0 1567211 0 1 0 0 41862183 21 0 0 0 0 2 0 0 0 0 0
320 16 2932885120 122558511 62 113393245 58 262417167 134 1 9165266 174188 82494 64 5 0.822 27957918 14 12427861 6 226100 55643710 28 79339640 40 0 0 0 0 2 0 0 0 0 0
384 17 157544832 12714535 6 12304262 6 21011263 10 1 410273 14478 1919 32 3 0.885 6643276 3 7615139 3 37497 10421123 5 53023673 27 0 0 0 0 2 0 0 0 0 0
448 18 334952128 16881996 8 16134335 8 58834235 30 1 747661 13326 2096 64 7 0.876 8124467 4 8920823 4 28987 13795848 7 55802038 28 0 0 0 0 2 0 0 0 0 0
512 19 644537856 24863932 12 23605069 12 59523909 30 1 1258863 161793 14032 8 1 0.972 10350240 5 10131701 5 297564 21628407 11 59507266 30 0 0 0 0 2 0 0 0 0 0
640 20 55880960 4849288 2 4761974 2 5055650 2 1 87314 2781 272 32 5 0.981 3152914 1 3688830 1 8585 4408393 2 45578090 23 0 0 0 0 2 0 0 0 0 0
768 21 906280704 19483641 10 18303588 9 88449678 45 1 1180053 73761 119 16 3 0.999 8563901 4 9123330 4 110805 17879647 9 56525797 29 0 0 0 0 2 0 0 0 0 0
896 22 3849479424 44503874 22 40207580 20 344843798 177 1 4296294 134261 56 32 7 0.999 13552692 6 10848640 5 168742 39058538 20 63297835 32 0 0 0 0 2 0 0 0 0 0
1024 23 2113713152 27406098 14 25341925 13 179936439 92 1 2064173 517276 4383 4 1 0.997 10561644 5 10214122 5 652728 26047444 13 60156255 30 0 0 0 0 2 0 0 0 0 0
1280 24 6931825920 48570147 24 43154658 22 480623087 247 1 5415489 339416 3639 16 5 0.997 15742404 8 11169697 5 385242 42974926 22 66025104 33 0 0 0 0 2 0 0 0 0 0
1536 25 7826273280 43626622 22 38531392 19 526161142 270 1 5095230 657443 31002 8 3 0.968 15480326 7 11119031 5 702340 39276159 20 66029458 33 0 0 0 0 2 0 0 0 0 0
1792 26 1914271744 16338257 8 15270025 7 111577299 57 1 1068232 66885 204 16 7 0.998 7674019 3 8373049 4 92645 15348441 7 54867474 28 0 0 0 0 2 0 0 0 0 0
2048 27 1466439680 15593205 8 14877170 7 79758443 40 1 716035 358046 57 2 1 0.999 7306059 3 8650424 4 728370 15135826 7 55412614 28 0 0 0 0 2 0 0 0 0 0
2560 28 9336739840 40412623 20 36765459 18 273909332 140 1 3647164 481626 66305 8 5 0.946 14114835 7 11065483 5 606397 35154397 18 64514476 33 0 0 0 0 2 0 0 0 0 0
3072 29 13573023744 40522389 20 36104087 18 366286187 188 1 4418302 1118139 44496 4 3 0.987 14100916 7 10896397 5 1269726 37701517 19 64994800 33 0 0 0 0 2 0 0 0 0 0
3584 30 9460745728 29128669 14 26488952 13 258149966 132 1 2639717 342211 36821 8 7 0.964 11254479 5 10130453 5 402810 26792847 13 60515503 31 0 0 0 0 2 0 0 0 0 0
4096 31 3990835200 15036503 7 14062178 7 119389306 61 1 974325 974325 0 1 1 1 7416606 3 8203721 4 15036503 0 0 69384591 35 0 0 0 0 2 0 0 0 0 0
5120 32 2852070400 10609582 5 10052537 5 72681356 37 1 557045 144743 8711 4 5 0.962 6232182 3 7121307 3 245123 10087039 5 52326373 26 0 0 0 0 2 0 0 0 0 0
6144 33 664098816 6794359 3 6686270 3 16682314 8 1 108089 54084 79 2 3 0.999 5425657 2 5817654 2 139774 6624093 3 50110846 25 0 0 0 0 2 0 0 0 0 0
7168 34 382606336 5037268 2 4983891 2 9664119 4 1 53377 13347 10 4 7 0.999 4261933 2 4450780 2 39037 4913525 2 47479511 24 0 0 0 0 2 0 0 0 0 0
8192 35 328687616 6394408 3 6354285 3 9854348 5 1 40123 40123 0 1 2 1 4349745 2 5059899 2 6394408 0 0 54531813 28 0 0 0 0 2 0 0 0 0 0
10240 36 486768640 5706068 2 5658532 2 11881243 6 1 47536 23777 18 2 5 0.999 4754660 2 4949184 2 105079 5369617 2 48536684 24 0 0 0 0 2 0 0 0 0 0
12288 37 220495872 3834028 1 3816084 1 5571560 2 1 17944 17944 0 1 3 1 3273397 1 3367774 1 3834028 0 0 49202960 25 0 0 0 0 2 0 0 0 0 0
14336 38 99319808 1717476 0 1710548 0 2005011 1 1 6928 3478 28 2 7 0.995 1500198 0 1542277 0 30362 1446903 0 41800598 21 0 0 0 0 2 0 0 0 0 0
large: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) curlextents
16384 39 92389376 2627997 1 2622358 1 3533044 1 5639
20480 40 96337920 20188578 10 20183874 10 1307559996 672 4704
24576 41 34996224 1542737 0 1541313 0 1758883 0 1424
28672 42 19468288 1265090 0 1264411 0 1406694 0 679
32768 43 10452992 758762 0 758443 0 813747 0 319
40960 44 14254080 1281480 0 1281132 0 1281480 0 348
49152 45 13762560 1311543 0 1311263 0 1311543 0 280
57344 46 13475840 5475054 2 5474819 2 5475054 2 235
65536 47 7798784 1674889 0 1674770 0 1674889 0 119
81920 48 27033600 2212723 1 2212393 1 2212723 1 330
98304 49 10321920 1040247 0 1040142 0 1040247 0 105
114688 50 5160960 651640 0 651595 0 651640 0 45
131072 51 6422528 512633 0 512584 0 512633 0 49
163840 52 8847360 748652 0 748598 0 748652 0 54
196608 53 5505024 454533 0 454505 0 454533 0 28
229376 54 4358144 246085 0 246066 0 246085 0 19
262144 55 1310720 123118 0 123113 0 123118 0 5
327680 56 3604480 205672 0 205661 0 205672 0 11
393216 57 1966080 60804 0 60799 0 60804 0 5
458752 58 458752 2339 0 2338 0 2339 0 1
---
655360 60 655360 73819 0 73818 0 73819 0 1
786432 61 0 5221 0 5221 0 5221 0 0
917504 62 917504 16854 0 16853 0 16854 0 1
---
268435456 95 0 1 0 1 0 1 0 0
---
536870912 99 536870912 1 0 0 0 1 0 1
---
extents: size ind ndirty dirty nmuzzy muzzy nretained retained ntotal total
4096 0 34 139264 0 0 382433 1566445568 382467 1566584832
8192 1 8 65536 0 0 198024 1622212608 198032 1622278144
12288 2 6 73728 0 0 100389 1233580032 100395 1233653760
16384 3 4 65536 0 0 66423 1088274432 66427 1088339968
20480 4 10 204800 0 0 9371 191918080 9381 192122880
24576 5 0 0 0 0 4765 117104640 4765 117104640
28672 6 0 0 0 0 4574 131145728 4574 131145728
32768 7 6 200704 0 0 7511 261169152 7517 261369856
40960 8 2 81920 0 0 8402 357789696 8404 357871616
49152 9 3 147456 0 0 5415 276668416 5418 276815872
57344 10 0 0 0 0 1885 111996928 1885 111996928
65536 11 0 0 0 0 2186 156086272 2186 156086272
81920 12 10 819200 0 0 291 25346048 301 26165248
98304 13 1 98304 0 0 234 24383488 235 24481792
114688 14 0 0 0 0 200 24137728 200 24137728
131072 15 0 0 0 0 326 46899200 326 46899200
163840 16 1 163840 0 0 206 36245504 207 36409344
196608 17 1 196608 0 0 126 26578944 127 26775552
229376 18 1 229376 0 0 71 17244160 72 17473536
262144 19 0 0 0 0 137 39718912 137 39718912
327680 20 0 0 0 0 58 20738048 58 20738048
393216 21 0 0 0 0 32 13492224 32 13492224
458752 22 0 0 0 0 29 14155776 29 14155776
524288 23 0 0 0 0 48 27873280 48 27873280
655360 24 0 0 0 0 9 6406144 9 6406144
786432 25 0 0 0 0 7 5959680 7 5959680
---
1048576 27 0 0 0 0 3 3551232 3 3551232
1310720 28 0 0 0 0 2 2842624 2 2842624
1572864 29 0 0 0 0 1 1830912 1 1830912
---
2097152 31 0 0 0 0 2 4767744 2 4767744
2621440 32 0 0 0 0 3 8437760 3 8437760
3145728 33 0 0 0 0 1 3665920 1 3665920
---
4194304 35 0 0 0 0 3 13004800 3 13004800
---
6291456 37 0 0 0 0 1 7094272 1 7094272
7340032 38 0 0 0 0 1 7753728 1 7753728
---
268435456 59 0 0 0 0 1 268435456 1 268435456
---
8589934592 79 0 0 0 0 1 10014318592 1 10014318592
---
Bytes in small extent cache: 0
HPA shard stats:
Purge passes: 0 (0 / sec)
Purges: 0 (0 / sec)
Hugeifies: 0 (0 / sec)
Dehugifies: 0 (0 / sec)
In full slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
In empty slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
size ind npageslabs_huge nactive_huge ndirty_huge npageslabs_nonhuge nactive_nonhuge ndirty_nonhuge nretained_nonhuge
---
arenas[0]:
assigned threads: 2
uptime: 1945734017456208
dss allocation precedence: "secondary"
decaying: time npages sweeps madvises purged
dirty: 10000 607 192661 9379295 55843676
muzzy: 0 0 0 0 0
allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nfill (#/sec) nflush (#/sec)
small: 74245817216 1661799210 854 1485679625 763 46140933538 23713 445707602 229 273172957 140
large: 379498496 42480470 21 42466069 21 1331169670 684 42480470 21 14733811 7
total: 74625315712 1704279680 875 1528145694 785 47472103208 24398 488188072 250 287906768 147
active: 76964421632
mapped: 78174867456
retained: 17510838272
base: 1063882896
internal: 131072
metadata_thp: 0
tcache_bytes: 163008
tcache_stashed_bytes: 0
resident: 78030798848
abandoned_vm: 0
extent_avail: 39624
n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
large 19363881 9 0 0 0 0 1 0 0 0 0 0
extent_avail 85474707 43 0 0 0 0 382753 0 0 0 0 0
extents_dirty 171435663 88 16 0 0 0 385325 0 0 0 0 1
extents_muzzy 23717779 12 0 0 0 0 1 0 0 0 0 0
extents_retained 39412006 20 1 0 0 0 385323 0 0 0 0 1
decay_dirty 23816307 12 0 0 0 0 392221 0 0 0 0 0
decay_muzzy 19560425 10 0 0 0 0 392199 0 0 0 0 0
base 45710147 23 0 0 0 0 1 0 0 0 0 0
tcache_list 19363883 9 0 0 0 0 3 0 0 0 0 0
hpa_shard 0 0 0 0 0 0 0 0 0 0 0 0
hpa_shard_grow 0 0 0 0 0 0 0 0 0 0 0 0
hpa_sec 0 0 0 0 0 0 0 0 0 0 0 0
bins: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nshards curregs curslabs nonfull_slabs regs pgs util nfills (#/sec) nflushes (#/sec) nslabs nreslabs (#/sec) n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
8 0 5968 8004467 4 8003721 4 6831505 3 1 746 2 0 512 1 0.728 2260498 1 2084667 1 2 105498 0 23709048 12 0 0 0 0 1 0 0 0 0 0
16 1 704653296 310663384 159 266622553 137 4587981766 2357 1 44040831 177793 22944 256 1 0.967 47999067 24 12400148 6 190222 212177452 109 79953318 41 0 0 0 0 1 0 0 0 0 0
24 2 1061954064 311515107 160 267267021 137 10207508943 5246 1 44248086 89195 9899 512 3 0.968 47629668 24 12399418 6 95312 212004987 108 79488279 40 0 0 0 0 1 0 0 0 0 0
32 3 179776 1570031 0 1564413 0 4581096974 2354 1 5618 58 37 128 1 0.756 888995 0 895423 0 804 658355 0 21149103 10 0 0 0 0 1 0 0 0 0 0
40 4 797116720 144153970 74 124226052 63 416415671 214 1 19927918 39437 576 512 5 0.986 26656163 13 12137641 6 45714 104000144 53 58203399 29 0 0 0 0 1 0 0 0 0 0
48 5 1116796848 209997671 107 186731070 95 527494267 271 1 23266601 99508 33900 256 3 0.913 32894033 16 12366418 6 117940 110619489 56 64742272 33 0 0 0 0 1 0 0 0 0 0
56 6 33528656 50520573 25 49921847 25 12660342748 6506 1 598726 1323 365 512 7 0.883 29165287 14 11354514 5 2396 49726595 25 59886078 30 0 0 0 0 1 0 0 0 0 0
64 7 26688 49005203 25 49004786 25 8829489031 4537 1 417 76 75 64 1 0.085 33169982 17 11125958 5 2485 34938894 17 63662306 32 0 0 0 0 1 0 0 0 0 0
80 8 280880 1312021 0 1308510 0 368445388 189 1 3511 20 16 256 5 0.685 856192 0 1094115 0 469 922880 0 21314657 10 0 0 0 0 1 0 0 0 0 0
96 9 1632 1560626 0 1560609 0 61411941 31 1 17 1 0 128 3 0.132 1559234 0 1559793 0 1 0 0 22482909 11 0 0 0 0 1 0 0 0 0 0
112 10 605360 41868 0 36463 0 236051428 121 1 5405 22 2 256 7 0.959 19707 0 25165 0 69 8518 0 19408822 9 0 0 0 0 1 0 0 0 0 0
128 11 28800 68877 0 68652 0 142144650 73 1 225 8 1 32 1 0.878 68576 0 68626 0 9 2 0 19501092 10 0 0 0 0 1 0 0 0 0 0
160 12 1760 426125 0 426114 0 92230017 47 1 11 1 0 128 5 0.085 424551 0 425204 0 1 0 0 20213637 10 0 0 0 0 1 0 0 0 0 0
192 13 1158528 3420523 1 3414489 1 37453508 19 1 6034 95 4 64 3 0.992 3366653 1 3393725 1 354 149252 0 26124613 13 0 0 0 0 1 0 0 0 0 0
224 14 672 5387839 2 5387836 2 15330206 7 1 3 1 0 128 7 0.023 5387318 2 5387355 2 1 0 0 30138555 15 0 0 0 0 1 0 0 0 0 0
256 15 768 1567446 0 1567443 0 6438643 3 1 3 1 0 16 1 0.187 1567210 0 1567211 0 1 0 0 22498303 11 0 0 0 0 1 0 0 0 0 0
320 16 2932885120 122558511 62 113393245 58 262417167 134 1 9165266 174188 82494 64 5 0.822 27957918 14 12427861 6 226100 55643710 28 59975760 30 0 0 0 0 1 0 0 0 0 0
384 17 157544832 12714535 6 12304262 6 21011263 10 1 410273 14478 1919 32 3 0.885 6643276 3 7615139 3 37497 10421123 5 33659793 17 0 0 0 0 1 0 0 0 0 0
448 18 334952128 16881996 8 16134335 8 58834235 30 1 747661 13326 2096 64 7 0.876 8124467 4 8920823 4 28987 13795848 7 36438158 18 0 0 0 0 1 0 0 0 0 0
512 19 644537856 24863932 12 23605069 12 59523909 30 1 1258863 161793 14032 8 1 0.972 10350240 5 10131701 5 297564 21628407 11 40143386 20 0 0 0 0 1 0 0 0 0 0
640 20 55880960 4849288 2 4761974 2 5055650 2 1 87314 2781 272 32 5 0.981 3152914 1 3688830 1 8585 4408393 2 26214210 13 0 0 0 0 1 0 0 0 0 0
768 21 906280704 19483641 10 18303588 9 88449678 45 1 1180053 73761 119 16 3 0.999 8563901 4 9123330 4 110805 17879647 9 37161917 19 0 0 0 0 1 0 0 0 0 0
896 22 3849479424 44503874 22 40207580 20 344843798 177 1 4296294 134261 56 32 7 0.999 13552692 6 10848640 5 168742 39058538 20 43933955 22 0 0 0 0 1 0 0 0 0 0
1024 23 2113713152 27406098 14 25341925 13 179936439 92 1 2064173 517276 4383 4 1 0.997 10561644 5 10214122 5 652728 26047444 13 40792375 20 0 0 0 0 1 0 0 0 0 0
1280 24 6931825920 48570147 24 43154658 22 480623087 247 1 5415489 339416 3639 16 5 0.997 15742404 8 11169697 5 385242 42974926 22 46661224 23 0 0 0 0 1 0 0 0 0 0
1536 25 7826273280 43626622 22 38531392 19 526161142 270 1 5095230 657443 31002 8 3 0.968 15480326 7 11119031 5 702340 39276159 20 46665578 23 0 0 0 0 1 0 0 0 0 0
1792 26 1914271744 16338257 8 15270025 7 111577299 57 1 1068232 66885 204 16 7 0.998 7674019 3 8373049 4 92645 15348441 7 35503594 18 0 0 0 0 1 0 0 0 0 0
2048 27 1466439680 15593205 8 14877170 7 79758443 40 1 716035 358046 57 2 1 0.999 7306059 3 8650424 4 728370 15135826 7 36048734 18 0 0 0 0 1 0 0 0 0 0
2560 28 9336739840 40412623 20 36765459 18 273909332 140 1 3647164 481626 66305 8 5 0.946 14114835 7 11065483 5 606397 35154397 18 45150596 23 0 0 0 0 1 0 0 0 0 0
3072 29 13573023744 40522389 20 36104087 18 366286187 188 1 4418302 1118139 44496 4 3 0.987 14100916 7 10896397 5 1269726 37701517 19 45630920 23 0 0 0 0 1 0 0 0 0 0
3584 30 9460745728 29128669 14 26488952 13 258149966 132 1 2639717 342211 36821 8 7 0.964 11254479 5 10130453 5 402810 26792847 13 41151623 21 0 0 0 0 1 0 0 0 0 0
4096 31 3990835200 15036503 7 14062178 7 119389306 61 1 974325 974325 0 1 1 1 7416606 3 8203721 4 15036503 0 0 50020711 25 0 0 0 0 1 0 0 0 0 0
5120 32 2852070400 10609582 5 10052537 5 72681356 37 1 557045 144743 8711 4 5 0.962 6232182 3 7121307 3 245123 10087039 5 32962493 16 0 0 0 0 1 0 0 0 0 0
6144 33 664098816 6794359 3 6686270 3 16682314 8 1 108089 54084 79 2 3 0.999 5425657 2 5817654 2 139774 6624093 3 30746966 15 0 0 0 0 1 0 0 0 0 0
7168 34 382606336 5037268 2 4983891 2 9664119 4 1 53377 13347 10 4 7 0.999 4261933 2 4450780 2 39037 4913525 2 28115631 14 0 0 0 0 1 0 0 0 0 0
8192 35 328687616 6394408 3 6354285 3 9854348 5 1 40123 40123 0 1 2 1 4349745 2 5059899 2 6394408 0 0 35167933 18 0 0 0 0 1 0 0 0 0 0
10240 36 486768640 5706068 2 5658532 2 11881243 6 1 47536 23777 18 2 5 0.999 4754660 2 4949184 2 105079 5369617 2 29172804 14 0 0 0 0 1 0 0 0 0 0
12288 37 220495872 3834028 1 3816084 1 5571560 2 1 17944 17944 0 1 3 1 3273397 1 3367774 1 3834028 0 0 29839080 15 0 0 0 0 1 0 0 0 0 0
14336 38 99319808 1717476 0 1710548 0 2005011 1 1 6928 3478 28 2 7 0.995 1500198 0 1542277 0 30362 1446903 0 22436718 11 0 0 0 0 1 0 0 0 0 0
large: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) curlextents
16384 39 92389376 2627997 1 2622358 1 3533044 1 5639
20480 40 96337920 20188578 10 20183874 10 1307559996 672 4704
24576 41 34996224 1542737 0 1541313 0 1758883 0 1424
28672 42 19468288 1265090 0 1264411 0 1406694 0 679
32768 43 10452992 758762 0 758443 0 813747 0 319
40960 44 14254080 1281480 0 1281132 0 1281480 0 348
49152 45 13762560 1311543 0 1311263 0 1311543 0 280
57344 46 13475840 5475054 2 5474819 2 5475054 2 235
65536 47 7798784 1674889 0 1674770 0 1674889 0 119
81920 48 27033600 2212723 1 2212393 1 2212723 1 330
98304 49 10321920 1040247 0 1040142 0 1040247 0 105
114688 50 5160960 651640 0 651595 0 651640 0 45
131072 51 6422528 512633 0 512584 0 512633 0 49
163840 52 8847360 748652 0 748598 0 748652 0 54
196608 53 5505024 454533 0 454505 0 454533 0 28
229376 54 4358144 246085 0 246066 0 246085 0 19
262144 55 1310720 123118 0 123113 0 123118 0 5
327680 56 3604480 205672 0 205661 0 205672 0 11
393216 57 1966080 60804 0 60799 0 60804 0 5
458752 58 458752 2339 0 2338 0 2339 0 1
---
655360 60 655360 73819 0 73818 0 73819 0 1
786432 61 0 5221 0 5221 0 5221 0 0
917504 62 917504 16854 0 16853 0 16854 0 1
---
extents: size ind ndirty dirty nmuzzy muzzy nretained retained ntotal total
4096 0 34 139264 0 0 382433 1566445568 382467 1566584832
8192 1 8 65536 0 0 198024 1622212608 198032 1622278144
12288 2 6 73728 0 0 100389 1233580032 100395 1233653760
16384 3 4 65536 0 0 66423 1088274432 66427 1088339968
20480 4 10 204800 0 0 9371 191918080 9381 192122880
24576 5 0 0 0 0 4765 117104640 4765 117104640
28672 6 0 0 0 0 4574 131145728 4574 131145728
32768 7 6 200704 0 0 7511 261169152 7517 261369856
40960 8 2 81920 0 0 8402 357789696 8404 357871616
49152 9 3 147456 0 0 5415 276668416 5418 276815872
57344 10 0 0 0 0 1885 111996928 1885 111996928
65536 11 0 0 0 0 2186 156086272 2186 156086272
81920 12 10 819200 0 0 291 25346048 301 26165248
98304 13 1 98304 0 0 234 24383488 235 24481792
114688 14 0 0 0 0 200 24137728 200 24137728
131072 15 0 0 0 0 326 46899200 326 46899200
163840 16 1 163840 0 0 206 36245504 207 36409344
196608 17 1 196608 0 0 126 26578944 127 26775552
229376 18 1 229376 0 0 71 17244160 72 17473536
262144 19 0 0 0 0 137 39718912 137 39718912
327680 20 0 0 0 0 58 20738048 58 20738048
393216 21 0 0 0 0 32 13492224 32 13492224
458752 22 0 0 0 0 29 14155776 29 14155776
524288 23 0 0 0 0 48 27873280 48 27873280
655360 24 0 0 0 0 9 6406144 9 6406144
786432 25 0 0 0 0 7 5959680 7 5959680
---
1048576 27 0 0 0 0 3 3551232 3 3551232
1310720 28 0 0 0 0 2 2842624 2 2842624
1572864 29 0 0 0 0 1 1830912 1 1830912
---
2097152 31 0 0 0 0 2 4767744 2 4767744
2621440 32 0 0 0 0 3 8437760 3 8437760
3145728 33 0 0 0 0 1 3665920 1 3665920
---
4194304 35 0 0 0 0 3 13004800 3 13004800
---
6291456 37 0 0 0 0 1 7094272 1 7094272
7340032 38 0 0 0 0 1 7753728 1 7753728
---
8589934592 79 0 0 0 0 1 10014318592 1 10014318592
---
Bytes in small extent cache: 0
HPA shard stats:
Purge passes: 0 (0 / sec)
Purges: 0 (0 / sec)
Hugeifies: 0 (0 / sec)
Dehugifies: 0 (0 / sec)
In full slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
In empty slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
size ind npageslabs_huge nactive_huge ndirty_huge npageslabs_nonhuge nactive_nonhuge ndirty_nonhuge nretained_nonhuge
---
arenas[1]:
assigned threads: 0
uptime: 1945734009455967
dss allocation precedence: "secondary"
decaying: time npages sweeps madvises purged
dirty: 0 0 0 1 65536
muzzy: 0 0 0 0 0
allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nfill (#/sec) nflush (#/sec)
small: 0 0 0 0 0 0 0 0 0 0 0
large: 536870912 2 0 1 0 2 0 2 0 0 0
total: 536870912 2 0 1 0 2 0 2 0 0 0
active: 536870912
mapped: 538968064
retained: 268435456
base: 92112
internal: 0
metadata_thp: 0
tcache_bytes: 0
tcache_stashed_bytes: 0
resident: 536965120
abandoned_vm: 0
extent_avail: 0
n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
large 19363880 9 0 0 0 0 1 0 0 0 0 0
extent_avail 19363882 9 0 0 0 0 1 0 0 0 0 0
extents_dirty 19363883 9 0 0 0 0 1 0 0 0 0 0
extents_muzzy 19363880 9 0 0 0 0 1 0 0 0 0 0
extents_retained 19363883 9 0 0 0 0 1 0 0 0 0 0
decay_dirty 19363882 9 0 0 0 0 1 0 0 0 0 0
decay_muzzy 19363880 9 0 0 0 0 1 0 0 0 0 0
base 38727763 19 0 0 0 0 1 0 0 0 0 0
tcache_list 19363880 9 0 0 0 0 1 0 0 0 0 0
hpa_shard 0 0 0 0 0 0 0 0 0 0 0 0
hpa_shard_grow 0 0 0 0 0 0 0 0 0 0 0 0
hpa_sec 0 0 0 0 0 0 0 0 0 0 0 0
bins: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nshards curregs curslabs nonfull_slabs regs pgs util nfills (#/sec) nflushes (#/sec) nslabs nreslabs (#/sec) n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
---
large: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) curlextents
---
268435456 95 0 1 0 1 0 1 0 0
---
536870912 99 536870912 1 0 0 0 1 0 1
---
extents: size ind ndirty dirty nmuzzy muzzy nretained retained ntotal total
---
268435456 59 0 0 0 0 1 268435456 1 268435456
---
Bytes in small extent cache: 0
HPA shard stats:
Purge passes: 0 (0 / sec)
Purges: 0 (0 / sec)
Hugeifies: 0 (0 / sec)
Dehugifies: 0 (0 / sec)
In full slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
In empty slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
size ind npageslabs_huge nactive_huge ndirty_huge npageslabs_nonhuge nactive_nonhuge ndirty_nonhuge nretained_nonhuge
---
--- End jemalloc statistics ---
Comment From: oranagra
from a quick look i took, we can see that in both cases, only one arena is actually being used, and the difference between allocated and active and resident is similar.
is that because the older redis didn't run for long? i wonder what to focus on...
regarding retained memory (unlike "muzzy"), it actually doesn't affect your RSS, it's safe to ignore.
Comment From: lyveng
I just noticed that the resident is ~75G in both the cases. Till now I thought that I had saved the output of the MEMORY MALLOC-STATS from the container that had memory fragmentation ratio close to 1.28. But seems like that is not the case and I have not noted down the name of the pod for which I had taken the output from. I have prometheus data that shows that the redis containers had restarted after their memory slowly touched 90G. Is there a possibility that the resident memory increased, but doesn't show up in the MEMORY MALLOC-STATS? (I'm assuming the answer is no) Unfortunately, pods are not running the older redis version now. I'll see if I can run the older redis version in prod on just a couple of pods or replay the prod access logs on dev environment to reproduce the issue.
Comment From: oranagra
well, in theory it could be that something else was consuming RSS (not jemalloc), so it'll show in the process RSS and used_memory_rss, but not in allocator_resident and not in the malloc stats. this could be:
* memory mapped files (redis doesn't use them)
* calls to mmap (either directly by some code, or via a second allocator)
* executable code loaded by some shared library)
none of these make sense for redis unless you're using some odd module.
it could also be that the resident memory was released from the time you first saw it, till the time you took the stats.