Describe the bug
Hello! We have been observing a strange issue since upgrading our redis standalone servers from 6.2.5 to 7.2.5. Occasionally, one of our redis servers will start growing in RSS memory until the system is running out of memory. We have activedefrag enabled and it starts running during this time but it doesn't seem to help. Can someone help take a look and see what is going on? I captured some data when this occurred last. If we need more data I will have to wait until it happens again to get back with it.
MEMORY STATS
1) "peak.allocated"
2) (integer) 9155310640
3) "total.allocated"
4) (integer) 9088238672
5) "startup.allocated"
6) (integer) 859496
7) "replication.backlog"
8) (integer) 0
9) "clients.slaves"
10) (integer) 0
11) "clients.normal"
12) (integer) 18185184
13) "cluster.links"
14) (integer) 0
15) "aof.buffer"
16) (integer) 0
17) "lua.caches"
18) (integer) 1416
19) "functions.caches"
20) (integer) 184
21) "db.0"
22) 1) "overhead.hashtable.main"
2) (integer) 547207768
3) "overhead.hashtable.expires"
4) (integer) 382011752
5) "overhead.hashtable.slot-to-keys"
6) (integer) 0
23) "overhead.total"
24) (integer) 948265800
25) "keys.count"
26) (integer) 10324751
27) "keys.bytes-per-key"
28) (integer) 880
29) "dataset.bytes"
30) (integer) 8139972872
31) "dataset.percentage"
32) "89.57447814941406"
33) "peak.percentage"
34) "99.26740264892578"
35) "allocator.allocated"
36) (integer) 9088373168
37) "allocator.active"
38) (integer) 12528521216
39) "allocator.resident"
40) (integer) 12667953152
41) "allocator-fragmentation.ratio"
42) "1.3785219192504883"
43) "allocator-fragmentation.bytes"
44) (integer) 3440148048
45) "allocator-rss.ratio"
46) "1.0111291408538818"
47) "allocator-rss.bytes"
48) (integer) 139431936
49) "rss-overhead.ratio"
50) "0.9998315572738647"
51) "rss-overhead.bytes"
52) (integer) -2134016
53) "fragmentation"
54) "1.393649697303772"
55) "fragmentation.bytes"
56) (integer) 3577581536
MALLOC-STATS
___ Begin jemalloc statistics ___
Version: "5.3.0-0-g0"
Build-time option settings
config.cache_oblivious: false
config.debug: false
config.fill: true
config.lazy_lock: false
config.malloc_conf: ""
config.opt_safety_checks: false
config.prof: false
config.prof_libgcc: false
config.prof_libunwind: false
config.stats: true
config.utrace: false
config.xmalloc: false
Run-time option settings
opt.abort: false
opt.abort_conf: false
opt.cache_oblivious: false
opt.confirm_conf: false
opt.retain: true
opt.dss: "secondary"
opt.narenas: 8
opt.percpu_arena: "disabled"
opt.oversize_threshold: 8388608
opt.hpa: false
opt.hpa_slab_max_alloc: 65536
opt.hpa_hugification_threshold: 1992294
opt.hpa_hugify_delay_ms: 10000
opt.hpa_min_purge_interval_ms: 5000
opt.hpa_dirty_mult: "0.25"
opt.hpa_sec_nshards: 4
opt.hpa_sec_max_alloc: 32768
opt.hpa_sec_max_bytes: 262144
opt.hpa_sec_bytes_after_flush: 131072
opt.hpa_sec_batch_fill_extra: 0
opt.metadata_thp: "disabled"
opt.mutex_max_spin: 600
opt.background_thread: false (background_thread: true)
opt.dirty_decay_ms: 10000 (arenas.dirty_decay_ms: 10000)
opt.muzzy_decay_ms: 0 (arenas.muzzy_decay_ms: 0)
opt.lg_extent_max_active_fit: 6
opt.junk: "false"
opt.zero: false
opt.tcache: true
opt.tcache_max: 32768
opt.tcache_nslots_small_min: 20
opt.tcache_nslots_small_max: 200
opt.tcache_nslots_large: 20
opt.lg_tcache_nslots_mul: 1
opt.tcache_gc_incr_bytes: 65536
opt.tcache_gc_delay_bytes: 0
opt.lg_tcache_flush_small_div: 1
opt.lg_tcache_flush_large_div: 1
opt.thp: "default"
opt.stats_print: false
opt.stats_print_opts: ""
opt.stats_print: false
opt.stats_print_opts: ""
opt.stats_interval: -1
opt.stats_interval_opts: ""
opt.zero_realloc: "free"
Arenas: 9
Quantum size: 8
Page size: 4096
Maximum thread-cached size class: 32768
Number of bin size classes: 39
Number of thread-cache bin size classes: 44
Number of large size classes: 196
Allocated: 9088438136, active: 12537638912, metadata: 139436448 (n_thp 0), resident: 12677103616, mapped: 12690853888, retained: 1994452992
Count of realloc(non-null-ptr, 0) calls: 0
Background threads: 1, num_runs: 74773, run_interval: 9706952082 ns
n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
background_thread 15920251 21 0 0 0 0 1 0 0 0 0 0
max_per_bg_thd 8447672 11 0 0 0 0 3 0 0 0 0 0
ctl 31840495 43 0 0 0 0 1 0 0 0 0 0
prof 0 0 0 0 0 0 0 0 0 0 0 0
prof_thds_data 0 0 0 0 0 0 0 0 0 0 0 0
prof_dump 0 0 0 0 0 0 0 0 0 0 0 0
prof_recent_alloc 0 0 0 0 0 0 0 0 0 0 0 0
prof_recent_dump 0 0 0 0 0 0 0 0 0 0 0 0
prof_stats 0 0 0 0 0 0 0 0 0 0 0 0
Merged arenas stats:
assigned threads: 1
uptime: 725830571650076
dss allocation precedence: "N/A"
decaying: time npages sweeps madvises purged
dirty: N/A 30 59808 593498 1893267
muzzy: N/A 0 0 0 0
allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nfill (#/sec) nflush (#/sec)
small: 8760041336 554242742 763 446073706 614 67182704643 92559 85734598 118 29525034 40
large: 328396800 1215150 1 1214086 1 11320091 15 1215150 1 859012 1
total: 9088438136 555457892 765 447287792 616 67194024734 92575 86949748 119 30384046 41
active: 12537638912
mapped: 12690853888
retained: 1994452992
base: 139338144
internal: 98304
metadata_thp: 0
tcache_bytes: 103760
tcache_stashed_bytes: 0
resident: 12677103616
abandoned_vm: 0
extent_avail: 1
n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
large 15829375 21 0 0 0 0 2 0 0 0 0 0
extent_avail 17998331 24 0 0 2 0 104942 0 0 0 0 0
extents_dirty 22838844 31 0 0 58 0 119674 0 0 0 0 0
extents_muzzy 15864762 21 0 0 0 0 2 0 0 0 0 0
extents_retained 17626774 24 0 0 6 0 119698 0 0 0 0 0
decay_dirty 17250502 23 0 0 1 0 297186 0 0 0 0 0
decay_muzzy 15978550 22 0 0 0 0 296934 0 0 0 0 0
base 32529371 44 0 0 0 0 2 0 0 0 0 0
tcache_list 15829376 21 0 0 0 0 2 0 0 0 0 0
hpa_shard 0 0 0 0 0 0 0 0 0 0 0 0
hpa_shard_grow 0 0 0 0 0 0 0 0 0 0 0 0
hpa_sec 0 0 0 0 0 0 0 0 0 0 0 0
bins: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nshards curregs curslabs nonfull_slabs regs pgs util nfills (#/sec) nflushes (#/sec) nslabs nreslabs (#/sec) n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
8 0 5992 1460121 2 1459372 2 165227552 227 1 749 2 0 512 1 0.731 908665 1 751437 1 19 15835 0 17489496 24 0 0 0 0 2 0 0 0 0 0
16 1 165063952 53338498 73 43022001 59 19912092842 27433 1 10316497 42768 21276 256 1 0.942 10855872 14 1551039 2 78512 23817166 32 73137144 100 0 0 0 0 6578224 9 0 0 0 0
24 2 865151040 164045682 226 127997722 176 8437774623 11625 1 36047960 73048 55411 512 3 0.963 14373712 19 2039254 2 95160 65526118 90 198481414 273 0 0 0 0 24699694 34 0 0 0 0
32 3 328121216 47508388 65 37254600 51 11295660858 15562 1 10253788 85030 40353 128 1 0.942 6842004 9 1592562 2 158625 18131374 24 69605730 95 0 0 0 0 7114564 9 0 0 0 0
40 4 5440 534324 0 534188 0 2931833591 4039 1 136 1 0 512 5 0.265 521512 0 251277 0 1 0 0 16602165 22 0 0 0 0 2 0 0 0 0 0
48 5 109056 1985674 2 1983402 2 354711849 488 1 2272 21 15 256 3 0.422 1258241 1 1048496 1 353 817728 1 18185250 25 0 0 0 0 57184 0 0 0 0 0
56 6 905802688 71069706 97 54894658 75 1922065392 2648 1 16175048 33453 17199 512 7 0.944 9502332 13 1586140 2 54004 24549924 33 94025222 129 0 0 0 0 10067686 13 0 0 0 0
64 7 220889984 20255554 27 16804148 23 9163599655 12624 1 3451406 57263 27030 64 1 0.941 3259472 4 1498164 2 134042 7486391 10 37275666 51 0 0 0 0 2912754 4 0 0 0 0
80 8 68633040 10648106 14 9790193 13 4291824716 5912 1 857913 3689 3351 256 5 0.908 2192822 3 1504358 2 13474 4109182 5 26367533 36 0 0 0 0 1574130 2 0 0 0 0
96 9 17339232 2499640 3 2319023 3 859528950 1184 1 180617 1532 682 128 3 0.921 1000405 1 1121574 1 4790 1638823 2 18579568 25 0 0 0 0 370174 0 0 0 0 0
112 10 62473936 5700197 7 5142394 7 574210007 791 1 557803 2320 1654 256 7 0.939 2136234 2 1322845 1 5747 3710376 5 20741666 28 0 0 0 0 476220 0 0 0 0 0
128 11 80135040 18561135 25 17935080 24 337172693 464 1 626055 22087 19550 32 1 0.885 6767882 9 1566604 2 107037 12767896 17 32374751 44 0 0 0 0 1755476 2 0 0 0 0
160 12 2426561600 56990302 78 41824292 57 2579470684 3553 1 15166010 119315 44864 128 5 0.993 7166304 9 1513051 2 186127 23672807 32 88689395 122 0 0 0 0 12148526 16 0 0 0 0
192 13 1408278144 31675569 43 24340787 33 2049712618 2823 1 7334782 119689 58758 64 3 0.957 5002488 6 1479207 2 200693 12090507 16 52419929 72 0 0 0 0 6362682 8 0 0 0 0
224 14 949042304 29418908 40 25182112 34 2048964999 2822 1 4236796 74111 69745 128 7 0.446 2849561 3 1491051 2 110749 8303720 11 60875629 83 0 0 0 0 1408028 1 0 0 0 0
256 15 20811776 1491703 2 1410407 1 44440621 61 1 81296 5296 1595 16 1 0.959 747211 1 776031 1 21314 1062821 1 17647301 24 0 0 0 0 228230 0 0 0 0 0
320 16 322758720 17105458 23 16096837 22 163920020 225 1 1008621 115337 111938 64 5 0.136 1210962 1 1267071 1 206011 1561018 2 48252296 66 0 0 0 0 1153456 1 0 0 0 0
384 17 492847104 4230320 5 2946864 4 22604066 31 1 1283456 40108 0 32 3 1 1338847 1 1067491 1 67691 1704949 2 20171049 27 0 0 0 0 333398 0 0 0 0 0
448 18 13349056 1277906 1 1248109 1 3152267 4 1 29797 475 213 64 7 0.980 1040379 1 630996 0 1311 1153062 1 17614587 24 0 0 0 0 63762 0 0 0 0 0
512 19 40167936 1413530 1 1335077 1 3221982 4 1 78453 9822 119 8 1 0.998 968131 1 722665 0 27368 1087814 1 17760997 24 0 0 0 0 98412 0 0 0 0 0
640 20 116151680 1510154 2 1328667 1 3868401 5 1 181487 5672 0 32 5 0.999 984093 1 630889 0 11067 1151388 1 17813467 24 0 0 0 0 51560 0 0 0 0 0
768 21 130656000 800815 1 630690 0 2335944 3 1 170125 10633 0 16 3 0.999 507694 0 285925 0 17375 566498 0 16834672 23 0 0 0 0 25354 0 0 0 0 0
896 22 84373632 633803 0 539636 0 1370478 1 1 94167 2943 0 32 7 0.999 443954 0 367760 0 4943 224736 0 16705338 23 0 0 0 0 14328 0 0 0 0 0
1024 23 28119040 4382005 6 4354545 5 6779850 9 1 27460 6869 15 4 1 0.999 1908121 2 1613523 2 32073 4316967 5 19406739 26 0 0 0 0 13934 0 0 0 0 0
1280 24 4357120 68112 0 64708 0 78116 0 1 3404 213 1 16 5 0.998 61798 0 39568 0 408 52685 0 15937093 21 0 0 0 0 7270 0 0 0 0 0
1536 25 1015296 50198 0 49537 0 43369 0 1 661 87 20 8 3 0.949 38045 0 36189 0 342 38659 0 15906588 21 0 0 0 0 3350 0 0 0 0 0
1792 26 684544 41289 0 40907 0 53208 0 1 382 26 12 16 7 0.918 38242 0 29987 0 107 23204 0 15899656 21 0 0 0 0 2710 0 0 0 0 0
2048 27 1288192 4957078 6 4956449 6 6486777 8 1 629 319 9 2 1 0.985 1431194 1 1285085 1 360643 4650579 6 18909840 26 0 0 0 0 6118 0 0 0 0 0
2560 28 768000 14111 0 13811 0 13860 0 1 300 43 24 8 5 0.872 13441 0 10157 0 86 8192 0 15854369 21 0 0 0 0 1646 0 0 0 0 0
3072 29 439296 34197 0 34054 0 32242 0 1 143 40 13 4 3 0.893 31317 0 30906 0 153 9327 0 15892405 21 0 0 0 0 850 0 0 0 0 0
3584 30 340480 17907 0 17812 0 17699 0 1 95 13 2 8 7 0.913 17454 0 16204 0 55 3026 0 15863489 21 0 0 0 0 532 0 0 0 0 0
4096 31 1421312 461127 0 460780 0 376549 0 1 347 347 0 1 1 1 259944 0 344160 0 461127 0 0 16896310 23 0 0 0 0 3288 0 0 0 0 0
5120 32 378880 15199 0 15125 0 15139 0 1 74 21 5 4 5 0.880 14958 0 13920 0 85 2254 0 15858668 21 0 0 0 0 400 0 0 0 0 0
6144 33 344064 4787 0 4731 0 4623 0 1 56 31 5 2 3 0.903 4363 0 2799 0 467 3298 0 15837168 21 0 0 0 0 238 0 0 0 0 0
7168 34 157696 2176 0 2154 0 2143 0 1 22 7 4 4 7 0.785 1950 0 1161 0 110 1194 0 15832719 21 0 0 0 0 110 0 0 0 0 0
8192 35 1540096 21667 0 21479 0 18668 0 1 188 188 0 1 2 1 17908 0 19952 0 21667 0 0 15889896 21 0 0 0 0 1888 0 0 0 0 0
10240 36 276480 2210 0 2183 0 2255 0 1 27 16 4 2 5 0.843 2092 0 1407 0 253 1760 0 15833207 21 0 0 0 0 96 0 0 0 0 0
12288 37 110592 13676 0 13667 0 13732 0 1 9 9 0 1 3 1 13509 0 13024 0 13676 0 0 15869608 21 0 0 0 0 48 0 0 0 0 0
14336 38 71680 1510 0 1505 0 1605 0 1 5 4 2 2 7 0.625 1485 0 1105 0 282 813 0 15832266 21 0 0 0 0 34 0 0 0 0 0
large: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) curlextents
16384 39 1769472 316178 0 316070 0 364172 0 108
20480 40 15667200 832600 1 831835 1 10874836 14 765
24576 41 737280 5624 0 5594 0 7639 0 30
28672 42 802816 17034 0 17006 0 29725 0 28
32768 43 1277952 1189 0 1150 0 1194 0 39
40960 44 40960 248 0 247 0 248 0 1
49152 45 0 2541 0 2541 0 2541 0 0
57344 46 0 35906 0 35906 0 35906 0 0
65536 47 1900544 957 0 928 0 957 0 29
81920 48 81920 1249 0 1248 0 1249 0 1
98304 49 0 113 0 113 0 113 0 0
114688 50 0 301 0 301 0 301 0 0
131072 51 1835008 266 0 252 0 266 0 14
163840 52 0 298 0 298 0 298 0 0
196608 53 0 90 0 90 0 90 0 0
229376 54 0 125 0 125 0 125 0 0
262144 55 6553600 133 0 108 0 133 0 25
327680 56 327680 112 0 111 0 112 0 1
393216 57 0 50 0 50 0 50 0 0
458752 58 0 8 0 8 0 8 0 0
524288 59 5242880 43 0 33 0 43 0 10
655360 60 655360 8 0 7 0 8 0 1
786432 61 0 8 0 8 0 8 0 0
917504 62 0 20 0 20 0 20 0 0
1048576 63 2097152 18 0 16 0 18 0 2
---
2097152 67 12582912 12 0 6 0 12 0 6
---
4194304 71 8388608 5 0 3 0 5 0 2
---
8388608 75 0 3 0 3 0 3 0 0
---
16777216 79 0 3 0 3 0 3 0 0
---
33554432 83 0 3 0 3 0 3 0 0
---
67108864 87 0 3 0 3 0 3 0 0
---
134217728 91 268435456 2 0 0 0 2 0 2
---
extents: size ind ndirty dirty nmuzzy muzzy nretained retained ntotal total
4096 0 10 40960 0 0 21144 86605824 21154 86646784
8192 1 10 81920 0 0 15494 126926848 15504 127008768
---
8388608 39 0 0 0 0 1 8388608 1 8388608
10485760 40 0 0 0 0 1 10485760 1 10485760
---
16777216 43 0 0 0 0 1 16777216 1 16777216
20971520 44 0 0 0 0 1 20971520 1 20971520
---
33554432 47 0 0 0 0 2 67108864 2 67108864
41943040 48 0 0 0 0 1 41943040 1 41943040
---
67108864 51 0 0 0 0 1 67108864 1 67108864
83886080 52 0 0 0 0 1 83886080 1 83886080
---
1342177280 68 0 0 0 0 1 1464250368 1 1464250368
---
Bytes in small extent cache: 0
HPA shard stats:
Purge passes: 0 (0 / sec)
Purges: 0 (0 / sec)
Hugeifies: 0 (0 / sec)
Dehugifies: 0 (0 / sec)
In full slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
In empty slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
size ind npageslabs_huge nactive_huge ndirty_huge npageslabs_nonhuge nactive_nonhuge ndirty_nonhuge nretained_nonhuge
---
arenas[0]:
assigned threads: 1
uptime: 725830571650076
dss allocation precedence: "secondary"
decaying: time npages sweeps madvises purged
dirty: 10000 30 59808 593486 1801107
muzzy: 0 0 0 0 0
allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nfill (#/sec) nflush (#/sec)
small: 8760041336 554242742 763 446073706 614 67182704643 92559 85734598 118 29525034 40
large: 59961344 1215136 1 1214074 1 11320077 15 1215136 1 859012 1
total: 8820002680 555457878 765 447287780 616 67194024720 92575 86949734 119 30384046 41
active: 12269203456
mapped: 12420321280
retained: 1677783040
base: 139244752
internal: 98304
metadata_thp: 0
tcache_bytes: 103760
tcache_stashed_bytes: 0
resident: 12408573952
abandoned_vm: 0
extent_avail: 0
n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
large 7960125 10 0 0 0 0 1 0 0 0 0 0
extent_avail 10129062 13 0 0 2 0 104941 0 0 0 0 0
extents_dirty 14969568 20 0 0 58 0 119673 0 0 0 0 0
extents_muzzy 7995512 11 0 0 0 0 1 0 0 0 0 0
extents_retained 9757493 13 0 0 6 0 119697 0 0 0 0 0
decay_dirty 9234703 12 0 0 1 0 148985 0 0 0 0 0
decay_muzzy 8034898 11 0 0 0 0 148831 0 0 0 0 0
base 16790858 23 0 0 0 0 1 0 0 0 0 0
tcache_list 7960126 10 0 0 0 0 1 0 0 0 0 0
hpa_shard 0 0 0 0 0 0 0 0 0 0 0 0
hpa_shard_grow 0 0 0 0 0 0 0 0 0 0 0 0
hpa_sec 0 0 0 0 0 0 0 0 0 0 0 0
bins: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nshards curregs curslabs nonfull_slabs regs pgs util nfills (#/sec) nflushes (#/sec) nslabs nreslabs (#/sec) n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
8 0 5992 1460121 2 1459372 2 165227552 227 1 749 2 0 512 1 0.731 908665 1 751437 1 19 15835 0 9620246 13 0 0 0 0 1 0 0 0 0 0
16 1 165063952 53338498 73 43022001 59 19912092842 27433 1 10316497 42768 21276 256 1 0.942 10855872 14 1551039 2 78512 23817166 32 65267894 89 0 0 0 0 6578223 9 0 0 0 0
24 2 865151040 164045682 226 127997722 176 8437774623 11625 1 36047960 73048 55411 512 3 0.963 14373712 19 2039254 2 95160 65526118 90 190612164 262 0 0 0 0 24699693 34 0 0 0 0
32 3 328121216 47508388 65 37254600 51 11295660858 15562 1 10253788 85030 40353 128 1 0.942 6842004 9 1592562 2 158625 18131374 24 61736480 85 0 0 0 0 7114563 9 0 0 0 0
40 4 5440 534324 0 534188 0 2931833591 4039 1 136 1 0 512 5 0.265 521512 0 251277 0 1 0 0 8732915 12 0 0 0 0 1 0 0 0 0 0
48 5 109056 1985674 2 1983402 2 354711849 488 1 2272 21 15 256 3 0.422 1258241 1 1048496 1 353 817728 1 10316000 14 0 0 0 0 57183 0 0 0 0 0
56 6 905802688 71069706 97 54894658 75 1922065392 2648 1 16175048 33453 17199 512 7 0.944 9502332 13 1586140 2 54004 24549924 33 86155972 118 0 0 0 0 10067685 13 0 0 0 0
64 7 220889984 20255554 27 16804148 23 9163599655 12624 1 3451406 57263 27030 64 1 0.941 3259472 4 1498164 2 134042 7486391 10 29406416 40 0 0 0 0 2912753 4 0 0 0 0
80 8 68633040 10648106 14 9790193 13 4291824716 5912 1 857913 3689 3351 256 5 0.908 2192822 3 1504358 2 13474 4109182 5 18498283 25 0 0 0 0 1574129 2 0 0 0 0
96 9 17339232 2499640 3 2319023 3 859528950 1184 1 180617 1532 682 128 3 0.921 1000405 1 1121574 1 4790 1638823 2 10710318 14 0 0 0 0 370173 0 0 0 0 0
112 10 62473936 5700197 7 5142394 7 574210007 791 1 557803 2320 1654 256 7 0.939 2136234 2 1322845 1 5747 3710376 5 12872416 17 0 0 0 0 476219 0 0 0 0 0
128 11 80135040 18561135 25 17935080 24 337172693 464 1 626055 22087 19550 32 1 0.885 6767882 9 1566604 2 107037 12767896 17 24505501 33 0 0 0 0 1755475 2 0 0 0 0
160 12 2426561600 56990302 78 41824292 57 2579470684 3553 1 15166010 119315 44864 128 5 0.993 7166304 9 1513051 2 186127 23672807 32 80820145 111 0 0 0 0 12148525 16 0 0 0 0
192 13 1408278144 31675569 43 24340787 33 2049712618 2823 1 7334782 119689 58758 64 3 0.957 5002488 6 1479207 2 200693 12090507 16 44550679 61 0 0 0 0 6362681 8 0 0 0 0
224 14 949042304 29418908 40 25182112 34 2048964999 2822 1 4236796 74111 69745 128 7 0.446 2849561 3 1491051 2 110749 8303720 11 53006379 73 0 0 0 0 1408027 1 0 0 0 0
256 15 20811776 1491703 2 1410407 1 44440621 61 1 81296 5296 1595 16 1 0.959 747211 1 776031 1 21314 1062821 1 9778051 13 0 0 0 0 228229 0 0 0 0 0
320 16 322758720 17105458 23 16096837 22 163920020 225 1 1008621 115337 111938 64 5 0.136 1210962 1 1267071 1 206011 1561018 2 40383046 55 0 0 0 0 1153455 1 0 0 0 0
384 17 492847104 4230320 5 2946864 4 22604066 31 1 1283456 40108 0 32 3 1 1338847 1 1067491 1 67691 1704949 2 12301799 16 0 0 0 0 333397 0 0 0 0 0
448 18 13349056 1277906 1 1248109 1 3152267 4 1 29797 475 213 64 7 0.980 1040379 1 630996 0 1311 1153062 1 9745337 13 0 0 0 0 63761 0 0 0 0 0
512 19 40167936 1413530 1 1335077 1 3221982 4 1 78453 9822 119 8 1 0.998 968131 1 722665 0 27368 1087814 1 9891747 13 0 0 0 0 98411 0 0 0 0 0
640 20 116151680 1510154 2 1328667 1 3868401 5 1 181487 5672 0 32 5 0.999 984093 1 630889 0 11067 1151388 1 9944217 13 0 0 0 0 51559 0 0 0 0 0
768 21 130656000 800815 1 630690 0 2335944 3 1 170125 10633 0 16 3 0.999 507694 0 285925 0 17375 566498 0 8965422 12 0 0 0 0 25353 0 0 0 0 0
896 22 84373632 633803 0 539636 0 1370478 1 1 94167 2943 0 32 7 0.999 443954 0 367760 0 4943 224736 0 8836088 12 0 0 0 0 14327 0 0 0 0 0
1024 23 28119040 4382005 6 4354545 5 6779850 9 1 27460 6869 15 4 1 0.999 1908121 2 1613523 2 32073 4316967 5 11537489 15 0 0 0 0 13933 0 0 0 0 0
1280 24 4357120 68112 0 64708 0 78116 0 1 3404 213 1 16 5 0.998 61798 0 39568 0 408 52685 0 8067843 11 0 0 0 0 7269 0 0 0 0 0
1536 25 1015296 50198 0 49537 0 43369 0 1 661 87 20 8 3 0.949 38045 0 36189 0 342 38659 0 8037338 11 0 0 0 0 3349 0 0 0 0 0
1792 26 684544 41289 0 40907 0 53208 0 1 382 26 12 16 7 0.918 38242 0 29987 0 107 23204 0 8030406 11 0 0 0 0 2709 0 0 0 0 0
2048 27 1288192 4957078 6 4956449 6 6486777 8 1 629 319 9 2 1 0.985 1431194 1 1285085 1 360643 4650579 6 11040590 15 0 0 0 0 6117 0 0 0 0 0
2560 28 768000 14111 0 13811 0 13860 0 1 300 43 24 8 5 0.872 13441 0 10157 0 86 8192 0 7985119 11 0 0 0 0 1645 0 0 0 0 0
3072 29 439296 34197 0 34054 0 32242 0 1 143 40 13 4 3 0.893 31317 0 30906 0 153 9327 0 8023155 11 0 0 0 0 849 0 0 0 0 0
3584 30 340480 17907 0 17812 0 17699 0 1 95 13 2 8 7 0.913 17454 0 16204 0 55 3026 0 7994239 11 0 0 0 0 531 0 0 0 0 0
4096 31 1421312 461127 0 460780 0 376549 0 1 347 347 0 1 1 1 259944 0 344160 0 461127 0 0 9027060 12 0 0 0 0 3287 0 0 0 0 0
5120 32 378880 15199 0 15125 0 15139 0 1 74 21 5 4 5 0.880 14958 0 13920 0 85 2254 0 7989418 11 0 0 0 0 399 0 0 0 0 0
6144 33 344064 4787 0 4731 0 4623 0 1 56 31 5 2 3 0.903 4363 0 2799 0 467 3298 0 7967918 10 0 0 0 0 237 0 0 0 0 0
7168 34 157696 2176 0 2154 0 2143 0 1 22 7 4 4 7 0.785 1950 0 1161 0 110 1194 0 7963469 10 0 0 0 0 109 0 0 0 0 0
8192 35 1540096 21667 0 21479 0 18668 0 1 188 188 0 1 2 1 17908 0 19952 0 21667 0 0 8020646 11 0 0 0 0 1887 0 0 0 0 0
10240 36 276480 2210 0 2183 0 2255 0 1 27 16 4 2 5 0.843 2092 0 1407 0 253 1760 0 7963957 10 0 0 0 0 95 0 0 0 0 0
12288 37 110592 13676 0 13667 0 13732 0 1 9 9 0 1 3 1 13509 0 13024 0 13676 0 0 8000358 11 0 0 0 0 47 0 0 0 0 0
14336 38 71680 1510 0 1505 0 1605 0 1 5 4 2 2 7 0.625 1485 0 1105 0 282 813 0 7963016 10 0 0 0 0 33 0 0 0 0 0
large: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) curlextents
16384 39 1769472 316178 0 316070 0 364172 0 108
20480 40 15667200 832600 1 831835 1 10874836 14 765
24576 41 737280 5624 0 5594 0 7639 0 30
28672 42 802816 17034 0 17006 0 29725 0 28
32768 43 1277952 1189 0 1150 0 1194 0 39
40960 44 40960 248 0 247 0 248 0 1
49152 45 0 2541 0 2541 0 2541 0 0
57344 46 0 35906 0 35906 0 35906 0 0
65536 47 1900544 957 0 928 0 957 0 29
81920 48 81920 1249 0 1248 0 1249 0 1
98304 49 0 113 0 113 0 113 0 0
114688 50 0 301 0 301 0 301 0 0
131072 51 1835008 266 0 252 0 266 0 14
163840 52 0 298 0 298 0 298 0 0
196608 53 0 90 0 90 0 90 0 0
229376 54 0 125 0 125 0 125 0 0
262144 55 6553600 133 0 108 0 133 0 25
327680 56 327680 112 0 111 0 112 0 1
393216 57 0 50 0 50 0 50 0 0
458752 58 0 8 0 8 0 8 0 0
524288 59 5242880 43 0 33 0 43 0 10
655360 60 655360 8 0 7 0 8 0 1
786432 61 0 8 0 8 0 8 0 0
917504 62 0 20 0 20 0 20 0 0
1048576 63 2097152 18 0 16 0 18 0 2
---
2097152 67 12582912 12 0 6 0 12 0 6
---
4194304 71 8388608 5 0 3 0 5 0 2
---
extents: size ind ndirty dirty nmuzzy muzzy nretained retained ntotal total
4096 0 10 40960 0 0 21144 86605824 21154 86646784
8192 1 10 81920 0 0 15494 126926848 15504 127008768
---
1342177280 68 0 0 0 0 1 1464250368 1 1464250368
---
Bytes in small extent cache: 0
HPA shard stats:
Purge passes: 0 (0 / sec)
Purges: 0 (0 / sec)
Hugeifies: 0 (0 / sec)
Dehugifies: 0 (0 / sec)
In full slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
In empty slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
size ind npageslabs_huge nactive_huge ndirty_huge npageslabs_nonhuge nactive_nonhuge ndirty_nonhuge nretained_nonhuge
---
arenas[8]:
assigned threads: 0
uptime: 717554192152229
dss allocation precedence: "secondary"
decaying: time npages sweeps madvises purged
dirty: 0 0 0 12 92160
muzzy: 0 0 0 0 0
allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nfill (#/sec) nflush (#/sec)
small: 0 0 0 0 0 0 0 0 0 0 0
large: 268435456 14 0 12 0 14 0 14 0 0 0
total: 268435456 14 0 12 0 14 0 14 0 0 0
active: 268435456
mapped: 270532608
retained: 316669952
base: 93392
internal: 0
metadata_thp: 0
tcache_bytes: 0
tcache_stashed_bytes: 0
resident: 268529664
abandoned_vm: 0
extent_avail: 1
n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
large 7869250 10 0 0 0 0 1 0 0 0 0 0
extent_avail 7869269 10 0 0 0 0 1 0 0 0 0 0
extents_dirty 7869276 10 0 0 0 0 1 0 0 0 0 0
extents_muzzy 7869250 10 0 0 0 0 1 0 0 0 0 0
extents_retained 7869281 10 0 0 0 0 1 0 0 0 0 0
decay_dirty 8015799 11 0 0 0 0 148201 0 0 0 0 0
decay_muzzy 7943652 11 0 0 0 0 148103 0 0 0 0 0
base 15738513 21 0 0 0 0 1 0 0 0 0 0
tcache_list 7869250 10 0 0 0 0 1 0 0 0 0 0
hpa_shard 0 0 0 0 0 0 0 0 0 0 0 0
hpa_shard_grow 0 0 0 0 0 0 0 0 0 0 0 0
hpa_sec 0 0 0 0 0 0 0 0 0 0 0 0
bins: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nshards curregs curslabs nonfull_slabs regs pgs util nfills (#/sec) nflushes (#/sec) nslabs nreslabs (#/sec) n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
---
large: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) curlextents
---
8388608 75 0 3 0 3 0 3 0 0
---
16777216 79 0 3 0 3 0 3 0 0
---
33554432 83 0 3 0 3 0 3 0 0
---
67108864 87 0 3 0 3 0 3 0 0
---
134217728 91 268435456 2 0 0 0 2 0 2
---
extents: size ind ndirty dirty nmuzzy muzzy nretained retained ntotal total
---
8388608 39 0 0 0 0 1 8388608 1 8388608
10485760 40 0 0 0 0 1 10485760 1 10485760
---
16777216 43 0 0 0 0 1 16777216 1 16777216
20971520 44 0 0 0 0 1 20971520 1 20971520
---
33554432 47 0 0 0 0 2 67108864 2 67108864
41943040 48 0 0 0 0 1 41943040 1 41943040
---
67108864 51 0 0 0 0 1 67108864 1 67108864
83886080 52 0 0 0 0 1 83886080 1 83886080
---
Bytes in small extent cache: 0
HPA shard stats:
Purge passes: 0 (0 / sec)
Purges: 0 (0 / sec)
Hugeifies: 0 (0 / sec)
Dehugifies: 0 (0 / sec)
In full slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
In empty slabs:
npageslabs: 0 huge, 0 nonhuge
nactive: 0 huge, 0 nonhuge
ndirty: 0 huge, 0 nonhuge
nretained: 0 huge, 0 nonhuge
size ind npageslabs_huge nactive_huge ndirty_huge npageslabs_nonhuge nactive_nonhuge ndirty_nonhuge nretained_nonhuge
---
--- End jemalloc statistics ---
To reproduce
Unsure what causes this issue to reproduce.
Expected behavior
I expect active defrag running to help with external memory fragmentation
Comment From: RohanNagar
Happened again today, here's some more data. Running memory purge does not help either.
INFO ALL
# Server
redis_version:7.2.5
redis_git_sha1:ba8435db
redis_git_dirty:0
redis_build_id:3074ab5ecf741c27
redis_mode:standalone
os:Linux 4.18.0-553.22.1.el8_10.x86_64 x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:8.5.0
process_id:4376
process_supervised:systemd
run_id:e811f080646ab2775a9a7f8cace050c05a7dfc92
tcp_port:6379
server_time_usec:1729555801281191
uptime_in_seconds:1133405
uptime_in_days:13
hz:10
configured_hz:10
lru_clock:1502553
executable:/usr/sbin/redis-server
config_file:/etc/redis.conf
io_threads_active:0
listener0:name=tcp,bind=0.0.0.0,port=6379
listener1:name=unix,bind=/var/run/redis_sock/redis-6379.sock
listener2:name=tls,bind=0.0.0.0,port=7379
# Clients
connected_clients:737
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:32768
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0
# Memory
used_memory:9088235600
used_memory_human:8.46G
used_memory_rss:12318998528
used_memory_rss_human:11.47G
used_memory_peak:10576946736
used_memory_peak_human:9.85G
used_memory_peak_perc:85.92%
used_memory_overhead:492113104
used_memory_startup:859624
used_memory_dataset:8596122496
used_memory_dataset_perc:94.59%
allocator_allocated:9088345344
allocator_active:12174733312
allocator_resident:12318003200
total_system_memory:16524062720
total_system_memory_human:15.39G
used_memory_lua:45056
used_memory_vm_eval:45056
used_memory_lua_human:44.00K
used_memory_scripts_eval:1416
number_of_cached_scripts:3
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:77824
used_memory_vm_total_human:76.00K
used_memory_functions:184
used_memory_scripts:1600
used_memory_scripts_human:1.56K
maxmemory:9088234496
maxmemory_human:8.46G
maxmemory_policy:allkeys-lru
allocator_frag_ratio:1.34
allocator_frag_bytes:3086387968
allocator_rss_ratio:1.01
allocator_rss_bytes:143269888
rss_overhead_ratio:1.00
rss_overhead_bytes:995328
mem_fragmentation_ratio:1.36
mem_fragmentation_bytes:3230781472
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:15918376
mem_cluster_links:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.3.0
active_defrag_running:3
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:3855724342
rdb_bgsave_in_progress:0
rdb_last_save_time:1728422396
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
# Stats
total_connections_received:543749
total_commands_processed:14896539480
instantaneous_ops_per_sec:23278
total_net_input_bytes:1879293071321
total_net_output_bytes:507751839476
total_net_repl_input_bytes:0
total_net_repl_output_bytes:0
instantaneous_input_kbps:2897.23
instantaneous_output_kbps:775.56
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:334
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:147716
evicted_keys:186471833
evicted_clients:0
total_eviction_exceeded_time:17441
current_eviction_exceeded_time:0
keyspace_hits:3265529322
keyspace_misses:786302739
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:47437824
active_defrag_misses:644438217
active_defrag_key_hits:24995110
active_defrag_key_misses:40306569
total_active_defrag_time:682394696
current_active_defrag_time:60226072
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:3
dump_payload_sanitizations:0
total_reads_processed:7080869174
total_writes_processed:7080287466
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:5614803
reply_buffer_expands:5846381
eventloop_cycles:6389187531
eventloop_duration_sum:134357470753
eventloop_duration_cmd_sum:44913538693
instantaneous_eventloop_cycles_per_sec:8754
instantaneous_eventloop_duration_usec:31
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:c0aae4db6311168f7d6c653c893f1d239d1b4e76
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:70291.693001
used_cpu_user:66864.358432
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:70286.924357
used_cpu_user_main_thread:66862.782976
# Modules
# Commandstats
cmdstat_hget:calls=4014097266,usec=7164352079,usec_per_call=1.78,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=72511,usec=99811,usec_per_call=1.38,rejected_calls=0,failed_calls=0
cmdstat_auth:calls=56674,usec=131705,usec_per_call=2.32,rejected_calls=0,failed_calls=0
cmdstat_del:calls=3002555646,usec=3529269530,usec_per_call=1.18,rejected_calls=0,failed_calls=0
cmdstat_acl|list:calls=1,usec=6,usec_per_call=6.00,rejected_calls=0,failed_calls=0
cmdstat_acl|save:calls=3,usec=5859,usec_per_call=1953.00,rejected_calls=0,failed_calls=0
cmdstat_acl|setuser:calls=3,usec=75,usec_per_call=25.00,rejected_calls=0,failed_calls=0
cmdstat_info:calls=18891,usec=4233845,usec_per_call=224.12,rejected_calls=0,failed_calls=0
cmdstat_set:calls=37734400,usec=50401772,usec_per_call=1.34,rejected_calls=0,failed_calls=0
cmdstat_latency|latest:calls=18891,usec=30860,usec_per_call=1.63,rejected_calls=0,failed_calls=0
cmdstat_latency|reset:calls=18891,usec=31469,usec_per_call=1.67,rejected_calls=0,failed_calls=0
cmdstat_hgetall:calls=395,usec=2737807,usec_per_call=6931.16,rejected_calls=0,failed_calls=0
cmdstat_hdel:calls=200,usec=397,usec_per_call=1.99,rejected_calls=0,failed_calls=0
cmdstat_expire:calls=3284830546,usec=2776575965,usec_per_call=0.85,rejected_calls=0,failed_calls=0
cmdstat_eval:calls=3,usec=139,usec_per_call=46.33,rejected_calls=0,failed_calls=0
cmdstat_hello:calls=414564,usec=3532850,usec_per_call=8.52,rejected_calls=0,failed_calls=0
cmdstat_hset:calls=504814497,usec=756635259,usec_per_call=1.50,rejected_calls=0,failed_calls=0
cmdstat_get:calls=37734400,usec=19816984,usec_per_call=0.53,rejected_calls=0,failed_calls=0
cmdstat_client|setinfo:calls=74432,usec=8017,usec_per_call=0.11,rejected_calls=0,failed_calls=0
cmdstat_evalsha:calls=4014097266,usec=41303237567,usec_per_call=10.29,rejected_calls=0,failed_calls=3
# Errorstats
errorstat_NOSCRIPT:count=3
# Latencystats
latency_percentiles_usec_hget:p50=2.007,p99=9.023,p99.9=17.023
latency_percentiles_usec_ping:p50=1.003,p99=6.015,p99.9=19.071
latency_percentiles_usec_auth:p50=2.007,p99=12.031,p99.9=28.031
latency_percentiles_usec_del:p50=1.003,p99=5.023,p99.9=16.063
latency_percentiles_usec_acl|list:p50=6.015,p99=6.015,p99.9=6.015
latency_percentiles_usec_acl|save:p50=1769.471,p99=2375.679,p99.9=2375.679
latency_percentiles_usec_acl|setuser:p50=9.023,p99=58.111,p99.9=58.111
latency_percentiles_usec_info:p50=208.895,p99=378.879,p99.9=679.935
latency_percentiles_usec_set:p50=1.003,p99=12.031,p99.9=24.063
latency_percentiles_usec_latency|latest:p50=1.003,p99=6.015,p99.9=17.023
latency_percentiles_usec_latency|reset:p50=1.003,p99=7.007,p99.9=18.047
latency_percentiles_usec_hgetall:p50=1.003,p99=43.007,p99.9=1002438.655
latency_percentiles_usec_hdel:p50=1.003,p99=9.023,p99.9=13.055
latency_percentiles_usec_expire:p50=1.003,p99=2.007,p99.9=14.015
latency_percentiles_usec_eval:p50=29.055,p99=82.431,p99.9=82.431
latency_percentiles_usec_hello:p50=7.007,p99=26.111,p99.9=40.191
latency_percentiles_usec_hset:p50=1.003,p99=10.047,p99.9=24.063
latency_percentiles_usec_get:p50=0.001,p99=2.007,p99.9=17.023
latency_percentiles_usec_client|setinfo:p50=0.001,p99=1.003,p99.9=1.003
latency_percentiles_usec_evalsha:p50=9.023,p99=35.071,p99.9=64.255
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=3232782,expires=3232782,avg_ttl=599116703
MEMORY STATS
1) "peak.allocated"
2) (integer) 10576946736
3) "total.allocated"
4) (integer) 9088233360
5) "startup.allocated"
6) (integer) 859624
7) "replication.backlog"
8) (integer) 0
9) "clients.slaves"
10) (integer) 0
11) "clients.normal"
12) (integer) 15919144
13) "cluster.links"
14) (integer) 0
15) "aof.buffer"
16) (integer) 0
17) "lua.caches"
18) (integer) 1416
19) "functions.caches"
20) (integer) 184
21) "db.0"
22) 1) "overhead.hashtable.main"
2) (integer) 263527688
3) "overhead.hashtable.expires"
4) (integer) 211803704
5) "overhead.hashtable.slot-to-keys"
6) (integer) 0
23) "overhead.total"
24) (integer) 492111760
25) "keys.count"
26) (integer) 3232749
27) "keys.bytes-per-key"
28) (integer) 2811
29) "dataset.bytes"
30) (integer) 8596121600
31) "dataset.percentage"
32) "94.59413146972656"
33) "peak.percentage"
34) "85.92491912841797"
35) "allocator.allocated"
36) (integer) 9088323640
37) "allocator.active"
38) (integer) 12174626816
39) "allocator.resident"
40) (integer) 12317818880
41) "allocator-fragmentation.ratio"
42) "1.3395899534225464"
43) "allocator-fragmentation.bytes"
44) (integer) 3086303176
45) "allocator-rss.ratio"
46) "1.0117615461349487"
47) "allocator-rss.bytes"
48) (integer) 143192064
49) "rss-overhead.ratio"
50) "1.0001137256622314"
51) "rss-overhead.bytes"
52) (integer) 1400832
53) "fragmentation"
54) "1.355513095855713"
55) "fragmentation.bytes"
56) (integer) 3230986296
Comment From: sundb
@RohanNagar what was the memory footprint before upgration?
can you give the report from redis-cli --bigkeys?
Comment From: RohanNagar
Hi @sundb before the upgrade, memory would reach maxmemory and stay there. RSS memory would never climb like this, definitely never climbed much higher than the dataset memory.
Our keyspace is pretty large so redis-cli --bigkeys takes a long time. I will try to run it tomorrow and give the result. Are you thinking some large keys could cause this issue?
Comment From: sundb
feel like it doesn't relate to big keys. It may be related to the change in the size (224bytes and 320 bytes) of some structures, resulting in a higher fragmentation rate of these two bin. please wait for me to take some time to verify these two versions.
bins: size ind allocated nmalloc (#/sec) ndalloc (#/sec) nrequests (#/sec) nshards curregs curslabs nonfull_slabs regs pgs util nfills (#/sec) nflushes (#/sec) nslabs nreslabs (#/sec) n_lock_ops (#/sec) n_waiting (#/sec) n_spin_acq (#/sec) n_owner_switch (#/sec) total_wait_ns (#/sec) max_wait_ns max_n_thds
224 14 949042304 29418908 40 25182112 34 2048964999 2822 1 4236796 74111 69745 128 7 0.446 2849561 3 1491051 2 110749 8303720 11 53006379 73 0 0 0 0 1408027 1 0 0 0 0
320 16 322758720 17105458 23 16096837 22 163920020 225 1 1008621 115337 111938 64 5 0.136 1210962 1 1267071 1 206011 1561018 2 40383046 55 0 0 0 0 1153455 1 0 0 0 0
Comment From: RohanNagar
Hi @sundb, have you had a chance to take a look at this?
Comment From: sundb
@RohanNagar sorry, i'm busy in other place, please give me more time.