my problem is, mem_fragmentation_ratio(2.26) is much larger than allocator_frag_ratio(1.06), I want to know where the memory not held by the allocator goes, how can I reclaim this part of memory.

redis_version:6.2.4

info memory show:

Memory

used_memory:5505324432 used_memory_human:5.13G used_memory_rss:12411400192 used_memory_rss_human:11.56G used_memory_peak:9650795216 used_memory_peak_human:8.99G used_memory_peak_perc:57.05% used_memory_overhead:367129880 used_memory_startup:1779576 used_memory_dataset:5138194552 used_memory_dataset_perc:93.36% allocator_allocated:5506484760 allocator_active:5833326592 allocator_resident:6006521856 total_system_memory:16577921024 total_system_memory_human:15.44G used_memory_lua:37888 used_memory_lua_human:37.00K used_memory_scripts:0 used_memory_scripts_human:0B number_of_cached_scripts:0 maxmemory:13261340672 maxmemory_human:12.35G maxmemory_policy:noeviction allocator_frag_ratio:1.06 allocator_frag_bytes:326841832 allocator_rss_ratio:1.03 allocator_rss_bytes:173195264 rss_overhead_ratio:2.07 rss_overhead_bytes:6404878336 mem_fragmentation_ratio:2.25 mem_fragmentation_bytes:6905584376 mem_not_counted_for_evict:0 mem_replication_backlog:67108864 mem_clients_slaves:61536 mem_clients_normal:31160912 mem_aof_buffer:0 mem_allocator:jemalloc-5.1.0 active_defrag_running:0 lazyfree_pending_objects:0 lazyfreed_objects:0

MEMORY STATS: 1) "peak.allocated" 2) (integer) 9650795216 3) "total.allocated" 4) (integer) 5513189576 5) "startup.allocated" 6) (integer) 1779576 7) "replication.backlog" 8) (integer) 67108864 9) "clients.slaves" 10) (integer) 61536 11) "clients.normal" 12) (integer) 33790920 13) "aof.buffer" 14) (integer) 0 15) "lua.caches" 16) (integer) 0 17) "db.0" 18) 1) "overhead.hashtable.main" 2) (integer) 266860664 3) "overhead.hashtable.expires" 4) (integer) 168088 19) "overhead.total" 20) (integer) 369769648 21) "keys.count" 22) (integer) 4993795 23) "keys.bytes-per-key" 24) (integer) 1103 25) "dataset.bytes" 26) (integer) 5143419928 27) "dataset.percentage" 28) "93.3231201171875" 29) "peak.percentage" 30) "57.126785278320312" 31) "allocator.allocated" 32) (integer) 5514926936 33) "allocator.active" 34) (integer) 5837991936 35) "allocator.resident" 36) (integer) 6025785344 37) "allocator-fragmentation.ratio" 38) "1.0585800409317017" 39) "allocator-fragmentation.bytes" 40) (integer) 323065000 41) "allocator-rss.ratio" 42) "1.0321674346923828" 43) "allocator-rss.bytes" 44) (integer) 187793408 45) "rss-overhead.ratio" 46) "2.0649123191833496" 47) "rss-overhead.bytes" 48) (integer) 6416932864 49) "fragmentation" 50) "2.2566990852355957" 51) "fragmentation.bytes" 52) (integer) 6929037248

I have turned on the activedefrag： 127.0.0.1:6380> config get frag 1) "activedefrag" 2) "yes" 3) "active-defrag-cycle-min" 4) "25" 5) "active-defrag-cycle-max" 6) "75" 7) "active-defrag-threshold-lower" 8) "10" 9) "active-defrag-threshold-upper" 10) "100" 11) "active-defrag-max-scan-fields" 12) "1000" 13) "active-defrag-ignore-bytes" 14) "104857600"

Comment From: oranagra

in your case, not only that it's not in the allocator active pages (defraggable), it's also not in the allocator rss (purgable), maybe something else (not the allocator at all) is holding this memory. maybe you can find something by looking at /proc/<pid>/smaps.

Comment From: oranagra

btw, maybe more info (redis version or full info) can help. are you using any modules?

Comment From: promisingMan

`# Server redis_version:6.2.4 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:5889ed3e00b5d908 redis_mode:cluster os:Linux 2.6.32-696.6.3.el6.x86_64 x86_64 arch_bits:64 multiplexing_api:epoll atomicvar_api:sync-builtin gcc_version:4.4.7 process_id:23336 process_supervised:no run_id:e395008ff55d88b2b9301e38acb34d2e8f8b8841 tcp_port:6380 server_time_usec:1681044287668955 uptime_in_seconds:9173061 uptime_in_days:106 hz:10 configured_hz:10 lru_clock:3322687 executable:/usr/local/bin/redis-server config_file:/etc/redis/redis_6380.conf io_threads_active:0

Clients

connected_clients:781 cluster_connections:46 maxclients:15000 client_recent_max_input_buffer:40984 client_recent_max_output_buffer:0 blocked_clients:0 tracking_clients:0 clients_in_timeout_table:0

Memory

used_memory:5482788960 used_memory_human:5.11G used_memory_rss:12196044800 used_memory_rss_human:11.36G used_memory_peak:9650795216 used_memory_peak_human:8.99G used_memory_peak_perc:56.81% used_memory_overhead:352023080 used_memory_startup:1779576 used_memory_dataset:5130765880 used_memory_dataset_perc:93.61% allocator_allocated:5483285016 allocator_active:5805314048 allocator_resident:5977997312 total_system_memory:16577921024 total_system_memory_human:15.44G used_memory_lua:37888 used_memory_lua_human:37.00K used_memory_scripts:0 used_memory_scripts_human:0B number_of_cached_scripts:0 maxmemory:13261340672 maxmemory_human:12.35G maxmemory_policy:noeviction allocator_frag_ratio:1.06 allocator_frag_bytes:322029032 allocator_rss_ratio:1.03 allocator_rss_bytes:172683264 rss_overhead_ratio:2.04 rss_overhead_bytes:6218047488 mem_fragmentation_ratio:2.22 mem_fragmentation_bytes:6713421464 mem_not_counted_for_evict:0 mem_replication_backlog:67108864 mem_clients_slaves:61536 mem_clients_normal:16010456 mem_aof_buffer:0 mem_allocator:jemalloc-5.1.0 active_defrag_running:0 lazyfree_pending_objects:0 lazyfreed_objects:0

Persistence

loading:0 current_cow_size:0 current_cow_size_age:0 current_fork_perc:0.00 current_save_keys_processed:0 current_save_keys_total:0 rdb_changes_since_last_save:13764943819 rdb_bgsave_in_progress:0 rdb_last_save_time:1671873769 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:28 rdb_current_bgsave_time_sec:-1 rdb_last_cow_size:241774592 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_last_cow_size:0 module_fork_in_progress:0 module_fork_last_cow_size:0

Stats

total_connections_received:151155929 total_commands_processed:34336740828 instantaneous_ops_per_sec:1059 total_net_input_bytes:7157253839215 total_net_output_bytes:49903654788940 instantaneous_input_kbps:45.54 instantaneous_output_kbps:495.77 rejected_connections:0 sync_full:29 sync_partial_ok:0 sync_partial_err:17 expired_keys:5247453 expired_stale_perc:0.00 expired_time_cap_reached_count:0 expire_cycle_cpu_milliseconds:2806824 evicted_keys:0 keyspace_hits:9693982230 keyspace_misses:16113386614 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:47603 total_forks:17 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:2036849 active_defrag_misses:127817333 active_defrag_key_hits:1344692 active_defrag_key_misses:7430146 tracking_total_keys:0 tracking_total_items:0 tracking_total_prefixes:0 unexpected_error_replies:0 total_error_replies:1425359 dump_payload_sanitizations:0 total_reads_processed:31636342699 total_writes_processed:33078747748 io_threaded_reads_processed:1707778189 io_threaded_writes_processed:4146751385

Replication

role:master connected_slaves:3 slave0:ip=10.21.0.48,port=6380,state=online,offset=5241164325895,lag=0 slave1:ip=10.21.0.44,port=6380,state=online,offset=5241164325895,lag=0 slave2:ip=10.21.0.51,port=6380,state=online,offset=5241164325895,lag=0 master_failover_state:no-failover master_replid:ca2c994298dba1d66e4f7c4d43b967804019b8b0 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:5241164325895 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:67108864 repl_backlog_first_byte_offset:5241097217032 repl_backlog_histlen:67108864

CPU

used_cpu_sys:180964.561205 used_cpu_user:2027231.768926 used_cpu_sys_children:35.553595 used_cpu_user_children:325.487518 used_cpu_sys_main_thread:180365.055343 used_cpu_user_main_thread:411460.736460

Modules

Errorstats

errorstat_ASK:count=133 errorstat_ERR:count=15 errorstat_MOVED:count=1425211

Cluster

cluster_enabled:1

Keyspace

db0:keys=4995624,expires=2637,avg_ttl=2721889`

Comment From: promisingMan

00400000-005b7000 r-xp 00000000 fc:01 672696 /usr/local/bin/redis-server Size: 1756 kB Rss: 868 kB Pss: 868 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 868 kB Private_Dirty: 0 kB Referenced: 868 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd ex mr mw me dw ?? ?? 007b7000-007c4000 rw-p 001b7000 fc:01 672696 /usr/local/bin/redis-server Size: 52 kB Rss: 48 kB Pss: 48 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 12 kB Private_Dirty: 36 kB Referenced: 48 kB Anonymous: 36 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me dw ?? ac ?? 007c4000-00831000 rw-p 00000000 00:00 0 Size: 436 kB Rss: 156 kB Pss: 156 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 156 kB Referenced: 80 kB Anonymous: 156 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me ac 0256c000-0258d000 rw-p 00000000 00:00 0 [heap] Size: 132 kB Rss: 56 kB Pss: 56 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 56 kB Referenced: 20 kB Anonymous: 56 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me ac 7f2cf0000000-7f2cf0021000 rw-p 00000000 00:00 0 Size: 132 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me nr 7f2cf0021000-7f2cf4000000 ---p 00000000 00:00 0 Size: 65404 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: mr mw me nr 7f2cf7e00000-7f2f00400000 rw-p 00000000 00:00 0 Size: 8525824 kB Rss: 6026452 kB Pss: 6026452 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 6026452 kB Referenced: 5812244 kB Anonymous: 6026452 kB AnonHugePages: 5470208 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me nr 7f2f005ff000-7f2f505ff000 rw-p 00000000 00:00 0 Size: 1310720 kB Rss: 916500 kB Pss: 916500 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 916500 kB Referenced: 835840 kB Anonymous: 916500 kB AnonHugePages: 718848 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me nr 7f2f505ff000-7f2f50600000 ---p 00000000 00:00 0 Size: 4 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: mr mw me ac 7f2f50600000-7f2f51000000 rw-p 00000000 00:00 0 Size: 10240 kB Rss: 2048 kB Pss: 2048 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 2048 kB Referenced: 2048 kB Anonymous: 2048 kB AnonHugePages: 2048 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me ac 7f2f51000000-7f3062800000 rw-p 00000000 00:00 0 Size: 4481024 kB Rss: 3383596 kB Pss: 3383596 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 3383596 kB Referenced: 3175056 kB Anonymous: 3383596 kB AnonHugePages: 1345536 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me nr 7f3062900000-7f30e4021000 rw-p 00000000 00:00 0 Size: 2120836 kB Rss: 1679240 kB Pss: 1679240 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 1679240 kB Referenced: 1632636 kB Anonymous: 1679240 kB AnonHugePages: 333824 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me nr 7f30e4021000-7f30e8000000 ---p 00000000 00:00 0 Size: 65404 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: mr mw me nr 7f30e8000000-7f30e8600000 rw-p 00000000 00:00 0 Size: 6144 kB Rss: 6144 kB Pss: 6144 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 6144 kB Referenced: 6144 kB Anonymous: 6144 kB AnonHugePages: 6144 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me nr 7f30e87ff000-7f30e97ff000 rw-p 00000000 00:00 0 Size: 16384 kB Rss: 14132 kB Pss: 14132 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 14132 kB Referenced: 14068 kB Anonymous: 14132 kB AnonHugePages: 2048 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me nr 7f30e97ff000-7f30e9800000 ---p 00000000 00:00 0 Size: 4 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: mr mw me ac 7f30e9800000-7f30ea200000 rw-p 00000000 00:00 0 Size: 10240 kB Rss: 2048 kB Pss: 2048 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 2048 kB Referenced: 2048 kB Anonymous: 2048 kB AnonHugePages: 2048 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me ac 7f30ea200000-7f30eb400000 rw-p 00000000 00:00 0 Size: 18432 kB Rss: 16792 kB Pss: 16792 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 16792 kB Referenced: 16720 kB Anonymous: 16792 kB AnonHugePages: 6144 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me nr 7f30eb466000-7f30edfe6000 rw-p 00000000 00:00 0 Size: 44544 kB Rss: 35944 kB Pss: 35944 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 35944 kB Referenced: 35808 kB Anonymous: 35944 kB AnonHugePages: 2048 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me nr 7f30edfe6000-7f30edfe7000 ---p 00000000 00:00 0 Size: 4 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: mr mw me ac 7f30edfe7000-7f30ee9e7000 rw-p 00000000 00:00 0 Size: 10240 kB Rss: 16 kB Pss: 16 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 16 kB Referenced: 16 kB Anonymous: 16 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me ac 7f30ee9e7000-7f30ee9e8000 ---p 00000000 00:00 0 Size: 4 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: mr mw me ac 7f30ee9e8000-7f30ef3e8000 rw-p 00000000 00:00 0 Size: 10240 kB Rss: 16 kB Pss: 16 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 16 kB Referenced: 16 kB Anonymous: 16 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me ac 7f30ef3e8000-7f30ef3e9000 ---p 00000000 00:00 0 Size: 4 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: mr mw me ac 7f30ef3e9000-7f30efde9000 rw-p 00000000 00:00 0 Size: 10240 kB Rss: 12 kB Pss: 12 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 12 kB Referenced: 0 kB Anonymous: 12 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me ac 7f30efde9000-7f30efdea000 ---p 00000000 00:00 0 Size: 4 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: mr mw me ac 7f30efdea000-7f30f07ea000 rw-p 00000000 00:00 0 Size: 10240 kB Rss: 12 kB Pss: 12 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 12 kB Referenced: 0 kB Anonymous: 12 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me ac 7f30f07ea000-7f30f07eb000 ---p 00000000 00:00 0 Size: 4 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: mr mw me ac 7f30f07eb000-7f30f11eb000 rw-p 00000000 00:00 0 Size: 10240 kB Rss: 16 kB Pss: 16 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 16 kB Referenced: 12 kB Anonymous: 16 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me ac 7f30f11eb000-7f30f176b000 rw-p 00000000 00:00 0 Size: 5632 kB Rss: 4860 kB Pss: 4860 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4860 kB Referenced: 4780 kB Anonymous: 4860 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me nr 7f30f176b000-7f30f7600000 r--p 00000000 fc:01 663334 /usr/lib/locale/locale-archive Size: 96852 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd mr mw me ?? 7f30f7600000-7f30f7a00000 rw-p 00000000 00:00 0 Size: 4096 kB Rss: 4092 kB Pss: 4092 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4092 kB Referenced: 3332 kB Anonymous: 4092 kB AnonHugePages: 2048 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me nr 7f30f7b77000-7f30f7d02000 r-xp 00000000 fc:01 131087 /lib64/libc-2.12.so Size: 1580 kB Rss: 552 kB Pss: 22 kB Shared_Clean: 552 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 552 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd ex mr mw me ?? 7f30f7d02000-7f30f7f01000 ---p 0018b000 fc:01 131087 /lib64/libc-2.12.so Size: 2044 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: mr mw me ?? 7f30f7f01000-7f30f7f05000 r--p 0018a000 fc:01 131087 /lib64/libc-2.12.so Size: 16 kB Rss: 16 kB Pss: 16 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 16 kB Referenced: 16 kB Anonymous: 16 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd mr mw me ac ?? 7f30f7f05000-7f30f7f07000 rw-p 0018e000 fc:01 131087 /lib64/libc-2.12.so Size: 8 kB Rss: 8 kB Pss: 8 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 8 kB Referenced: 8 kB Anonymous: 8 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me ac ?? 7f30f7f07000-7f30f7f0b000 rw-p 00000000 00:00 0 Size: 16 kB Rss: 12 kB Pss: 12 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 12 kB Referenced: 8 kB Anonymous: 12 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me ac 7f30f7f0b000-7f30f7f22000 r-xp 00000000 fc:01 131111 /lib64/libpthread-2.12.so Size: 92 kB Rss: 72 kB Pss: 3 kB Shared_Clean: 72 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 72 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd ex mr mw me ?? 7f30f7f22000-7f30f8122000 ---p 00017000 fc:01 131111 /lib64/libpthread-2.12.so Size: 2048 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: mr mw me ?? 7f30f8122000-7f30f8123000 r--p 00017000 fc:01 131111 /lib64/libpthread-2.12.so Size: 4 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd mr mw me ac ?? 7f30f8123000-7f30f8124000 rw-p 00018000 fc:01 131111 /lib64/libpthread-2.12.so Size: 4 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me ac ?? 7f30f8124000-7f30f8128000 rw-p 00000000 00:00 0 Size: 16 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me ac 7f30f8128000-7f30f812f000 r-xp 00000000 fc:01 131178 /lib64/librt-2.12.so Size: 28 kB Rss: 8 kB Pss: 0 kB Shared_Clean: 8 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 8 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd ex mr mw me ?? 7f30f812f000-7f30f832e000 ---p 00007000 fc:01 131178 /lib64/librt-2.12.so Size: 2044 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: mr mw me ?? 7f30f832e000-7f30f832f000 r--p 00006000 fc:01 131178 /lib64/librt-2.12.so Size: 4 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd mr mw me ac ?? 7f30f832f000-7f30f8330000 rw-p 00007000 fc:01 131178 /lib64/librt-2.12.so Size: 4 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced: 0 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me ac ?? 7f30f8330000-7f30f8332000 r-xp 00000000 fc:01 131103 /lib64/libdl-2.12.so Size: 8 kB Rss: 8 kB Pss: 0 kB Shared_Clean: 8 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 8 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd ex mr mw me ?? 7f30f8332000-7f30f8532000 ---p 00002000 fc:01 131103 /lib64/libdl-2.12.so Size: 2048 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: mr mw me ?? 7f30f8532000-7f30f8533000 r--p 00002000 fc:01 131103 /lib64/libdl-2.12.so Size: 4 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd mr mw me ac ?? 7f30f8533000-7f30f8534000 rw-p 00003000 fc:01 131103 /lib64/libdl-2.12.so Size: 4 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced: 0 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me ac ?? 7f30f8534000-7f30f85b7000 r-xp 00000000 fc:01 131113 /lib64/libm-2.12.so Size: 524 kB Rss: 12 kB Pss: 2 kB Shared_Clean: 12 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 12 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd ex mr mw me ?? 7f30f85b7000-7f30f87b6000 ---p 00083000 fc:01 131113 /lib64/libm-2.12.so Size: 2044 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: mr mw me ?? 7f30f87b6000-7f30f87b7000 r--p 00082000 fc:01 131113 /lib64/libm-2.12.so Size: 4 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd mr mw me ac ?? 7f30f87b7000-7f30f87b8000 rw-p 00083000 fc:01 131113 /lib64/libm-2.12.so Size: 4 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced: 0 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me ac ?? 7f30f87b8000-7f30f87d8000 r-xp 00000000 fc:01 131081 /lib64/ld-2.12.so Size: 128 kB Rss: 40 kB Pss: 0 kB Shared_Clean: 40 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 40 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd ex mr mw me dw ?? 7f30f89c9000-7f30f89ce000 rw-p 00000000 00:00 0 Size: 20 kB Rss: 20 kB Pss: 20 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 20 kB Referenced: 20 kB Anonymous: 20 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me ac 7f30f89d7000-7f30f89d8000 rw-p 00000000 00:00 0 Size: 4 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me ac 7f30f89d8000-7f30f89d9000 r--p 00020000 fc:01 131081 /lib64/ld-2.12.so Size: 4 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd mr mw me dw ac ?? 7f30f89d9000-7f30f89da000 rw-p 00021000 fc:01 131081 /lib64/ld-2.12.so Size: 4 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me dw ac ?? 7f30f89da000-7f30f89db000 rw-p 00000000 00:00 0 Size: 4 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced: 4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me ac 7ffcc3455000-7ffcc356a000 rw-p 00000000 00:00 0 [stack] Size: 2132 kB Rss: 272 kB Pss: 272 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 272 kB Referenced: 260 kB Anonymous: 272 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd wr mr mw me gd ac 7ffcc35e8000-7ffcc35e9000 r-xp 00000000 00:00 0 [vdso] Size: 4 kB Rss: 4 kB Pss: 0 kB Shared_Clean: 4 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 4 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd ex mr mw me de ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Size: 4 kB Rss: 0 kB Pss: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 0 kB Referenced: 0 kB Anonymous: 0 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB VmFlags: rd ex

Comment From: oranagra

so the big ones appear to be:

7f3062900000-7f30e4021000 rw-p 00000000 00:00 0
Size: 2120836 kB
Rss: 1679240 kB
7f2f51000000-7f3062800000 rw-p 00000000 00:00 0
Size: 4481024 kB
Rss: 3383596 kB
7f2f005ff000-7f2f505ff000 rw-p 00000000 00:00 0
Size: 1310720 kB
Rss: 916500 kB
7f2cf7e00000-7f2f00400000 rw-p 00000000 00:00 0
Size: 8525824 kB
Rss: 6026452 kB

they seem to be summed to roughly 12gb which matches the used_memory_rss, but sadly, it doesn't help us know who allocated them.

maybe MEMORY MALLOC-STATS can show something useful.

Comment From: promisingMan

`___ Begin jemalloc statistics ___ Version: "5.1.0-0-g0" Build-time option settings config.cache_oblivious: true config.debug: false config.fill: true config.lazy_lock: false config.malloc_conf: "" config.prof: false config.prof_libgcc: false config.prof_libunwind: false config.stats: true config.utrace: false config.xmalloc: false Run-time option settings opt.abort: false opt.abort_conf: false opt.retain: true opt.dss: "secondary" opt.narenas: 16 opt.percpu_arena: "disabled" opt.metadata_thp: "disabled" opt.background_thread: false (background_thread: true) opt.dirty_decay_ms: 10000 (arenas.dirty_decay_ms: 10000) opt.muzzy_decay_ms: 10000 (arenas.muzzy_decay_ms: 10000) opt.junk: "false" opt.zero: false opt.tcache: true opt.lg_tcache_max: 15 opt.thp: "not supported" opt.stats_print: false opt.stats_print_opts: "" Arenas: 16 Quantum size: 8 Page size: 4096 Maximum thread-cached size class: 32768 Number of bin size classes: 39 Number of thread-cache bin size classes: 44 Number of large size classes: 196 Allocated: 5503413744, active: 5827477504, metadata: 172479648 (n_thp 0), resident: 6011027456, mapped: 6031572992, retained: 10898735104 Background threads: 3, num_runs: 3918076, run_interval: 4688354407 ns n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds background_thread 200170641 0 0 5 0 0 0 ctl 400395445 0 0 1 0 0 0 prof 0 0 0 0 0 0 0 Merged arenas stats: assigned threads: 3 uptime: 9184927908159298 dss allocation precedence: "N/A" decaying: time npages sweeps madvises purged dirty: N/A 2724 3425730 24880816 1313583396 muzzy: N/A 0 0 0 0 allocated nmalloc ndalloc nrequests small: 4549443056265305928065265243768158439832731037 large: 953970688 3185136072 3185126773 3185136072 total: 5503413744268491064137268428894931443017867109

active: 5827477504 mapped: 6031572992 retained: 10898735104 base: 172380960 internal: 98688 metadata_thp: 0 tcache_bytes: 39296 resident: 6011027456 large extent_avail extents_dirty extents_muzzy extents_retained decay_dirty decay_muzzy base tcache_list bins: size ind 48 5 64 7 96 9 112 10 128 11 160 12 192 13 224 14 256 15 320 16 384 17 448 18 512 19 640 20 768 21 896 22 1024 23 1280 24 1536 25 1792 26 2048 27 2560 28 3072 29 3584 30 4096 31 5120 32 6144 33 7168 34 8192 35 10240 36 12288 37 14336 38 large: size ind 16384 39 20480 40 24576 41 28672 42 32768 43 40960 44 49152 45 57344 46 65536 47 81920 48 98304 49 114688 50 131072 51 163840 52 196608 53 229376 54 262144 55 327680 56 393216 57 458752 58 524288 59 655360 60 786432 61 917504 62 1048576 63 1310720 64 1572864 65 1835008 66 2097152 67 2621440 68 3145728 69 3670016 70 4194304 71 5242880 72 6291456 73 7340032 74 8388608 75 10485760 76 12582912 77 14680064 78 16777216 79 20971520 80 25165824 81 29360128 82 33554432 83 41943040 84 50331648 85 58720256 86 67108864 87 83886080 88 --- 134217728 91 --- 335544320 96 --- arenas[0]: assigned threads: 1 uptime: 9184927908159298 dss allocation precedence: "secondary" decaying: time dirty: 10000 muzzy: 10000 small: large: total: n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds 386031104 148 8062 34607097 41000789 5000098 1 1016627512 106 15073 35941757 19000364 8000154 1 7411337451 3424 1556581 94372548 513009812 17000313 2 316304154 0 0 3901576 0 0 0 374304438 21 1848 10396281 0 0 1 628960886 1 992 13075086 0 0 1 624215128 0 1027 12723494 0 0 0 601360615 0 0 3663 0 0 0 300253460 0 0 5 0 0 0 allocated nmalloc ndalloc nrequests curregs curslabs regs pgs util nfills nflushes nslabs nreslabs n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds 8 0 40560576 1651778337 1646708265 12344845014 5070072 13560 512 1 0.730 305023334 58712922 31546 347299191 667656379 21 304 4469923 0 0 1 16 1 135862960 11394679506 11386188071 54067200193 8491435 49672 256 1 0.667 2285190655 151462615 1293723 615396062 2836456034 343 300 7989186 41000786 7000135 1 24 2 521092128 17299167374 17277455202 91990322481 21712172 54347 512 3 0.780 1532695164 198909562 471857 927776615 2115381140 359 11106 84424947 36000690 6000114 1 32 3 119421984 207395083730 207391351793 216158733984 3731937 30115 128 1 0.968 5685128151 2084664747 3316399 15279668047 8133108603 2 33 7994910 0 0 1 40 4 401036160 11971713347 11961687443 15249620578 10025904 19690 512 5 0.994 2870194884 143336960 35824 3324507385 3337876314 1 34 3851700 0 0 1 83112384 7733586467 7731854959 12903814119 1731508 7036 256 3 0.961 1891971996 107060190 12047 3843346905 2311642582 2 13 3650574 0 0 1 56 6 169193360 5327933593 5324912283 15650538382 3021310 5946 512 7 0.992 1292741252 87915103 9630 1660336200 1719048322 42 231 3923382 0 0 1 77605120 1946621004 1945408424 19136029767 1212580 19215 64 1 0.986 645338069 73830818 35956 1290570595 1067193590 279 1308 8770756 9000175 2000040 1 80 8 213966480 175862264 173187683 1063228861 2674581 10467 256 5 0.998 103255277 33724558 18454 108981614 449515268 1 26 1913683 0 0 1 65535264 96895551 96212892 478660854 682659 5348 128 3 0.997 47699024 36180264 10543 70145893 386955579 0 2 1426314 0 0 0 41328000 22589917 22220917 50886218 369000 1443 256 7 0.998 15731387 16195784 4032 9797429 333372648 0 0 998115 0 0 0 38032000 6442953 6145828 8192591 297125 9287 32 1 0.999 4507472 4948965 20408 2779479 310366050 0 0 170187 0 0 0 32116320 8098648 7897921 19539320 200727 1573 128 5 0.996 5047016 5427124 4082 4872511 311241721 0 1 372177 0 0 0 10213056 4033508 3980315 5121533 53193 833 64 3 0.997 3021914 3407139 2458 1220741 306770268 0 0 101597 0 0 0 9673440 3219509 3176324 2951848 43185 339 128 7 0.995 1937049 2331523 833 562538 304548546 0 0 43699 0 0 0 91963904 3803149 3443915 4244300 359234 22454 16 1 0.999 2474753 2847304 51244 1936416 306280418 0 0 40137 0 0 0 256018560 4054620 3254562 6000717 800058 12502 64 5 0.999 1973989 2289842 26158 790815 306121865 0 0 106497 0 0 0 97936128 151346671 151091629 349390770 255042 8862 32 3 0.899 144080635 25416445 20320 148808365 471130522 0 18 2672173 0 0 0 41198528 3401695 3309734 3967545 91961 1457 64 7 0.986 2240764 2651139 5701 1904681 305504394 0 0 135379 0 0 0 25893376 2841021 2790448 3223907 50573 7217 8 1 0.875 2109796 2485726 18104 1857164 305005249 0 0 68579 0 0 0 112568320 4514075 4338187 20531197 175888 5530 32 5 0.993 2593139 3077953 10807 2870343 306218380 0 1 202866 0 0 0 627857664 3639280 2821757 6023203 817523 51096 16 3 0.999 2162671 2413508 74461 1458217 305792838 0 0 94327 0 0 0 21657216 3482611 3458440 6725663 24171 760 32 7 0.993 2680116 3038939 2491 2135190 306037946 0 0 95663 0 0 0 8706048 3879767 3871265 48599838 8502 2173 4 1 0.978 3169736 3483813 10647 2472518 306996268 0 0 140483 0 0 0 12692480 4756363 4746447 20996696 9916 651 16 5 0.951 3615267 4077969 2713 2996302 308120899 0 0 237847 0 0 0 8722944 4142886 4137207 13855935 5679 741 8 3 0.957 3288522 3659336 4371 2959487 307309227 0 1 181953 0 0 0 7865088 5161089 5156700 29007690 4389 337 16 7 0.813 3659343 4345457 3261 3583283 308422829 0 0 311045 0 0 0 19179520 4722517 4713152 16265628 9365 4820 2 1 0.971 3585784 4121743 68917 3573734 308194573 0 1 243524 0 0 0 7690240 4089999 4086995 11334800 3004 445 8 5 0.843 3059566 3381692 3807 3186687 306776339 0 0 205049 0 0 0 6829056 5853172 5850949 10017593 2223 585 4 3 0.950 5399490 5485066 14603 1500867 311240848 0 0 221441 0 0 0 246181376 3046725 2978036 9462449 68689 8822 8 7 0.973 2413110 2573943 18958 1727112 305493811 0 1 264659 0 0 0 44744704 3307546 3296622 5970296 10924 10924 1 1 1 2940991 3028498 3307546 0 312913071 0 0 219726 0 0 0 94044160 15365443 15347075 50973390 18368 4663 4 5 0.984 14700843 14869481 19938 2250494 330065069 0 2 546740 0 0 0 587857920 14649948 14554268 52009200 95680 48310 2 3 0.990 13825187 14001854 108883 2417299 328609448 0 1 517029 0 0 0 96803840 3833592 3820087 9410561 13505 3547 4 7 0.951 3400082 3522812 13293 1480867 307341236 0 1 373480 0 0 0 83402752 4017477 4007296 5495482 10181 10181 1 2 1 3463025 3671574 4017477 0 315508743 0 4 277891 0 0 0 36843520 4180333 4176735 5440725 3598 1820 2 5 0.988 3803786 3923741 33814 1333573 308159504 0 3 361908 0 0 0 23801856 4760892 4758955 6604137 1937 1937 1 3 1 4404031 4542452 4760892 0 318862382 0 3 462760 0 0 0 30234624 5371486 5369377 7493572 2109 1168 2 7 0.902 4994521 5151875 34400 1383584 310642677 0 3 563247 0 0 0 allocated nmalloc ndalloc nrequests curlextents 38879232 5135298 5132925 7955896 2373 74362880 396410716 396407085 1489685698 3631 8134656 10348758 10348427 24084824 331 13705216 8971761 8971283 18164319 478 7733248 6418028 6417792 11576846 236 18227200 2710015978 2710015533 2710015978 445 32342016 7277302 7276644 7277302 658 7970816 4118581 4118442 4118581 139 6225920 2632863 2632768 2632863 95 8192000 3010524 3010424 3010524 100 8355840 2422980 2422895 2422980 85 4358144 1738588 1738550 1738588 38 6946816 2445039 2444986 2445039 53 11796480 5007743 5007671 5007743 72 11599872 2910950 2910891 2910950 59 11927552 1039871 1039819 1039871 52 13631488 1040208 1040156 1040208 52 20971520 1452854 1452790 1452854 64 22020096 1069406 1069350 1069406 56 21561344 869296 869249 869296 47 22020096 777861 777819 777861 42 41287680 1213006 1212943 1213006 63 31457280 1977083 1977043 1977083 40 24772608 677005 676978 677005 27 15728640 465047 465032 465047 15 7864320 880664 880658 880664 6 9437184 833073 833067 833073 6 7340032 516775 516771 516775 4 12582912 405607 405601 405607 6 13107200 845515 845510 845515 5 12582912 720333 720329 720333 4 0 219514 219514 219514 0 16777216 322425 322421 322425 4 26214400 458671 458666 458671 5 12582912 196410 196408 196410 2 7340032 194820 194819 194820 1 8388608 83313 83312 83313 1 0 12164 12164 12164 0 0 4 4 4 0 0 1 1 1 0 0 13 13 13 0 0 2 2 2 0 0 1 1 1 0 0 1 1 1 0 0 3 3 3 0 0 1 1 1 0 0 1 1 1 0 0 1 1 1 0 201326592 4 1 4 3 0 5 5 5 0 134217728 1 0 1 1 0 4 4 4 0 npages sweeps madvises purged 2724 1681547 21353346 1241965037 0 0 0 0 allocated nmalloc ndalloc nrequests 4434971344260439421202260377928112434078780389 953282560 3118734938 3118725677 3118734938 5388253904263558156140263496653789437197515327

active: 5693775872 mapped: 5889482752 retained: 10674872320 base: 169617648 internal: 98688 metadata_thp: 0 tcache_bytes: 21528 resident: 5874556928 n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds large 185657196 148 8062 34579663 41000789 5000098 1 extent_avail 786548999 105 14632 25093520 19000364 8000154 1 extents_dirty 7051037649 3389 1538440 47469232 511009774 17000313 2 extents_muzzy 109819755 0 0 5417 0 0 0 extents_retained 157016972 18 455 3377673 0 0 1 decay_dirty 414328600 0 159 8979556 0 0 0 decay_muzzy 412631034 0 51 8857681 0 0 0 base 201003237 0 0 9 0 0 0 tcache_list 100085325 0 0 1 0 0 0 bins: size ind allocated nmalloc ndalloc nrequests curregs curslabs regs pgs util nfills nflushes nslabs nreslabs n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds 8 0 40560568 1651778182 1646708111 12224152157 5070071 13559 512 1 0.730 305023282 57627299 31511 347299191 466402453 21 304 2362799 0 0 1 16 1 129885600 11114881191 11106763341 53465659680 8117850 46664 256 1 0.679 2282331849 151409640 1288611 509191693 2566940303 0 0 2050583 0 0 0 24 2 521091864 17298912994 17277200833 91526715640 21712161 54346 512 3 0.780 1532446828 175858214 471856 927776615 1891865396 359 11105 48954255 36000690 6000114 1 32 3 118519584 203708967078 203705263341 212471389856 3703737 29801 128 1 0.970 5647955122 2084636159 3115928 15073187553 7844172531 0 0 343357 0 0 0 40 4 400351960 11753444503 11743435704 15034464598 10008799 19620 512 5 0.996 2865311680 143079962 35028 3294125381 3126592724 1 16 648965 0 0 1 48 5 82536336 7554036913 7552317406 12729107391 1719507 6978 256 3 0.962 1889537210 106793438 11752 3818982310 2099169018 0 0 433239 0 0 0 56 6 163423400 5132310662 5129392387 15456952491 2918275 5725 512 7 0.995 1290643732 87814651 9268 1590699987 1483574788 0 0 139373 0 0 0 64 7 72632896 1666175844 1665040955 18855992259 1134889 17779 64 1 0.997 640914247 73801548 33999 1089724785 816819148 0 0 123155 0 0 0 80 8 213128240 156757114 154093011 1050300605 2664103 10408 256 5 0.999 102756867 33280580 17594 103253511 240755283 0 0 100650 0 0 0 96 9 65436096 93608061 92926435 476790368 681626 5336 128 3 0.997 47353533 35841824 10408 69129018 184467823 0 0 103888 0 0 0 112 10 41280064 21585293 21216721 50365404 368572 1440 256 7 0.999 15494680 15961953 3509 9635789 132249352 0 0 77481 0 0 0 128 11 38030080 6370504 6073394 8158162 297110 9286 32 1 0.999 4477504 4917535 20400 2779475 110104432 0 0 47669 0 0 0 160 12 31993760 7833529 7633568 19408625 199961 1566 128 5 0.997 4962289 5346993 3874 4761597 110783899 0 0 43675 0 0 0 192 13 10145472 3996942 3944101 5101425 52841 826 64 3 0.999 3004601 3388247 2205 1219524 106547333 0 0 28969 0 0 0 224 14 9640736 3206541 3163502 2945289 43039 337 128 7 0.997 1931324 2324434 786 562537 104361862 0 0 18151 0 0 0 256 15 91843072 3794997 3436235 4239952 358762 22424 16 1 0.999 2470822 2842719 50921 1935287 106099108 0 0 22455 0 0 0 320 16 255616960 4042740 3243937 5994564 798803 12482 64 5 0.999 1969014 2284114 26071 790280 105937095 0 0 84101 0 0 0 384 17 97827840 151342458 151087698 349379715 254760 8851 32 3 0.899 144077306 24778948 20279 148807520 270317744 0 18 1406630 0 0 0 448 18 41084288 3380370 3288664 3957512 91706 1452 64 7 0.986 2232769 2642686 5658 1898415 105310058 0 0 101945 0 0 0 512 19 25820672 2822734 2772303 3213118 50431 7189 8 1 0.876 2100397 2475391 16943 1849994 104805008 0 0 28809 0 0 0 640 20 109866240 4345648 4173982 20450374 171666 5387 32 5 0.995 2550869 3035924 10162 2767232 105896469 0 0 33779 0 0 0 768 21 612661248 3578430 2780694 5988656 797736 49859 16 3 0.999 2145751 2397274 71889 1446633 105556072 0 0 27031 0 0 0 896 22 21392896 3409904 3386028 6694269 23876 749 32 7 0.996 2659690 3018337 2347 2096282 105797600 0 0 13337 0 0 0 1024 23 8512512 3779757 3771444 48545814 8313 2116 4 1 0.982 3138872 3453275 8951 2381249 106710990 0 0 18177 0 0 0 1280 24 12012800 4520662 4511277 20850908 9385 592 16 5 0.990 3564586 4028335 2311 2792641 107712536 0 0 37933 0 0 0 1536 25 8243712 3996500 3991133 13770115 5367 686 8 3 0.977 3246738 3620184 3606 2828199 106974757 0 0 20767 0 0 0 1792 26 7214592 4842162 4838136 28861230 4026 312 16 7 0.806 3585069 4272282 2325 3333641 107961490 0 0 17067 0 0 0 2048 27 18796544 4562986 4553808 16186433 9178 4722 2 1 0.971 3530253 4066857 51601 3433281 107804196 0 0 23265 0 0 0 2560 28 7495680 3977973 3975045 11273466 2928 434 8 5 0.843 3015479 3336502 2821 3107742 106460194 0 0 27157 0 0 0 3072 29 6807552 5777303 5775087 9951770 2216 582 4 3 0.951 5362728 5438779 11688 1456271 110940502 0 0 57109 0 0 0 3584 30 232064000 2920965 2856215 9367626 64750 8270 8 7 0.978 2367930 2524410 16929 1620498 105149044 0 1 72291 0 0 0 4096 31 43868160 3225378 3214668 5914812 10710 10710 1 1 1 2903092 2983438 3225378 0 112451901 0 0 54620 0 0 0 5120 32 85647360 15201380 15184652 50821969 16728 4229 4 5 0.988 14617387 14757789 14212 2135187 129592019 0 2 159343 0 0 0 6144 33 545366016 14467128 14378364 51836306 88764 44568 2 3 0.995 13746559 13898695 97457 2268513 128122094 0 1 155211 0 0 0 7168 34 93234176 3722981 3709974 9317460 13007 3363 4 7 0.966 3343295 3446461 10429 1398199 106967450 0 1 108750 0 0 0 8192 35 80715776 3934110 3924257 5435677 9853 9853 1 2 1 3418038 3614236 3934110 0 115023396 0 4 74467 0 0 0 10240 36 36372480 4074010 4070458 5362662 3552 1794 2 5 0.989 3744459 3847378 23318 1262163 107772334 0 3 93398 0 0 0 12288 37 23715840 4625338 4623408 6499614 1930 1930 1 3 1 4329393 4444494 4625338 0 118170665 0 3 121470 0 0 0 14336 38 30134272 5209937 5207835 7362417 2102 1164 2 7 0.902 4905856 5032610 12569 1284515 110127391 0 3 153823 0 0 0 large: size ind allocated nmalloc ndalloc nrequests curlextents 16384 39 38289408 5050603 5048266 7840184 2337 20480 40 74362880 396410716 396407085 1489629248 3631 24576 41 8134656 10293749 10293418 23984368 331 28672 42 13705216 8929514 8929036 18092919 478 32768 43 7733248 6393068 6392832 11538640 236 40960 44 18227200 2644061696 2644061251 2644061696 445 49152 45 32243712 7262666 7262010 7262666 656 57344 46 7970816 4113947 4113808 4113947 139 65536 47 6225920 2627395 2627300 2627395 95 81920 48 8192000 3003767 3003667 3003767 100 98304 49 8355840 2415022 2414937 2415022 85 114688 50 4358144 1724184 1724146 1724184 38 131072 51 6946816 2416656 2416603 2416656 53 163840 52 11796480 4914514 4914442 4914514 72 196608 53 11599872 2862408 2862349 2862408 59 229376 54 11927552 1033122 1033070 1033122 52 262144 55 13631488 1039325 1039273 1039325 52 327680 56 20971520 1452002 1451938 1452002 64 393216 57 22020096 1068659 1068603 1068659 56 458752 58 21561344 868696 868649 868696 47 524288 59 22020096 777239 777197 777239 42 655360 60 41287680 1212037 1211974 1212037 63 786432 61 31457280 1976119 1976079 1976119 40 917504 62 24772608 676491 676464 676491 27 1048576 63 15728640 464632 464617 464632 15 1310720 64 7864320 880039 880033 880039 6 1572864 65 9437184 832437 832431 832437 6 1835008 66 7340032 516512 516508 516512 4 2097152 67 12582912 405474 405468 405474 6 2621440 68 13107200 845275 845270 845275 5 3145728 69 12582912 720132 720128 720132 4 3670016 70 0 219416 219416 219416 0 4194304 71 16777216 322304 322300 322304 4 5242880 72 26214400 458559 458554 458559 5 6291456 73 12582912 196341 196339 196341 2 7340032 74 7340032 194766 194765 194766 1 8388608 75 8388608 83265 83264 83265 1 10485760 76 0 12149 12149 12149 0 12582912 77 0 4 4 4 0 14680064 78 0 1 1 1 0 16777216 79 0 13 13 13 0 20971520 80 0 2 2 2 0 25165824 81 0 1 1 1 0 29360128 82 0 1 1 1 0 33554432 83 0 3 3 3 0 41943040 84 0 1 1 1 0 50331648 85 0 1 1 1 0 58720256 86 0 1 1 1 0 67108864 87 201326592 4 1 4 3 83886080 88 0 5 5 5 0 --- 134217728 91 134217728 1 0 1 1 --- 335544320 96 0 4 4 4 0 --- arenas[1]: assigned threads: 1 uptime: 9184896330812013 dss allocation precedence: "secondary" decaying: time npages sweeps madvises purged dirty: 10000 0 1744183 3527470 71618359 muzzy: 10000 0 0 0 0 allocated nmalloc ndalloc nrequests small: 114471712 4866506863 4865840046 5753950648 large: 688128 66401134 66401096 66401134 total: 115159840 4932907997 4932241142 5820351782

active: 133701632 mapped: 139993088 retained: 223862784 base: 2734368 internal: 0 metadata_thp: 0 tcache_bytes: 17480 resident: 136437760 n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds large 100290751 0 0 27433 0 0 0 extent_avail 129995356 1 441 10848236 0 0 1 extents_dirty 260216645 35 18141 46903315 2000038 1000019 1 extents_muzzy 106401242 0 0 3896158 0 0 0 extents_retained 117204309 3 1393 7018607 0 0 1 decay_dirty 114549125 1 833 4095528 0 0 1 decay_muzzy 111500933 0 976 3865811 0 0 0 base 200191072 0 0 3652 0 0 0 tcache_list 100084977 0 0 2 0 0 0 bins: size ind allocated nmalloc ndalloc nrequests curregs curslabs regs pgs util nfills nflushes nslabs nreslabs n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds 8 0 8 155 154 120692857 1 1 512 1 0.001 52 1085623 35 0 101170769 0 0 2107123 0 0 0 16 1 5977360 279798315 279424730 601540513 373585 3008 256 1 0.485 2858806 52975 5112 106204369 169432574 343 300 5938602 41000786 7000135 1 24 2 264 254380 254369 463606841 11 1 512 3 0.021 248336 23051348 1 0 123432587 0 1 35470691 0 0 0 32 3 902400 3686116652 3686088452 3687344128 28200 314 128 1 0.701 37173029 28588 200471 206480494 188852915 2 33 7651552 0 0 1 40 4 684200 218268844 218251739 215155980 17105 70 512 5 0.477 4883204 256998 796 30382004 111200433 0 18 3202734 0 0 0 48 5 576048 179549554 179537553 174706728 12001 58 256 3 0.808 2434786 266752 295 24364595 112390407 2 13 3217334 0 0 1 56 6 5769960 195622931 195519896 193585891 103035 221 512 7 0.910 2097520 100452 362 69636213 135390377 42 231 3784008 0 0 1 64 7 4972224 280445160 280367469 280037508 77691 1436 64 1 0.845 4423822 29270 1957 200845810 150291285 279 1308 8647600 9000175 2000040 1 80 8 838240 19105150 19094672 12928256 10478 59 256 5 0.693 498410 443978 860 5728103 108676828 1 26 1813032 0 0 1 96 9 99168 3287490 3286457 1870486 1033 12 128 3 0.672 345491 338440 135 1016875 102404599 0 2 1322425 0 0 0 112 10 47936 1004624 1004196 520814 428 3 256 7 0.557 236707 233831 523 161640 101040139 0 0 920633 0 0 0 128 11 1920 72449 72434 34429 15 1 32 1 0.468 29968 31430 8 4 100178461 0 0 122517 0 0 0 160 12 122560 265119 264353 130695 766 7 128 5 0.854 84727 80131 208 110914 100374665 0 1 328501 0 0 0 192 13 67584 36566 36214 20108 352 7 64 3 0.785 17313 18892 253 1217 100139778 0 0 72627 0 0 0 224 14 32704 12968 12822 6559 146 2 128 7 0.570 5725 7089 47 1 100103527 0 0 25547 0 0 0 256 15 120832 8152 7680 4348 472 30 16 1 0.983 3931 4585 323 1129 100098153 0 0 17681 0 0 0 320 16 401600 11880 10625 6153 1255 20 64 5 0.980 4975 5728 87 535 100101613 0 0 22395 0 0 0 384 17 108288 4213 3931 11055 282 11 32 3 0.801 3329 637497 41 845 100729621 0 0 1265542 0 0 0 448 18 114240 21325 21070 10033 255 5 64 7 0.796 7995 8453 43 6266 100111179 0 0 33433 0 0 0 512 19 72704 18287 18145 10789 142 28 8 1 0.633 9399 10335 1161 7170 100117084 0 0 39769 0 0 0 640 20 2702080 168427 164205 80823 4222 143 32 5 0.922 42270 42029 645 103111 100238754 0 1 169086 0 0 0 768 21 15196416 60850 41063 34547 19787 1237 16 3 0.999 16920 16234 2572 11584 100153609 0 0 67295 0 0 0 896 22 264320 72707 72412 31394 295 11 32 7 0.838 20426 20602 144 38908 100157189 0 0 82325 0 0 0 1024 23 193536 100010 99821 54024 189 57 4 1 0.828 30864 30538 1696 91269 100202121 0 0 122305 0 0 0 1280 24 679680 235701 235170 145788 531 59 16 5 0.562 50681 49634 402 203661 100325206 0 0 199913 0 0 0 1536 25 479232 146386 146074 85820 312 55 8 3 0.709 41784 39152 765 131288 100251313 0 1 161185 0 0 0 1792 26 650496 318927 318564 146460 363 25 16 7 0.907 74274 73175 936 249642 100378182 0 0 293977 0 0 0 2048 27 382976 159531 159344 79195 187 98 2 1 0.954 55531 54886 17316 140453 100307220 0 1 220258 0 0 0 2560 28 194560 112026 111950 61334 76 11 8 5 0.863 44087 45190 986 78945 100232988 0 0 177891 0 0 0 3072 29 21504 75869 75862 65823 7 3 4 3 0.583 36762 46287 2915 44596 100217189 0 0 164331 0 0 0 3584 30 14117376 125760 121821 94823 3939 552 8 7 0.891 45180 49533 2029 106614 100261610 0 0 192367 0 0 0 4096 31 876544 82168 81954 55484 214 214 1 1 1 37899 45060 82168 0 100378013 0 0 165105 0 0 0 5120 32 8396800 164063 162423 151421 1640 434 4 5 0.944 83456 111692 5726 115307 100389893 0 0 387396 0 0 0 6144 33 42491904 182820 175904 172894 6916 3742 2 3 0.924 78628 103159 11426 148786 100404197 0 0 361817 0 0 0 7168 34 3569664 110611 110113 93101 498 184 4 7 0.676 56787 76351 2864 82668 100290629 0 0 264729 0 0 0 8192 35 2686976 83367 83039 59805 328 328 1 2 1 44987 57338 83367 0 100402190 0 0 203423 0 0 0 10240 36 471040 106323 106277 78063 46 26 2 5 0.884 59327 76363 10496 71410 100304013 0 0 268509 0 0 0 12288 37 86016 135554 135547 104523 7 7 1 3 1 74638 97958 135554 0 100608560 0 0 341289 0 0 0 14336 38 100352 161549 161542 131155 7 4 2 7 0.875 88665 119265 21831 99069 100432129 0 0 409423 0 0 0 large: size ind allocated nmalloc ndalloc nrequests curlextents 16384 39 589824 84695 84659 115712 36 20480 40 0 0 0 56450 0 24576 41 0 55009 55009 100456 0 28672 42 0 42247 42247 71400 0 32768 43 0 24960 24960 38206 0 40960 44 0 65954282 65954282 65954282 0 49152 45 98304 14636 14634 14636 2 57344 46 0 4634 4634 4634 0 65536 47 0 5468 5468 5468 0 81920 48 0 6757 6757 6757 0 98304 49 0 7958 7958 7958 0 114688 50 0 14404 14404 14404 0 131072 51 0 28383 28383 28383 0 163840 52 0 93229 93229 93229 0 196608 53 0 48542 48542 48542 0 229376 54 0 6749 6749 6749 0 262144 55 0 883 883 883 0 327680 56 0 852 852 852 0 393216 57 0 747 747 747 0 458752 58 0 600 600 600 0 524288 59 0 622 622 622 0 655360 60 0 969 969 969 0 786432 61 0 964 964 964 0 917504 62 0 514 514 514 0 1048576 63 0 415 415 415 0 1310720 64 0 625 625 625 0 1572864 65 0 636 636 636 0 1835008 66 0 263 263 263 0 2097152 67 0 133 133 133 0 2621440 68 0 240 240 240 0 3145728 69 0 201 201 201 0 3670016 70 0 98 98 98 0 4194304 71 0 121 121 121 0 5242880 72 0 112 112 112 0 6291456 73 0 69 69 69 0 7340032 74 0 54 54 54 0 8388608 75 0 48 48 48 0 10485760 76 0 15 15 15 0 --- arenas[2]: assigned threads: 1 uptime: 9184717246171156 dss allocation precedence: "secondary" decaying: time npages sweeps madvises purged dirty: 10000 0 0 0 0 muzzy: 10000 0 0 0 0 allocated nmalloc ndalloc nrequests small: 0 0 0 0 large: 0 0 0 0 total: 0 0 0 0

active: 0 mapped: 2097152 retained: 0 base: 28944 internal: 0 metadata_thp: 0 tcache_bytes: 288 resident: 32768 n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds large 100083157 0 0 1 0 0 0 extent_avail 100083157 0 0 1 0 0 0 extents_dirty 100083157 0 0 1 0 0 0 extents_muzzy 100083157 0 0 1 0 0 0 extents_retained 100083157 0 0 1 0 0 0 decay_dirty 100083161 0 0 2 0 0 0 decay_muzzy 100083161 0 0 2 0 0 0 base 200166306 0 0 2 0 0 0 tcache_list 100083158 0 0 2 0 0 0 bins: size ind allocated nmalloc ndalloc nrequests curregs curslabs regs pgs util nfills nflushes nslabs nreslabs n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds 8 0 0 0 0 0 0 0 512 1 1 0 0 0 0 100083157 0 0 1 0 0 0 16 1 0 0 0 0 0 0 256 1 1 0 0 0 0 100083157 0 0 1 0 0 0 24 2 0 0 0 0 0 0 512 3 1 0 0 0 0 100083157 0 0 1 0 0 0 32 3 0 0 0 0 0 0 128 1 1 0 0 0 0 100083157 0 0 1 0 0 0 40 4 0 0 0 0 0 0 512 5 1 0 0 0 0 100083157 0 0 1 0 0 0 48 5 0 0 0 0 0 0 256 3 1 0 0 0 0 100083157 0 0 1 0 0 0 56 6 0 0 0 0 0 0 512 7 1 0 0 0 0 100083157 0 0 1 0 0 0 64 7 0 0 0 0 0 0 64 1 1 0 0 0 0 100083157 0 0 1 0 0 0 80 8 0 0 0 0 0 0 256 5 1 0 0 0 0 100083157 0 0 1 0 0 0 96 9 0 0 0 0 0 0 128 3 1 0 0 0 0 100083157 0 0 1 0 0 0 112 10 0 0 0 0 0 0 256 7 1 0 0 0 0 100083157 0 0 1 0 0 0 128 11 0 0 0 0 0 0 32 1 1 0 0 0 0 100083157 0 0 1 0 0 0 160 12 0 0 0 0 0 0 128 5 1 0 0 0 0 100083157 0 0 1 0 0 0 192 13 0 0 0 0 0 0 64 3 1 0 0 0 0 100083157 0 0 1 0 0 0 224 14 0 0 0 0 0 0 128 7 1 0 0 0 0 100083157 0 0 1 0 0 0 256 15 0 0 0 0 0 0 16 1 1 0 0 0 0 100083157 0 0 1 0 0 0 320 16 0 0 0 0 0 0 64 5 1 0 0 0 0 100083157 0 0 1 0 0 0 384 17 0 0 0 0 0 0 32 3 1 0 0 0 0 100083157 0 0 1 0 0 0 448 18 0 0 0 0 0 0 64 7 1 0 0 0 0 100083157 0 0 1 0 0 0 512 19 0 0 0 0 0 0 8 1 1 0 0 0 0 100083157 0 0 1 0 0 0 640 20 0 0 0 0 0 0 32 5 1 0 0 0 0 100083157 0 0 1 0 0 0 768 21 0 0 0 0 0 0 16 3 1 0 0 0 0 100083157 0 0 1 0 0 0 896 22 0 0 0 0 0 0 32 7 1 0 0 0 0 100083157 0 0 1 0 0 0 1024 23 0 0 0 0 0 0 4 1 1 0 0 0 0 100083157 0 0 1 0 0 0 1280 24 0 0 0 0 0 0 16 5 1 0 0 0 0 100083157 0 0 1 0 0 0 1536 25 0 0 0 0 0 0 8 3 1 0 0 0 0 100083157 0 0 1 0 0 0 1792 26 0 0 0 0 0 0 16 7 1 0 0 0 0 100083157 0 0 1 0 0 0 2048 27 0 0 0 0 0 0 2 1 1 0 0 0 0 100083157 0 0 1 0 0 0 2560 28 0 0 0 0 0 0 8 5 1 0 0 0 0 100083157 0 0 1 0 0 0 3072 29 0 0 0 0 0 0 4 3 1 0 0 0 0 100083157 0 0 1 0 0 0 3584 30 0 0 0 0 0 0 8 7 1 0 0 0 0 100083157 0 0 1 0 0 0 4096 31 0 0 0 0 0 0 1 1 1 0 0 0 0 100083157 0 0 1 0 0 0 5120 32 0 0 0 0 0 0 4 5 1 0 0 0 0 100083157 0 0 1 0 0 0 6144 33 0 0 0 0 0 0 2 3 1 0 0 0 0 100083157 0 0 1 0 0 0 7168 34 0 0 0 0 0 0 4 7 1 0 0 0 0 100083157 0 0 1 0 0 0 8192 35 0 0 0 0 0 0 1 2 1 0 0 0 0 100083157 0 0 1 0 0 0 10240 36 0 0 0 0 0 0 2 5 1 0 0 0 0 100083157 0 0 1 0 0 0 12288 37 0 0 0 0 0 0 1 3 1 0 0 0 0 100083157 0 0 1 0 0 0 14336 38 0 0 0 0 0 0 2 7 1 0 0 0 0 100083157 0 0 1 0 0 0 --- large: size ind allocated nmalloc ndalloc nrequests curlextents --- --- End jemalloc statistics ---`

Comment From: 627721565

您好，邮件已经收到，查阅后会尽快给您回复！

Comment From: oranagra

Allocated: 5503413744, active: 5827477504, metadata: 172479648 (n_thp 0), resident: 6011027456, mapped: 6031572992, retained: 10898735104

10gb retained!!

maybe you have THP enabled? do you see a warning about Transparent Huge Pages in the redis log when it starts?

please post your

cat /sys/kernel/mm/transparent_hugepage/enabled

Comment From: promisingMan

I use this command cat /sys/kernel/mm/transparent_hugepage/enabled and the result shows: [always] madvise never

I checked the number of huge pages already allocated by the system and the result shows:

HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0

I found that the thp configuration of redis is disabled： Redis mem_fragmentation_ratio is too high

I found the warning when redis starts:

23336:C 24 Dec 2022 16:40:26.749 # Redis version=6.2.4, bits=64, commit=00000000, modified=0, pid=23336, just started
23336:M 24 Dec 2022 16:40:26.751 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').

the memory doctor show that:

Sam, I detected a few issues in this Redis instance memory implants:

 * Peak memory: In the past this instance used more than 150% the memory that is currently using. The allocator is normally not able to release memory after a peak, so you can expect to see a big fragmentation ratio, however this is actually harmless and is only due to the memory peak, and if the Redis instance Resident Set Size (RSS) is currently bigger than expected, the memory will be used as soon as you fill the Redis instance with more data. If the memory peak was only occasional and you want to try to reclaim memory, please try the MEMORY PURGE command, otherwise the only other option is to shutdown and restart the instance.

 * High total RSS: This instance has a memory fragmentation and RSS overhead greater than 1.4 (this means that the Resident Set Size of the Redis process is much larger than the sum of the logical allocations Redis performed). This problem is usually due either to a large peak memory (check if there is a peak memory entry above in the report) or may result from a workload that causes the allocator to fragment memory a lot. If the problem is a large peak memory, then there is no issue. Otherwise, make sure you are using the Jemalloc allocator and not the default libc malloc. Note: The currently used allocator is "jemalloc-5.1.0".

 * High process RSS overhead: This instance has non-allocator RSS memory overhead is greater than 1.1 (this means that the Resident Set Size of the Redis process is much larger than the RSS the allocator holds). This problem may be due to Lua scripts or Modules.

I'm here to keep you safe, Sam. I want to help you.

This is the information I found, I hope to get your guidance, thank you very much.

Comment From: oranagra

so the disable-thp is set to yes, and yet you get the warning and the side effects of THP? which OS are you using? is it really kernel 2.6? (in which case PR_SET_THP_DISABLE isn't supported). either way, i suggest you disable THP echo madvise > /sys/kernel/mm/transparent_hugepage/enabled as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. restart redis and report if the problem happens again...

Comment From: promisingMan

I use this command /proc/meminfo | grep -i huge to find that the memory allocated by thp is 8155136 kB, so it matches the memory displayed by smap.

AnonHugePages:   8155136 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

So this parameter disable-thp does not take effect when the system thp parameter is configured as always?

Comment From: promisingMan

The kernel version of my machine is Linux 2.6.32-696.6.3.el6.x86_64 x86_64

Comment From: oranagra

Yes. Your kernel doesn't support it, which is why you still get the warning on startup. You'll have to change your system configuration and restart the process.

Comment From: promisingMan

Back to my question at the beginning : my problem is, mem_fragmentation_ratio(2.26) is much larger than allocator_frag_ratio(1.06), I want to know where the memory not held by the allocator goes, how can I reclaim this part of memory. why does thp cause this to happen?

Comment From: promisingMan

Yes. Your kernel doesn't support it, which is why you still get the warning on startup. You'll have to change your system configuration and restart the process.

So my machine's thp is on, even if the disable-thp is set to yes

Comment From: oranagra

Your machine THP is set to always. Redis can attempt to disable it on the process (without affecting the system) but that isn't supported by your (very old) kernel.

Comment From: promisingMan

Back to my question at the beginning : my problem is, mem_fragmentation_ratio(2.26) is much larger than allocator_frag_ratio(1.06), I want to know where the memory not held by the allocator goes, how can I reclaim this part of memory. why does thp cause this to happen?

"Would you be able to help me with this question? Thank you very much.🙏"

Comment From: oranagra

AFAIK retained means memory that was freed by the allocator (madvise), without discarding the virtual memory space (munmap), in that case the the can kernel still associates it with the process, but it can also reclaim it when it needs to. i.e. it doesn't mean this memory is really resident (preventing other processes from using that physical memory), but my understanding of the mechanics of THP are limited, so i could be wrong.

in any way, your configuration (THP being enabled) is invalid for Redis, so you must fix it and restart the process to resolve the problem.