With current code, if anyone will enable active memory defragmentation on a server with modules who have their own data types, it can cause an overload of CPU. The defragger will detect a fragmentation problem, and will consume a lot of CPU cycles to try and fix it, but will be unable to do so, and will continue to burn CPU cycles.
what should be done is allow modules with data types to implement callback for defragmentation work. this should be very similar to the DEBUG DIGEST mechanism we have today. Redis will call a module's callback for each key, and the module will iterate on it's data structure, calling redis defrag function on each pointer. then, if the defrag function returns a non-null value, the module has to replace the pointer in it's data structures, and do any necessary adjustments on the data in the new pointer (without accessing the old pointer which was already released).
Comment From: oranagra
@antirez do we wanna tackle this for v6.0? As far as i can tell, the API should include:
- each module should announce support (yes,no,forbidden)
- defrag callback per module type key
- defrag callback for out of keyspace globals
- api to defrag a pointer (which returns non-null if the pointer was moved and references should be fixed)
- api to defrag redis primitives like dict and string
- on each callback a module can use an API to check if time is up and exit early before the iteration on the key is complete, and resume later.
- if the module exited early, it will be called again on the same key on the next interval.
- modules can use static variables or private data context to be able to resume scans that exited early.