I have a redis module that uses a third-party library which handles all memory allocations internally.
Since I don't use any RM_Alloc functions, the INFO command will not take into account my RedisModuleType variables.
It would be nice to have a way to proactively inform redis about the module's current memory usage (alloc/free), perhaps through a command like RM_UpdateMemoryUsage(ssize_t mem).
Does this make sense? Would you be interested in a pull request?
Comment From: itamarhaber
I fully agree that INFO should report the overall resources usage, modules included. The current memory section details the server-managed RAM usage and I don't think that should be changed. Perhaps a new modules section is a better approach, in which each module's signature and resources can be included.
Also, to have a module report about non-RM_Alloc stuff, it would be better to have an API that accepts the delta rather than an absolute value in order to let the server keep track of accounting. Reporting the absolute value requires the module author to manage that, whereas this approach would require just calling the method with whatever size was just allocated/freed.
That said, I'm not sure that this is a bullet-proof solution. Some 3rdpty may not provide accurate RAM introspection capabilities, preventing any module using them from making meaningful reports.
Comment From: antirez
Hello, I agree on the general idea that the modules memory should be part of Redis reported memory, however the API proposed by @aviggiano, while it works, is going to be extremely fragile, once you miss a malloc or free and the count starts to be monotonically increasing/decreasing. For this reason Redis itself internally resorts to the allocator help, or stores the size inside the chunk if the malloc does not provide malloc_size(), becuase otherwise is very unreliable.
If there will not be better solutions, an alternative could be to use the API proposed by @aviggiano but track the memory usage per-module in a different counter, so that at least if a module is buggy only the module reported memory will be affected, and not the rest of Redis.
Btw ideally third-party libraries should just have a way to use a different allocator compared to the default one. Note that to re-compile the lib to use a different memory allocator is just a matter of #defineing a few things in the header file, so that all the call to malloc/free/realloc/... are remapped.
Comment From: aviggiano
@itamarhaber My initial idea was to provide the delta instead of the absolute value, but I agree with you that it might not always be possible. @antirez is also right when he says that either way this API would be very fragile, since the developer would have to report the allocation by hand.
Indeed the best solution would be to provide a different allocator to the library, but I am not sure if all of them are designed that way.
In my specific case it is not, so I think I'll open an issue on their repo proposing this change. In the worst case I will remap their allocation calls to RM_Alloc and so on.
Comment From: itamarhaber
Now that modules can add their own INFO sections (https://redis.io/topics/modules-api-ref#coderedismoduleinfoaddsectioncode) this is partly resolved, at least for reporting purposes. Theoretically, one could interpret the request to have Redis also enforce maxmemory on the module's external RAM consumption, but as RAM is just another resource, it would be better if the module itself would manage its resources.
Closing as resolved.