Module API V1

This is an attempt to scope out all missing or broken Redis Module API capabilities, and create an exahustive list that defines a stable V1 interface.

The Module API was in continuous state of development since Redis 4. Although it maintains ABI backwards compatibility, this does make things more difficult for module developers to mange.

General

Better documentation is desparately needed. Things that come in mind:

A step by step / tutorial-like stuff.
Better organize the growing reference for better navigation, etc.
Some real world examples.

Big Topics

New, significant capabilities.
Need work to define the concept or reach a solid API design.
Potentially have a system-wide effect.
[ ] Asynchronous RM_Call support (#7992)
[ ] An API to query command flags. This should be done in tandem with improved command introspection, like #7297 and can also provide help as described in #4175 and an alternative to #3700 (#7991)
[ ] I/O APIs for use cases such as encryption or compression of AOF/RDB files, streaming files directly to and from non-file sink or source
[x] RESP3 Support (#6860, No. 1) (#8521, #9202)
[x] Opt-in ACL support for modules
[ ] APIs to extend ACLs and built-in users (#8329)
[ ] Better support for ACL and module commands (categories?)
[ ] A way for in-place upgrade of a module (old version pass state to new version)

Native Key Access

Provide better coverage for accessing native Redis data types.
We should not try to cover all commands and just provide the necessary primitives.
[x] Streams support as a native type (#5760)
[ ] Missing hash functions:
- [x] RM_HashDel (using RM_HashSet with REDISMODULE_HASH_DELETE).
- [x] RM_HashLen (using RM_ValueLength).
- [ ] Field iteration API, similar to the one provided by sorted sets.
[x] Missing list functions (see #8439):
- [ ] RM_ListGet
- [ ] RM_ListInsert
- [x] RM_ListLen (using RM_ValueLength).
- [ ] RM_ListRemove
- [ ] RM_ListSet
- [ ] Iteration API
[ ] Missing set functions:
- [ ] RM_SetAdd
- [ ] RM_SetCard
- [ ] RM_SetIsMember
- [ ] RM_SetRemove
- [ ] Iteration API
[ ] Optimize the API to provide zero-copy (but safe) access to elements. see #8473

Minor Improvements / Fixes

Need some level of API design or decision making.
Impact has a limited scope.
[ ] Module Pub/Sub subscribe support (#7298)
[x] RM_HashSet return code is ambigious (#6914)
[ ] All variadic API functions should also have a non-variadic version (#7860)
[ ] Support opt-in keyspace notification on native data type access (#6544)
[ ] Better way to handle RM_BlockClient when called in a context that does not allow that (#5567)
[x] Add samples support to mem_usage callback (#4177)
[ ] Module command validation step for better MULTI/EXEC handling (#3635)
[x] Add RM_KeyExists function (#6860, No. 11)
[ ] Additional hooks: Before fork, before keyspace iteration on main thread, before/after AOF rewrite to allow aux data storage (#7350, #6860 No. 12+13))
[ ] Extend defrag API to handle late-defrag of globals and defrag of RAX structures.
[ ] Create more unit tests to improve module API coverage
[x] Graduate from experimental API some of the stuff (#6860, No. 4)

Trivial

[ ] Optimize network usage when REDISMODULE_POSTPONED_ARRAY_LEN is used (#6444)
[ ] SendClusterMessage crashes thread safe context (#5500)
[ ] Get hash slots for a Cluster Node (#5371)
[ ] Concurrent MODULE UNLOAD and a module command in MULTI/EXEC will crash (#4826)
[ ] Provide module file name in MODULE LIST (#4413)
[x] Fix RM_StringTruncate() with a zero length which currently does nothing, or document the otherwise unexpected behavior (#3717)
[ ] Fix RM_WrongArity() crash on thread safe context (#4756)
[ ] Add NOTIFY_MODULE (Need to better understand the use case) (#6963)
[ ] Fix RM_BlockClient misuse leading to crashes (#4366)
[x] Optimize RM_Call by recycling a single client (#4623)
[ ] Refuse to load a module without aof_rewrite callback if preamble is disabled, or disable preamble if such modules are loaded

Module API V2 and beyond

Big things beyond the current Module API V1 effort.

[ ] Expose event loop for arbitrary file descriptor I/O (#4158)
[ ] Expose the connection abstraction layer to allow modules to do arbitrary TCP and TLS

Comment From: zuiderkwast

@yossigo Please mark as done above:

~~RM_HashLen → already covered by RM_ValueLength~~ :heavy_check_mark: Marked
~~RM_HashDel → already possible by setting a field to the special value REDISMODULE_HASH_DELETE using RM_HashSet.~~ :heavy_check_mark: Marked
~~RM_ListLen → already covered by RM_ValueLength~~ :heavy_check_mark: Marked
~~RM_HashSet return code is ambigious (PR: #8446, issue #6914)~~ :heavy_check_mark: Marked
RM_ZsetRem may leave empty sorted set keys, leading to crashes (done in a different PR: #8453) -- :question:
~~Optimize RM_Call by recycling a single client (done in a different PR: #8516)~~ :heavy_check_mark: Marked
~~Missing list functions (done in #8439)~~ :heavy_check_mark: Marked
~~Add RM_KeyExists function (done in #9600)~~ :heavy_check_mark: Marked

Comment From: hwware

Is there anyone working on the missing list and set functions? If not then we'd like to take a look at it and contribute.