Redis [NEW] AOF annotations

^{Originally posted by **ShooterIT** July 29, 2021} As we know, in AOF, there are only Redis commands, in some cases, we may want to know context of commands for updating database. In RocksDB, it allows us to annotate each `Put` with some arbitrary metadata, so we can implement interesting features based on that. Of course, Redis is a server instead of a library, not casual for changing data format. But there may be several common requires that we can support, at least, we also can make "annotations" is legal in AOF, AOF tools could skip them if they don't want to parse them. In current idea, the annotations doesn't fully break AOF format, just add some lines that starts with `#`, like some languages annotation, after `#`, we could add useful information which we want. The annotations are only text instead of binary safe, of course, there can be multi lines together to explain context much clearly. The followings are several scenarios I think of. - Restore from a specific point-in-time Users may ask us to provide one ability: restore data from a specific point-in-time, but there is no time info even during writing AOF (rather than AOF rewrite). One idea is that add some timestamp in AOF, timestamp precision may be minute or second, to avoid making AOF over-inflated. If there are some timestamps, it is easy to restore from a specific point-in-time by replaying AOF to a certain point in timestamp. ``` #timestamp SET A B SET B C #timestamp SET C D SET D E ``` - Keys eviction Redis sometimes evicts keys silently, especially before #7653, actually, sometimes we may want to know keys are evicted or deleted, and the reason why redis start to evict keys (we lose context forever after finishing eviction, moreover, since of eviction is blocking, we can't learn useful things by `INFO` command), so it is very useful to record these important infos if possible. ``` # start to evict keys since of maxmemory is 10GB, but used memory is 11GB # overhead memory 500MB, dataset memory 8GB, ... # timestamp DEL A (RESP) DEL B (RESP) # end evicting keys ``` - PSYNC-AOF AFAIK, there is not only one company that implement PSYNC based on AOF, antirez also opened an issue #4357 to discuss this topic, he also designed one kind of AOF annotations for implementing PSYNC based on AOF. Maybe we can record some commands' offset, after syncing several commands in AOF, and then force to reset replica's replication offset by the record replication offset in AOF (because the content of AOF is not the same as replication backlog). This feature may provide convenience for implementing PSYNC-AOF. ``` # replication offset $offset SET A B SET C D ... # replication offset $offset EXPIRE A 10 EXPIRE C 10 ``` - Dangerous commands In production environments, for some dangerous command, we usually block them by `rename-command` or ACL rules, but if actually they are executed, we still want to know who did this operation, when it happened, maybe other things, and we may also want to get this information fo audit. Moreover, for `slowlog` that update database, we maybe could put the metrics of slow log around the write command, so we could see much more slow logs on disk instead of just only those on limited memory. Of course, we can record these in server log, but we always store these commands in AOF, so it is not costly, but this is worth arguing. ``` # timestamp # host: 127.0.0.1 # ACL user flushall flushdb ``` If you do secondary development based on Redis, maybe you can add more infos for your purpose, such as active-active geo-distribution, audit log. Welcome to communicate!