The problem/use-case that the feature addresses the MP-AOF is powerful and it's history files both base and incr files are still useful, Multiple extended functions can be done based on these files, such as snapshot backup, incr log subscription , and so on Now the aof history files was auto removed by redis after new file was created, we want to have a config that history file not to be deleted by redis.
Description of the feature the new configs:
aof-no-delete-history-base-file yes
aof-no-delete-history-incr-file yes
this configs default could be no, maintain compatibility with older versions. when administrators need the historical files, it could change to yes, which means that all historical files are handed over to the administrators
thanks for reading, wish to reply
Comment From: enjoy-binbin
we do have a "aof-disable-auto-gc" config (control the history one but not incr), now it is just for testing. Not mention in redis.conf, i suppose you will read the source code and find the meaning
Comment From: oranagra
retaining just the history files may not be enough. you may also need to keep copies of the manifest files.
i think a better way would be to make snapshots of the whole folder using hard links, as described here: https://redis.io/docs/manual/persistence/#backing-up-aof-persistence
Comment From: judeng
we do have a "aof-disable-auto-gc" config (control the history one but not incr), now it is just for testing. Not mention in redis.conf, i suppose you will read the source code and find the meaning
thank you, I just readed the source code, please point it out if I understand it wrong, it seems that in the case of aof-disable-auto-gc is yes, the server will not delete all the history files, and the history file information is still saved in the manifest file, and there is no means to manually delete the history files, unless we set aof-disable-auto-gc is no. It seems to be called "aof-disable-gc" is more appropriate.
in my case, I want to keep only the history files of the last day forever for the log subscription function. In other words, I don't want to clear all the history files every time, I just want to clean up expired files that are too old. Is there any other better way to do it?
Comment From: oranagra
@judeng what about the snapshoting suggestion using hard-links?
Comment From: judeng
@oranagra thank you for your reply, in my case, I only want to backup the most recent base file, not the current data snapshot, it have less expensive and enough backup for me.
Comment From: oranagra
ok, so you wanna use AOF for a form of snaphost instead of using the other snapshots save mechanism.
we do intend to unify them into one mechanism, probably in 7.2.
anyway, i still think that instead of configuring redis not to delete these, keep track of them up to a certain age, and list them in the manifest, it is preferable to do that outside of redis, but using hard links, and deleting them at a certain age.
am i wrong? any reason why that's not sufficient or why you think it should be a build in feature of redis?
Comment From: chenyang8094
I think a good practice is that if redis senses that there are external systems backing up these AOF files (the external system can send specific commands to redis, such as disable_auto_gc), then redis will not automatically delete these history AOFs, on the contrary, When the backup system confirms that a history AOF has been safely backed up, it sends a delaof aof_name command to redis, and then redis deletes the history AOF. Then the backup system will continue to back up the next history AOF, and so on.
There are many things to consider, such as whether the AOF deletion command needs to be synchronized between the master and standby, How to prevent redis from being unable to delete the history aof due to the downtime of the backup component, resulting in the accumulation of AOF and so on.
Comment From: judeng
@oranagra Sorry for the late reply, I thought about it carefully, it is really better to put it outside the server, but what I point is that redis does not clean up the historical files, and the manifest will not track these historical files, the historical files are completely handed over to the administrator, they can The sequence number in the file name goes to backup, or consumption; historical files are no longer visible to redis. When the administrator intends to back up the latest files managed by manifest, use hard link
Comment From: oranagra
@judeng some parts of your last comment are unclear to me. like: redis does not clean up the historical files.
i suppose you're referring to some theoretical solution / design, and not the current state.
so i suppose the bottom line is that we agree that the current approach is good, and that is someone wants to keep older snapshots of the AOF, they should do that from outside of Redis, by using hard links.
Do we agree on that?
Comment From: judeng
yes，it is
Comment From: judeng
@chenyang8094 This is a complete design, but I think the implementation would be too complicated that the AOF deletion command synchronized with replica :-)