The problem

In Redis 7 we introduced multi-part AOFs (#9788) based on a manifest file that indicates which files are part of Redis's AOF persistence. Our updated documentation about backup tells the user to simply copy the directory where all the AOF files (and manifest) reside:

https://github.com/redis/redis-doc/blob/fa946440a75b508edc681470b9300efea34d6462/topics/persistence.md?plain=1#L334-L335

This, of course, is wrong and bad advice. The reason is that during the copy files in the directory might change. During a rewrite we might end up copying an old manifest file and new base and increment files. We'll end up with a useless backup.

A word about Redis and backups

Backups are either periodic or a one-shot thing. They're not ongoing. For ongoing "backup" Redis has replication. For periodic backups Redis recommends backing up the periodic snapshot (RDB) files. RDBs are perfect for backup requirements. So there's no real logic in backing up AOF persistence anyway, except for one edge case: If you don't want to pay the cost of the extra forks and disk space of periodic RDB snapshoting and you already have AOF persistence configured. This single excuse for backing up AOFs is short lived since we already use RDBs as the default base format for AOF persistence and plan to leverage this so we can get rid of the dual AOF/RDB persistence model.

Possible solutions

I see two ways to go here:

No official support AOF backups

Drop official support for AOF backups and document this. Once AOF and RDB persistence models are merged this will resolve itself. And until then users can simply enable RDB snapshotting for their backup needs.

AOF backup tool

We can resolve the atomicy issues of copying the appendonlydir by creating a tool (can be a bash script) that'll make sure we produce a valid AOF directory backup. Something like this: 1. Verify the server isn't performing a rewrite. 2. Create hard links to files in the directory. 3. Verify no rewrite started or happened since (1). If it did delete the hard links and go back to (1). 3. Copy/gzip the hard links. 4. Delete the hard links.

The script can even be just a blog post with referral from our official docs saying we recommend RDB snapshots as backups, but if you insist you may use this script or something similar.

@yossigo @chenyang8094 @oranagra WDYT?

Comment From: oranagra

i vote for a bash script

Comment From: chenyang8094

@yoav-steinberg Thank you for taking this into consideration, I think it does solve some users' backup problems. i vote for a bash script too.

Comment From: oranagra

Note, it was eventually decided not to include a backup script in the redis repository and instead document how to safely backup AOF files, see: https://github.com/redis/redis-doc/pull/1794