The problem/use-case that the feature addresses

I am using Redis as a disk-based database (i.e., configured with AOF mode), where the write performance is the most concerning. Currently, the Redis uses the write(2) syscall to issue requests. There may be two limitations: - (1) write(2) is a synchronized I/O interface, which means only one request can stay in-flight. While many new devices can have very deep queue depth, which can serve many requests at a time. - (2) even with asynchronized I/O,
> as devices get extremely fast, interrupt-driven work is no longer as efficient as polling for completions — a common theme that underlies the architecture of performance-oriented I/O systems.

Description of the feature

  • For (1), I suggest making Redis support asynchronized I/O. It is very encouraging that PostgreSQL has already working on it. To achieve so, possible choices are: libaio, io_uring, etc. Since Redis does not manage buffer itself (relying on OS page cache),libaio will fall back to synchronous (in buffered-IO mode and with appendfsync=always).
  • For (2), it would be awesome if Redis can support io_uring as a possible I/O interface. Even if completely changing write to io_uring may bring new problems. Shall we make it configurable (e.g., adding a parameter use-io-uring)?

Thanks & best.

Comment From: oranagra

closed in favor of #10882

Comment From: oranagra

ohh, sorry, i got carried away, confusing it with the other dup, but actually it's a different topic. re-opening.

Comment From: oranagra

for the record, we do already have plans to improve IO threads (hopefully in the near future) to better offload writes (and TLS) towards client, and we are also already looking into io_uring for some time now, but i'm not certain any of that is gonna effect AOF.

Comment From: TimHe95

Very looking forward to that version! Thanks for the answer and support.

Comment From: oranagra

@TimHe95 can you just clear out the AOF topic? our plans is to use io_uring and threads mainly to scale writes to sockets (run in parallel). if AOF is your bottleneck, i'm not certain if what we have in mind is gonna make a difference.

Comment From: uvletter

@oranagra unfortunately I think you has a misunderstanding and the concern of this issue is about disk IO, some keyword like queue depth, libaio can prove it, and io-uring is usually efficient in disk IO scene but arguable in performance comparing with multiplexing e.g. epoll. This issue may wanna us optimize the AOF write path with AIO or io-uring.

Comment From: oranagra

ok, so the reference to IO threads is confusing. first since IO threads today are only used for concurrent write to sockets. but secondly, since i don't see any way this could be applicable for AOF.

We can maybe compromise on the guarantees of AOF (when not using appendfsync=always) to use async IO, this would mean that we'll lose more writes even when redis crashes (not only on power failures), but i'm not certain it's an actual issue.

@TimHe95 maybe you can re-write the top comment to make it clearer what you're suggesting. and also avoid confusing references if they don't help your case.

also, maybe it'll help if you give some background on your use-case / configuration.

Comment From: uvletter

An unshaped thought, why not fsync AOF in the background thread(or AIO etc.), and put the client into clocked, thus other processing won't be blocked simultaneously. Once the operation is done, a callback will recovery the blocked client, replying to the request.

Comment From: yaxing

for the record, we do already have plans to improve IO threads (hopefully in the near future) to better offload writes (and TLS) towards client, and we are also already looking into io_uring for some time now, but i'm not certain any of that is gonna effect AOF.

Apologies if this is not the focus on this thread, but @oranagra we're actually waiting on this improvement for TLS, is there a thread/issue where we can track the progress? thank you!!

Comment From: oranagra

@yaxing sorry, the only thing i could find was a note in our backlog project without a corresponding issue, so i created one: #11119

@TimHe95 i'm still not sure i understand the issue for which you opened here. to the best of my understandings our plans to use io_uring and and improve our IO threading are only intended to serve client connections.

Comment From: soloestoy

To simplify this issue IIUC, users wanna run redis with appendonly yes, but hope redis could be not blocked by write(2) when flushAppendOnlyFile().