October 2 in London a number of parties interested in Redis developments met in order to talk about future directions of the project. During the chats, a few critical issues that should be fixed as soon as possible were identified. This issue will list each issue, and for each issue a sub-issue will be created in order to discuss its design. The list now covers much more than discussed at the developers meeting.

Redis Server

• [DONE] Diskless replication: directly send the RDB file from the forked child to the slave, in order allow to replicate without touching the disk. Motivations: the main use case is environments where disks are a source of latency and there are no other reasons for the master to write to disk (for example cache usage where replication is used in order to have a pre-populated backup instance). Similarly in setups where persistence is provided by slaves since disks are not able to cope, this completely avoids on disk RDB generation. Status: merged into unstable, 3.0, 2.8, and documented at redis.io.
• RDB preamble on AOF to speedup both AOF rewriting and reloading.
• [DONE] Merge @mattsta Geocoding commands. This requires some refactoring but nothing that can't be done in the context of the unstable branch. See issue #1745 for comments reporting progresses.
• Memory efficient small Sets storage. Currently small Sets are stored in a memory efficient way only when composed of numbers.
• [DONE via Matt's quicklists] More memory efficient Lists (linked lists of ziplists).
• Lua script single commands replication mode, that can be enabled by the script with a special command.
• Linux specific OOM settings, to improve behavior under memory pressure. For example the slaves should be favored. Also there is to investigate if getting info via /proc about memory usage, caches, buffers, it is possible for the Redis process to anticipate when the OOM killer is going to be triggered, and kill the saving child automatically and ASAP. Issue #1690.
• RDB analysis and tools similar to redis-rdb-tools with a robust C implementation, part of redis-cli.
• [DONE] RDB additional key-val fields to store additional information. This can be also used to store the size of DBs in order to give an hint for the hash table resize. This translates in faster RDB loading time.
• RDB partial resync information, so that slaves are able to partial resynchronize after a restart, as long as the master has the same run ID and there is enough backlog.
• Global clients memory usage and limits.
• SSL support, we have an implementation already available here to check. In theory this would be: opt-in (not complied by default), and optimized with different latency / compression tradeoffs in different cases, for example the replication channel could demand high compression, the client-server chat low latency. There is a use case at Square where currently stunnels are used extensively almost only for Redis. At this point we may be in a condition where lack of SSL support in Redis, at least optional, is making things too much complex in certain environments. CCing the PR for merging it into Redis in the original repo of the fork. Stand alone Issue about SSL is here.
• Retain master and slave replication offsets after a restart. This has the effect to make Sentinel slave selection process safer.
• Protected restarts in order to support the use case of a master-slave setup which is entirely diskless (persistence off, and the new diskless replication on), but yet is able to survive the restart of N-1 instances. Basically an instance with protected-restart set to yes, once started, will just reply with an error to queries, unless it is unblocked using the SLAVEOF command, to turn it into either a slave or a master. Issue #2087.
• [ABANDONED IDEA] Enhancements to the HyperLogLogs API. Issue #2100.
• Stream SYNC-less replication: slave connects and just get new updates from the master. Useful for caching with mostly idempotent writes. Reference implementation.
• Pub/Sub UNSUBSCRIBE requires a linear time proportional to the number of subscribers to a given channel.

Redis Sentinel

• Sentinel authentication.
• Sentinel: better handling of changing majority when some sentinel is removed explicitly via the SENTINEL FORGET command.
• Sentinel: simpler ways to remove a master without requiring SENTINEL RESET. Broadcasting of this information.
• [DONE] Better Sentinel documentation with example deployments, maintenance tasks, general rules for deployments before to go deep into algorithms and failure modes descriptions.
• Stand alone state file similar to nodes.conf of Redis Cluster.
• [DONE] Sentinel ID remembered across restarts: the runid will no longer be used for this task.
• [DONE] Sentinel scalability improvements when there are many masters monitored by the same set of Sentinels. Details are described into issue #2257.
• Improve Sentinel desynchronization during the voting process to avoid split brain (no Sentinel getting the majority) that delays the failover to a successive retry.

For a number of the above issues there is already some old pull request. We should link the above entires with the existing PRs, sometimes a new PR will be created when there is additional context available and when it is better to start from scratch with a clean description and design.

Comment From: mattsta

• Replication with streaming RDB: https://github.com/antirez/redis/issues/1501 and https://github.com/antirez/redis/pull/1185

• Linux OOM: https://github.com/antirez/redis/pull/1690

• Related to global client memory usage + limits: https://github.com/antirez/redis/pull/1618

Comment From: antirez

@mattsta thank you. Btw I searched the issues list for a great deal of time yesterday without being able to find the in-memory replication stuff. I'm sure it was posted, either here on the google group. Btw I don't remember it to be a polished implementation, but I was curious at this point about the approach that was taken.

Comment From: antirez

@mattsta p.s. it is neither of what you mentioned, just to be clear. It was an implementation much similar to: directly send the RDB file to the slave.

Comment From: badboy

/subscribe (just so I get updates)

Comment From: amix

First of, great work on Redis! We have been using it for years and it's truly a marvel.

I discussed briefly with antirez today ( https://twitter.com/antirez/status/519057585576894464 ) that Sentinel documentation could be better. The conclusion was following: It should start with example of docs and general rules for a deployment, and later get into failure modes, algorithms, and other low level details.

I added this here so it's not lost :+1:

Thank you and keep on rocking!

Comment From: djanowski

I'm so sorry I missed the meetup :(

It's great to see this list of issues to be prioritized. I'm particularly interested in diskless replication!

Thank you :)

Comment From: jzawodn

+1 for diskless replication (even though we don't have latency issues)

Comment From: michael-grunder

I would like to give a +1 to linked lists of ziplists. I think that would show an exponential reduction in ram usage for zsets.

Comment From: antirez

@michael-grunder implementing this for sorted sets would be very hard or impractical unfortunately. Lists have a mostly append workload, zsets are continuously modified in the middle, and are one of the few operations that is more CPU bound currently. That would require a lot of work to both have the "far" points in the different levels and still be able to degragment the ziplist blobs. At least for now the feature is intended for the "List" type.

Comment From: michael-grunder

@antirez. Maybe one for the future then. We're using a custom data type that keeps things really tiny but it's not I think generic enough for widespread use. Glad the meet up was so productive! :)

Comment From: antirez

Update: Added SSL support as a new point in the wishlist. About RDB diskless creation for replication, I stopped a bit for Redis Cluster RC1 release testing, but actually I'm already half-way to provide an implementation in a branch to link at this issue for community evaluation.

Comment From: mezzatto

SSL would make possible the use of redis as a broker for https://github.com/elasticsearch/logstash-forwarder. Nice for people using logstash (very popular logging manager)!

Comment From: xaviershay

SSL support would be amazing.

Comment From: stouset

Another useful aspect of SSL support is that stunnels only provide host-based security. If you have multiple services running on a machine, any of them can talk over that stunnel. There are solutions, but nothing's as simple as managing UNIX permissions on secret keys.

Comment From: mattsta

Instead of stunnel (bad software anyway), you can run spiped on each host. It even retains the same "one shared password everywhere" semantics that Redis currently follows, except you set the shared password on your spiped tunnels instead of Redis.

Comment From: stouset

spiped doesn't allow for easy key rotation. With stunnel, you can restart each side of the tunnel independently, adding new certificates into the trust store. When both are bounced, you can restart them independently again with new keys (that are now vouched for by the new certificates). Once this is done, you can remove the original certificates, bounce the stunnels again, and carry on.

With spiped, you have to restart both sides of the pipe simultaneously. Even if you launch the second pipe in parallel over a second port, there's still a moment where both services have to be switched to the new UNIX socket simultaneously. This is a PITA to do, especially when you have lots of services talking to these Redises.

Worst, though, is that spiped is a new, custom designed transport protocol. I have nothing but respect for Colin Percival, but TLS (for all its warts) is well-studied, and under constant review from security engineers around the world. The protocol Colin designed for spiped simply hasn't had the breadth and depth of analysis that TLS has had, and that's a risk we're not eager to expose ourselves to.

Comment From: mattsta

While we're listing things... - Fix an error message in Cluster: https://github.com/antirez/redis/pull/1582 - Add bitop to Lua (Redis has bit-level operations, but Lua doesn't expose bit operators without this module, so there's a lot you can't do in scripts): https://github.com/antirez/redis/pull/1662 - Scripting upgrades, including a Solaris compile fix (we currently can't compile on Solaris? Some users are complaining about that): https://github.com/antirez/redis/pull/1663 - Probably worth reviewing other small build Solaris fixes too: https://github.com/antirez/redis/pull/642 - pidfile creation fix (re-submitted from #1906): https://github.com/antirez/redis/pull/1967 - Improve SETBIT and SETRANGE to not over-allocate memory: https://github.com/antirez/redis/pull/2050 - Sentinel INFO improvement: https://github.com/antirez/redis/pull/1966 - Sentinel quorum bounds check: https://github.com/antirez/redis/pull/2054

Other upcoming PRs: - I've been quietly showing off my custom proctitle formatting improvements — when I can convince myself it's okay to keep my ugly parsing code for {{blah}} I'll throw it in a PR. [That's also where a custom config-defined NAME comes in handy. Most people use the config file name to be their "server name," but I think we need an optional unique name too.] - I started working on the linked list of skiplists two weeks ago, but haven't had much time to make progress recently. Hopefully I can get that working and passing all tests in another week or two?

Comment From: antirez

Update: we have a reference implementation of diskless replication now.

Comment From: mattsta

Amazing! 10 days, 11 changed files, 763 additions and 114 deletions. Future amount of SSD wear saved: priceless.

Comment From: tarcieri

@antirez any word on native SSL support? Should I open a ticket on redis itself?

Comment From: mattsta

There are 30 issues listed above, so it'll get handled in time. :)

There are currently 3 to 5 different features "in process" plus clustering ("you shouldn't use Redis because it doesn't have clustering" <-- actual quote I saw posted recently) plus sentinel improvements ("redis loses all your data all the time!") plus everything else everybody wants done right now hurry up and get it done please for free no we aren't paying you but we want you to do more for free the feature is super important and our business will collapse without it but no we can't pay anything our billion dollar businesses are cash poor. :grimacing:

dramatization, but not by much. :purple_heart:

Comment From: tarcieri

@mattsta there's already a fork which integrates OpenSSL support for Redis 2.6:

https://github.com/bbroerman30/ssl-redis/tree/2.6 https://github.com/bbroerman30/ssl-redis/issues/2

I'm curious what the thoughts are on that.

If we had assurances this code were going to make it upstream, I think my company (Square) can finance a security audit.

Comment From: mattsta

Wow, that repository is really weird. It's not a clone of Redis, it's a copy/paste new repo of Redis with bulk updates. So, nothing can be merged or rebased between the two repositories using git (any direct merges would have to be by manual diff/patch).

So, stage one is probably use that as a template, but re-write everything from a Redis base. Bonus points for using non-openssl (http://www.libressl.org/ or alike) too. Extra bonus points for writing it in such a way as to have an encryption adapter layer so other implementations can be plugged in without touching the networking code again.

(and continual gripe: I'm still a fan of spiped because it's fast enough for most use cases and simpler than any CA-based setup. All advanced "we require live key rotation" needs tend to come from large organizations which could develop the features or fund priority development then contribute back?)

Comment From: antirez

@tarcieri the security audit is a great news. Probably given the fact it does not touch any sensitive stuff, we'll be able to backport it into 2.8, even if marking it "experimental" or alike for some time.

@mattsta it could be already a starting point, to proper support the whole thing (Sentinel, Cluster) will require more time so I would start with producing a diff of the more close version of standard Redis the patch is based on, and start addressing the client connections part. I hope we can do everything so that certificates can be updated while the server is running with CONFIG SET / CONFIG REWRITE.

Ramdon random thoughts: 1. Would be cool to have it running in a different port so that it is possible to support a mix of setups, like unencrypted connections + encrypted connections at the same time, but also exclude the unencrypted port if needed. 2. redis-cli should get the support as well of course, however it may not be too bad to have the ability to blacklist the loopback interface in order to be able to connect to an instance locally if shit happens and one is trapped outside the instance in a production environment. That would be optional, I guess. 3. Writing it in a layered thing should be easy. Maybe it's not needed to go to the extend of having it as a module, but at least to design it beforehand so that if in the future we'll need to support other libs, a full rewrite will not be needed. 4. A full implementation should also support replication, with selectable compression level. This is very useful in WAN setups.

Comment From: mattsta

Agree on everything. I haven't looked at the existing other repo, but openssl code can be tricky and just because the current code exists and works (for some value of "works") doesn't mean it's working correctly.

There are decisions to make about cipher suites, PFS/DHE + DHparams (optional? better perf without, better security with), then the actual integration and support for all individual parts of Redis including Sentinel, Cluster, redis-cli, hiredis, and blah blah blah. It's more involved than a + b = SSL because there are important decisions to make at every step along the way.

For reference, a good (but not pretty) implementation exists for nginx at https://github.com/nginx/nginx/blob/master/src/event/ngx_event_openssl.c (warning: it's over 3,000 lines just for openssl integration) — but, Redis doesn't need all of their complexity since we don't need backwards compatability across every SSL client ever created.

And, just because I'm curious, does anybody have performance numbers for redis+haproxy-ssl (configured as Redis->/tmp/redis.sock->haproxy-ssl) or redis+stud or redis+stunnel?

I'm going to cry the first time I have to restart Redis because of a newly discovered SSL vulnerability.

Comment From: tarcieri

Hi @mattsta

We run hundreds of stunnels across a half-dozen datacenters. We have some experience in this area.

spiped looks cool and I like Colin Percival, however due to various compliance requirements we need to use TLS. While Colin's encrypted transport looks OK it would complicate our security audits. Also we can leverage more advanced crypto via stunnel (ECC, GCM, etc). Native termination via OpenSSL or an OpenSSL-alike would offer us the same thing.

Key rotation is a big issue for us.

All advanced "we require live key rotation" needs tend to come from large organizations which could develop the features or fund priority development then contribute back?)

We're offering you a free security audit, from a company we trust, and from reqs insiders from the company-we'd-recommend would provide. Please don't look a gift horse in the mouth, this is the best thing you could possibly hope for in this regard. Various people here have been looking at implementing this feature de novo, but if we have a working base to start from, I'd strongly recommend that, auditing it, and trying to patch up the issues.

Bonus points for using non-openssl (http://www.libressl.org/ or alike) too.

While I like LibreSSL's idealism, BoringSSL is the better choice at this point. Reasons (mostly systematic input validation):

https://ruxconbreakpoint.com/assets/2014/slides/TLS%20Under%20Siege.pdf

That said both libraries expose a mostly-common API so even bringing up OpenSSL vs. LibreSSL vs. BoringSSL is a pointless diversion.

All that said, I think I'll open a new issue...

Comment From: mattsta

We run hundreds of stunnels

Any reason for stunnel instead of haproxy? Just a legacy issue?

We're offering you a free security audit

Yup. Just resource constraints. Free audits are always lovely, but remember Redis is extremely resource constrained as a project. It has one developer, two million users, and two dozen large improvement requests, each of which somebody thinks is top priority. There is no parallel development with Redis. If you get attention, everybody else doesn't get attention. For large companies, it seems they could try to parallelize some of the bring-up of new features if the features largely benefit them directly. Don't want it to look like people are queue hopping and trying to dominate the development todo list out of turn.

It comes down to weighted priorities. How many and what types of users want/need SSL vs stable cluster vs improved sentinel vs improved replication vs improved memory efficiency vs unified RDB+AOF writing vs a built-in trie vs built-in bloom filter vs (well, see everything in the first comment of this issue) ....

For SSL, I mostly see demand coming from companies selling hosted Redis services or from financial companies who need regulatory compliance. We're all happy and open source and free software forever, but are we an outsourced zero-cost corporate dev division to make big company life easier while temporarily ignoring features for the wider community?

oops—i don't actually have a point here other than: encryption workarounds exist for now, the in-process SSL development effort can take a while (it's definitely on the todo list—we're just arguing over priority at this point!), and since we already have workarounds for it, maybe other user-facing features should be prioritized for now (cluster, replication, aof+rdb, sentinel, more memory efficient lists, ...).

SSL would make a great project during the "internal refactoring" phase, but there's currently a lot of user-facing functionality to change too. Of course, there's a heavy chance I'm completely wrong and Salvatore can do a full SSL rewrite of the code in three days. :smile:

:m:

Comment From: tarcieri

Any reason for stunnel instead of haproxy? Just a legacy issue?

Apples and oranges. If we used haproxy (or twemproxy), it would be to talk to multiple Redis instances across multiple machines. However, we'd want encrypted flows. So we'd still need stunnels terminating TLS on the backend. Twemproxy or Haproxy could provide an encrypted frontend. We still need something to terminate TLS on the backend.

I would love to have haproxy or twemproxy providing encrypted access to multiple backends, however it would absolutely have to be encrypted access to meet our policies, and per our policies we would only support TLS as an encrypted transport protocol.

To further emphasize our case, here are some anecdotal quotes from our team when I announced that @antirez was even considering native TLS support for Redis. Please keep in mind that supporting the sorts of proxies and tunnels that you are advocating has provided us an ongoing support burden that cuts across teams and provides us all headaches, ongoing work, and pain which unless you are actively running hundreds of encrypted tunnels in production I really don't think you can possibly have a sense of. This is a huge pain point for us. I really wish you can solve it.

Anecdotes go. I swear these are all from different people at our organization:

"This is great." "YES DIE STUNNELS DIE" "Nice!" "DO THIS OH GOD PLEASE DO THIS" "Very nice!" "Yes yes yes."

Comment From: tarcieri

For SSL, I mostly see demand coming from companies selling hosted Redis services or from financial companies who need regulatory compliance.

Anyone who is interested in using Redis in any sort of security critical scenario should be interested in native SSL/TLS support. I don't think it's a premium feature for cloud hosting companies or financial companies. It's something anyone who wants to run a secure infrastructure should be interested in.

Comment From: antirez

Issue #2087 added, that documents a draft design for protected master restarts.

Comment From: antirez

@mattsta @tarcieri give me some days to look at the diff and form an idea about the ability to add SSL inside Redis without a huge refactoring or alike, and especially, in a self contained way with little interactions with the rest of the core. I used OpenSSL to create a server only once many years ago, I remember the API to be fairly higher level so that we may easily put it into the mix without too much problems but I may be wrong. The idea would be to have ssl.c with all the stuff to manage the connection, and also higher level wrappers to readable/writable events that handle the SSL specific reading/writing tasks, passing the control to the other functions in networking.c as soon as possible.

Comment From: tarcieri

@antirez awesome, thanks!

Comment From: yossigo

@antirez it will be great to see this in Redis! We added it to the service a while back and it's already supported built-in by some client libraries. OpenSSL should integrate well into networking.c, just pay attention to SSL read/writes vs. socket events which can get a bit confusing (in some cases in order to read from SSL, SSL needs to write to the socket and vice versa).

I think a supporting client certificate authentication is also a good idea, to serve as a secure alternative to the limited AUTH plain-text option.

Comment From: bbroerman30

@antirez The code I had was tightly integrated at the lowest levels... I had to essentially keep the bio object(s) in parallel to the raw sockets, and based on whether we were using ssl or not it would operate on the bio object or the socket. I'm not sure if it can easily be separated out without refactoring the socket and the read/write/select operations out into a class heirarchy. Honestly, at the time I did this, i didn't have the time to go to that level. The company I was working for wanted something quick that worked for all of our use cases. After they let me release it open source, I went and updated it with the 2.6 changes (up to the last 2.6 release), but haven't had a chance to really dig in and refactor.

If I get a Christmas vacation this year at my current company, maybe I can spend that time going through and clean it up and refactor. I also have some clean-up to do on my ssl changes to phpredis, and I was going to add ssl to jredis as well.

Comment From: bbroerman30

@mattsta Sorry about that (the repo being a copy with bulk changes)... I was new to GIT at the time, and since it was pretty obvious at the time that SSL was not wanted as part of the core for Redis, I didn't spend the time to go back and fix that... I did spend some time merging (by hand) the 2.6 (up to 2.6.17 i think) branch updates into mine, and got hiredis and redis-cli working properly. Sentinel compiles, but I haven't tested it. The work that was done was kind of a quick and dirty implementation, as I was trying to get the company i was working for at the time to buy into the idea of using it. Using stunnel was killing us operationally... It did pass QA, but it was never put into production before i left the company due to issues with PHP and phpredis (which i also fixed later on). I started looking at merging in the 2.7 branch into my code, but haven't had the time to really do anything with it yet. I'm hoping that I get some Christmas vacation time to look into it, and maybe do some refactoring, to clean thinks up.

Comment From: antirez

@yossigo @bbroerman30 thanks! Very interesting. We should move all this into an OpenSSL issue ASAP and link it from here.

Comment From: antirez

First issue in the list completed and merged into all the branches: diskless replication. Next target is protected restarts which is basically a must have now that people can drop replication in master-slaves setups in order to avoid incidents similar to the one at Stripe (https://stripe.com/blog/game-day-exercises-at-stripe).

Comment From: tarcieri

@antirez so how about that SSL issue? :wink: I tried to open one, but I guess it won't let me...

Comment From: darionyaphet

diskless replication ! seems interesting

Comment From: badboy

@darionyaphet Diskless replication is already implemented.

Comment From: antirez

Just added a new thing into this list: "Stream SYNC-less replication: slave connects and just get new updates from the master. Useful for caching with mostly idempotent writes".

Comment From: bbroerman30

@tarcieri If you need SSL right away, my repo has it for a couple versions of Redis. I'm working on refactoring, clean-up, and adding additional OpenSSL options (like cypher lists, etc) before I create a pull request for @antirez.

Comment From: tarcieri

We're "fine" with stunnels for now, but I've been tracking your progress and it's very exciting.

On Tuesday, February 3, 2015, Brad Broerman notifications@github.com wrote:

@tarcieri https://github.com/tarcieri If you need SSL right away, my repo has it for a couple versions of Redis. I'm working on refactoring, clean-up, and adding additional OpenSSL options (like cypher lists, etc) before I create a pull request for @antirez https://github.com/antirez.

— Reply to this email directly or view it on GitHub https://github.com/antirez/redis/issues/2045#issuecomment-72728166.

Tony Arcieri

Comment From: antirez

More items added, listing split in two sections, Redis and Sentinel.

Comment From: yossigo

That was a great dev meeting indeed, but most issues here have been resolved and the rest have their own dedicated issues - closing this now.