Hey,

First off, the HyperLogLog commands are a truly brilliant edition!

I was wondering what your thoughts were on the addition of a PFADDEX command.

Use case

We use Redis for real time analytic storage and querying. We have several large Redis clusters, with different data sharded across multiple servers and nodes.

The ZSET key name itself specifies which client owns the event, and "when" it happened. In each ZSET are unique identifiers and their respective score for that time period. One metric we track is essentially a randomly generated string, and can vary greatly in volume from user to user.

For lower volume users, a simple ZUNIONSTORE across the number of sets is fine, but some users have hundreds of thousands or millions of events in a day (and many billion events over weeks/months/years), requiring us to trim down the sets before performing the operation.

For very high volume users, we use HyperLogLog sets as the "unique count" metric, one corresponding to each data set (which is trimmed to provide a reasonable top N items).

What would be great, is the ability to PFADD only if the key already exists, such that trimming of sets that attained a certain threshold could be done dynamically and in real time (whenever the data was requested).

Once the ZSET was trimmed and our HyperLogLog set created the unique counting would be taken over by those sets, rather than just the cardinality of the ZUNIONSTORE output.

I understand that this can be done in a LUA script, which is probably how I will implement it first, but would you be open to such a feature? I'm happy to write it if you don't think it's the silliest thing you've ever heard. :smiley:

Cheers and thanks for Redis! Mike

Comment From: mattsta

Summary of request: PFADDEX - if key doesn't exist, return error instead of creating new HLL. :banana:

Comment From: michael-grunder

Sorry for verbosity, but I figured I would explain why such a thing could be useful.

TLDR: Read @mattsta's summary. :smiley:

Comment From: mattsta

Long explanations are great! It's always fun to see clever ways people use Redis. Just wanted to add a quick one liner for drive-by issue reading. :octocat:

Comment From: antirez

I'm not convinced by the rationale for adding such a command. TL;DR of my arguments: - Either this is a Lua scripting job. - Or... if we want to state, NX EX commands are a notable case so a script should not be required, then we need a command to abort a transaction, which is the IF command proposed in issue #1736.

So you could do:

MULTI
IF XX key
PFADD key ...
EXEC

Which does not require Lua, nor to pay the RTT penalty. However is this different than calling a Lua script performance-wise? There is to meter that. Morever it is possible to implement the IF command in Lua directly as expressed in the already mentioned issue #1736.

Comment From: michael-grunder

Hey,

In my experience you generally pay a bit of a penalty invoking LUA, which is why a pure command was preferable to me. This could be something of a bias given how I am using Redis (rather than a generally useful requirement).

Your alternative of aborting a transaction inside of a MULTI...EXEC block would work nicely as that can be pipelined client side.

Cheers! Mike