Redis [BUG] Bug in Redis-Benchmark

1. Firstly, Redis-benchmark will return 'NULL' in get test if there is an empty database, so all the benchmark begin with a get test cannot response with right result, because all the return are 'NULL'.
2. Secondly, the problem continue from the first one. The -d option of Redis-benchmark has no effect in get test due to we can only get a 'NULL' response(just several bytes).
3. Thirdly, but if your Redis-benchmark start with a set test or set some payloads in advance, especially with a big size payload, all the get test from that on will be influenced, no matter what -d option pointed out.

Comment From: glarer

Here I start with an empty database and request for get test with -d 16384, then I can get 174K RPS. But if I do a set like this, all the get test will be influencd like below.(-d 3 & -d 1024 & -d 16384)

Comment From: filipecosta90

@GlareR IMHO the responsibility for ensuring that the same conditions are met at the start of the benchmark are outside the scope of the benchmark tool and should be part of the responsibility of the person doing the benchmark. Meaning the tool is just a tool, and IMO you're describing that you get non-deterministic results due to the benchmark -- which is not true: If you provide the same start conditions to the benchmark, you'll always get the approximate same outcome. Bottom line, if you properly setup and teardown the benchmark, the above won't happen. - I suggest for any GET test that you ensure that you have the proper key(s) pre-populated. Likewise, tearing down the setup should also take care of cleaning the used benchmark keys. Marking this issue to be closed given IMHO see no change should be applied to redis-benchmark as a consequence of the described above.

Comment From: hnhyzz

@filipecosta90 The problem is that the -d argument does not make sense for -t GET at all. The usage indicates that "-d Data size of SET/GET value in bytes (default 3)" However, for -t GET, redis-benchmark does not set the data size. The data size is solely decided by the entry in the DB for the key. Maybe we could make clear in the usage that the -d argument only decides the data size of SET?

BTW, it is not intuitive for the benchmark user to know which key the GET benchmark would use. To test GET, I will simply bencmark SET for the same size first to add the entry, but I still do not know which key was set. Is it possible that redis-benchmark could firstly set an entry for the key to be accessed, when benchmarking GET? For users, it would be more friendly.