The problem/use-case that the feature addresses

I'm using the WD SN-850 NVMe SSD, and I find the write performance (YCSB-A) is not as good as I expected. Even sometimes worse than my Samsung 860 EVO SATA SSD. Exploring the root cause, I found that WD SN-850 SSD is very sensitive to the size of written data. As shown in the following figure, when the size of write (random write) is small or not multiple of 32KB, the performance of WD SN-850 is even slower than Samsung 860 EVO. I know that till now it is only related to the devices.
Redis [NEW] control the bytes of data being written or support alternatives for fsync
Then I look into the write issued by redis-server: firstly with bcc/tools/trace and find:

420.3227 0   54133   54133   redis-server    ksys_write       fd=12 size=5
420.3230 0   54133   54133   redis-server    ksys_write       fd=13 size=5
420.3232 0   54133   54133   redis-server    ksys_write       fd=10 size=5
420.3235 0   54133   54133   redis-server    ksys_write       fd=18 size=963
420.3237 0   54133   54133   redis-server    do_fsync         fd=18
420.3242 0   54133   54133   redis-server    ksys_write       fd=12 size=5
420.3244 0   54133   54133   redis-server    ksys_write       fd=13 size=5
420.3247 0   54133   54133   redis-server    ksys_write       fd=8 size=5
420.3255 0   54133   54133   redis-server    ksys_write       fd=18 size=963
420.3258 0   54133   54133   redis-server    do_fsync         fd=18

where fd=18 is the AOF file. It seems that Redis just write the buffer with arbitrary size of data (depending on the SET value size being written). 

nwritten = aofWrite(server.aof_fd,server.aof_buf,sdslen(server.aof_buf));   // aof.c:1091

After that, the file system may merge and write the corresponding page in the page cache (observe with bcc/tools/biosnoop):

401.654886  redis-server   54133  nvme0n1 W 83886336   4096      0.16
401.663039  redis-server   54133  nvme0n1 W 84157280   8192      0.27
401.663323  redis-server   54133  nvme0n1 W 83886336   4096      0.19
401.714591  redis-server   54133  nvme0n1 W 84157296   4096      0.22
401.714778  redis-server   54133  nvme0n1 W 83886344   4096      0.15
401.715035  redis-server   54133  nvme0n1 W 83951880   4096      0.16
401.716304  redis-server   54133  nvme0n1 W 84157304   4096      0.18
401.716720  redis-server   54133  nvme0n1 W 83886336   4096      0.38
401.716977  redis-server   54133  nvme0n1 W 83951880   4096      0.22

Note that here will result in many small write requests (i.e., 4KB < 32KB), which seems not very friendly with my NVMe SSD. Currently, the AOF buffer is simple

char * aof_buf; // server.h:1665

so controlling the bytes (and alignment) being written precisely (coarsely with auto-aof-rewrite-min-size) is not simple.

Description of the feature

So could we organize the AOF buffer as pages as classic rational databases do? In this way, writes in one page can be merged or covered (SET k 1; SET k 2;).

Alternatives you've considered

If implementing page-based buffer is too heavy, I have also found a way that magically works around my problem. As documented here:

in some Linux configurations Redis may block too long on the fsync() call. Note that there is no fix for this currently, as even performing fsync in a different thread will block our synchronous write(2) call.

Only fsync can be chosen to provide reliability. While I find if I use open with O_SYNC flag, the problem in my device (WD SN-850) gets mitigated. So shall we support more options (O_DSYNC, fdatasync, etc.) as alternative solutions for fsync, and make it configurable?

Additional information

  • Linux kernel version 5.4.0
  • Redis version 7.0.0
  • Configured with aof mode

Thanks & best!

Comment From: oranagra

Buffering the aof writes and doing them only when enough data is accumulated, doesn't seem like a good option. We do want to persist the changes as soon as their committed (note that we don't write them after every command, but as a bunch after processing all the command of a single event loop, before going to sleep). We can of course split big writes to smaller ones, but it seems like some tuning that's better done at the OS / mount level.

Regarding fsync, are you using fsync every write or every second?

You're saying that using O_SYNC is better than write followed by fsync?

I don't think fdatasync / O_DSYNC is appropriate for AOF since it doesn't provide any guarantees in the event of catastrophe.

Comment From: TimHe95

I am using appendfsync=always. I agree that fdatasync/O_DSYNC can not ensure integrity. While using O_SYNC can magically make my performance problem disappear. The boost moving from write + fsync to O_SYNC + write is huge (benchmarked via fio )

NVMe SSD
+-------------+--------------+
| write fsync | O_SYNC write |
+-------------+--------------+
|   1.6kIOPS  |   23.3kIOPS  |
+-------------+--------------+

The possible reason may lie in synchronization as discussed widely (e.g., [1] [2]). So I think adding O_SYNC as an alternative option may workaround my problem in redis.

I agree that persisting the changes as soon as their commit can ensure the highest level of safety. So buffering may lie in the middle between appendfsync=always and appendfsync=everysec. I can also understand Redis is mainly used as a caching server rather than persistent storage. But since sometimes we want to use it as persistent storage, I'm very hopeful that I can find a good balance among device properties, performance, and safety. So I request for alternative solutions for persisting AOF files. So I really hope that the advice would be taken into consideration in some future version.

Comment From: yossigo

@TimHe95 I'm not against supporting O_SYNC (and maybe O_DIRECT) in the future. I understand how it can perform better than fsync() in specific conditions (fs intensively used by many other processes, etc.), but in most cases I think the difference should be minimal. O_SYNC does offer other advantages, like more accurate error handling.

Also, I don't see how it can result with different sizes of blocks being written so I doubt that's the root cause for any speedup. Can you provide the full details of the fio benchmarks you did, and maybe also take a look at the biosnoop output while running the different benchmarks?

Ultimately, we should run this benchmark comparing Redis with a patched version that does O_SYNC and no fsync(), rather than rely on a synthetic benchmark.

Comment From: TimHe95

  • fio benchmarks commands

bash fio --filename=/dev/nvme2n1 --size=50g --ioengine=[sync/libaio] --iodepth=[1/32] --numjobs=16 --rw=randwrite --buffered=0 --direct=1 --fsync=[1/0] --bs=[4k/128k] --sync=[none/sync]
- Device: Samsung 980 Pro (a little different from WD SN850 shown in my first post) - Results

1

  • biosnoop output
  • --bs=4k --fsync=1 --ioengine=sync --iodepth=1

TIME(s) COMM PID DISK T SECTOR BYTES LAT(ms) 6.983948 fio 10291 nvme2n1 W 353177752 4096 3.14 6.989027 jbd2/nvme2n1p4 477 nvme2n1 W 416855768 4096 5.81 6.989031 fio 10282 nvme2n1 W 456247032 4096 5.05 6.994052 jbd2/nvme2n1p4 477 nvme2n1 W 416855776 8192 4.99 6.999017 fio 10289 nvme2n1 W 382450648 4096 4.95 7.001455 ? 0 R 0 0 7.39 7.002256 fio 10292 nvme2n1 W 438637416 4096 3.21 7.007327 jbd2/nvme2n1p4 477 nvme2n1 W 416855792 4096 5.86 7.007333 fio 10288 nvme2n1 W 446884744 4096 5.05 7.012347 jbd2/nvme2n1p4 477 nvme2n1 W 416855800 8192 4.99 7.017369 fio 10297 nvme2n1 W 369945064 4096 5.01 7.019815 ? 0 R 0 0 7.46 7.020550 fio 10284 nvme2n1 W 409096632 4096 3.15 7.029362 jbd2/nvme2n1p4 477 nvme2n1 W 416855816 4096 9.53 7.029365 fio 10295 nvme2n1 W 458145128 4096 8.79 7.034440 fio 10287 nvme2n1 W 437047208 4096 5.05 7.034440 jbd2/nvme2n1p4 477 nvme2n1 W 416855824 8192 5.04 7.039415 fio 10294 nvme2n1 W 365170256 4096 4.94 7.041797 ? 0 R 0 0 7.34 7.042534 fio 10285 nvme2n1 W 403628520 4096 3.09 7.047527 jbd2/nvme2n1p4 477 nvme2n1 W 416855840 4096 5.72 7.047532 fio 10293 nvme2n1 W 413944080 4096 4.97 7.052536 jbd2/nvme2n1p4 477 nvme2n1 W 416855848 8192 4.98 7.054906 ? 0 R 0 0 7.35 7.055754 fio 10286 nvme2n1 W 445670032 4096 3.20 7.058176 ? 0 R 0 0 5.63 7.058979 fio 10290 nvme2n1 W 439342784 4096 3.19 7.064041 jbd2/nvme2n1p4 477 nvme2n1 W 416855864 4096 5.85 7.064046 fio 10296 nvme2n1 W 379742680 4096 5.04 7.069048 jbd2/nvme2n1p4 477 nvme2n1 W 416855872 8192 4.98 7.073996 fio 10285 nvme2n1 W 359729792 4096 4.93 7.076344 ? 0 R 0 0 7.28 7.077143 fio 10291 nvme2n1 W 371313512 4096 3.12 7.082149 jbd2/nvme2n1p4 477 nvme2n1 W 416855888 4096 5.80 7.082151 fio 10282 nvme2n1 W 457379960 4096 4.98 7.087104 jbd2/nvme2n1p4 477 nvme2n1 W 416855896 8192 4.92 7.092039 fio 10283 nvme2n1 W 414461016 4096 4.92 7.094467 ? 0 R 0 0 7.35 7.095261 fio 10289 nvme2n1 W 430365400 4096 3.19 7.100200 jbd2/nvme2n1p4 477 nvme2n1 W 416855912 4096 5.73 7.100204 fio 10288 nvme2n1 W 397311472 4096 4.92 7.105249 jbd2/nvme2n1p4 477 nvme2n1 W 416855920 8192 5.03 7.110246 fio 10292 nvme2n1 W 377664984 4096 4.98 7.112668 ? 0 R 0 0 7.41 7.113468 fio 10284 nvme2n1 W 360699208 4096 3.19 7.118404 jbd2/nvme2n1p4 477 nvme2n1 W 416855936 4096 5.73 7.118407 fio 10295 nvme2n1 W 389111888 4096 4.91 7.123352 jbd2/nvme2n1p4 477 nvme2n1 W 416855944 8192 4.91 7.128338 fio 10297 nvme2n1 W 397316280 4096 4.97 7.130766 ? 0 R 0 0 7.39 7.131564 fio 10287 nvme2n1 W 382146376 4096 3.19 7.136623 jbd2/nvme2n1p4 477 nvme2n1 W 416855960 4096 5.85 7.136629 fio 10294 nvme2n1 W 386625696 4096 5.03 7.141624 jbd2/nvme2n1p4 477 nvme2n1 W 416855968 8192 4.98 7.146600 fio 10293 nvme2n1 W 415321488 4096 4.96 7.149008 ? 0 R 0 0 7.37 7.149861 fio 10296 nvme2n1 W 348771008 4096 3.23 7.154940 jbd2/nvme2n1p4 477 nvme2n1 W 416855984 4096 5.93 7.154945 fio 10290 nvme2n1 W 386815800 4096 5.05 7.159948 jbd2/nvme2n1p4 477 nvme2n1 W 416855992 8192 4.97 7.164895 fio 10286 nvme2n1 W 404918872 4096 4.93 7.167277 ? 0 R 0 0 7.32 ...... no big difference below

  • --bs=4k --fsync=1 --ioengine=libaio --iodepth=32

TIME(s) COMM PID DISK T SECTOR BYTES LAT(ms) 7.309330 fio 10790 nvme2n1 W 374499464 4096 8.21 7.309330 fio 10782 nvme2n1 W 369786744 4096 8.19 7.309320 fio 10786 nvme2n1 W 403675976 4096 8.17 7.309329 fio 10788 nvme2n1 W 445437504 4096 8.35 7.309329 fio 10791 nvme2n1 W 359585376 4096 8.32 7.309330 fio 10779 nvme2n1 W 457410192 4096 8.28 7.309335 fio 10780 nvme2n1 W 400166576 4096 8.27 7.309332 fio 10792 nvme2n1 W 359002192 4096 8.30 7.309324 fio 10787 nvme2n1 W 371933600 4096 8.33 7.309336 fio 10784 nvme2n1 W 382917104 4096 8.37 7.309338 fio 10789 nvme2n1 W 415109456 4096 8.25 7.309334 fio 10783 nvme2n1 W 376900920 4096 8.16 7.313590 jbd2/nvme2n1p4 477 nvme2n1 W 416873072 4096 5.07 7.318658 jbd2/nvme2n1p4 477 nvme2n1 W 416873080 8192 5.02 7.323801 fio 10793 nvme2n1 W 387496576 4096 5.11 7.323801 fio 10785 nvme2n1 W 426565592 4096 5.09 7.323798 fio 10781 nvme2n1 W 426384912 4096 5.13 7.324135 ? 0 R 0 0 5.46 7.329186 jbd2/nvme2n1p4 477 nvme2n1 W 416873096 4096 5.04 7.334280 jbd2/nvme2n1p4 477 nvme2n1 W 416873104 8192 5.05 7.339453 fio 10789 nvme2n1 W 357026296 4096 5.13 7.339451 fio 10794 nvme2n1 W 434562104 4096 5.16 7.339453 fio 10787 nvme2n1 W 432931360 4096 5.11 7.341980 ? 0 R 0 0 7.67 7.342808 fio 10786 nvme2n1 W 453591360 4096 8.44 7.342810 fio 10791 nvme2n1 W 364942840 4096 8.43 7.342806 fio 10779 nvme2n1 W 354733632 4096 8.45 7.342814 fio 10792 nvme2n1 W 364838904 4096 8.40 7.342831 fio 10784 nvme2n1 W 355350560 4096 8.36 7.342816 fio 10783 nvme2n1 W 380602288 4096 8.27 7.342835 fio 10790 nvme2n1 W 352405056 4096 8.32 7.342835 fio 10782 nvme2n1 W 364744376 4096 8.31 7.342816 fio 10788 nvme2n1 W 373409688 4096 8.25 7.342818 fio 10780 nvme2n1 W 360460296 4096 8.38 7.347201 jbd2/nvme2n1p4 477 nvme2n1 W 416873120 4096 5.21 7.352300 jbd2/nvme2n1p4 477 nvme2n1 W 416873128 8192 5.06 7.357274 fio 10793 nvme2n1 W 455786000 4096 4.93 7.357272 fio 10781 nvme2n1 W 433888416 4096 4.96 7.359790 ? 0 R 0 0 7.48 7.360585 fio 10785 nvme2n1 W 368767664 4096 8.22 7.364853 jbd2/nvme2n1p4 477 nvme2n1 W 416873144 4096 5.06 7.369949 jbd2/nvme2n1p4 477 nvme2n1 W 416873152 8192 5.06 7.375093 fio 10786 nvme2n1 W 370051016 4096 5.13 7.375096 fio 10792 nvme2n1 W 370297672 4096 5.10 7.375103 fio 10787 nvme2n1 W 356440680 4096 5.12 7.377604 ? 0 R 0 0 7.64 7.378411 fio 10783 nvme2n1 W 434470736 4096 8.35 7.378415 fio 10791 nvme2n1 W 368057552 4096 8.28 7.378411 fio 10788 nvme2n1 W 433177536 4096 8.38 7.378404 fio 10780 nvme2n1 W 434230848 4096 8.22 7.378418 fio 10782 nvme2n1 W 398481400 4096 8.27 7.378409 fio 10794 nvme2n1 W 439839720 4096 8.36 7.378421 fio 10784 nvme2n1 W 429783576 4096 8.20 7.378416 fio 10789 nvme2n1 W 371904344 4096 8.30 7.378427 fio 10790 nvme2n1 W 427768000 4096 8.33 7.378420 fio 10779 nvme2n1 W 439586392 4096 8.26 7.382668 jbd2/nvme2n1p4 477 nvme2n1 W 416873168 4096 5.05 7.387755 jbd2/nvme2n1p4 477 nvme2n1 W 416873176 8192 5.04 7.392873 fio 10793 nvme2n1 W 366075424 4096 5.10 7.392873 fio 10785 nvme2n1 W 399119832 4096 5.09 7.395390 ? 0 R 0 0 7.62 7.396201 fio 10781 nvme2n1 W 425836664 4096 8.38 7.400454 jbd2/nvme2n1p4 477 nvme2n1 W 416873192 4096 5.05 7.405546 jbd2/nvme2n1p4 477 nvme2n1 W 416873200 8192 5.04 7.410650 fio 10792 nvme2n1 W 396238656 4096 5.09 7.410652 fio 10787 nvme2n1 W 425463848 4096 5.07 7.413173 ? 0 R 0 0 7.61 7.413919 fio 10784 nvme2n1 W 373870032 4096 8.16 7.413920 fio 10789 nvme2n1 W 359569024 4096 8.24 7.413914 fio 10783 nvme2n1 W 373773808 4096 8.26 7.413909 fio 10786 nvme2n1 W 357737184 4096 8.29 7.413916 fio 10791 nvme2n1 W 381150360 4096 8.25 7.413920 fio 10779 nvme2n1 W 357504840 4096 8.22 ...... no big difference below

  • --bs=128k --fsync=1 --ioengine=libaio --iodepth=32

TIME(s) COMM PID DISK T SECTOR BYTES LAT(ms) 7.398827 fio 11117 nvme2n1 W 439124056 4096 6.58 7.398830 fio 11121 nvme2n1 W 373613408 4096 6.57 7.398823 fio 11119 nvme2n1 W 379329912 4096 6.59 7.398824 fio 11109 nvme2n1 W 420150664 4096 6.58 7.401566 ? 0 R 0 0 9.74 7.402301 fio 11111 nvme2n1 W 442981544 4096 10.03 7.402301 fio 11107 nvme2n1 W 377788496 4096 10.01 7.402300 fio 11108 nvme2n1 W 349197392 4096 10.02 7.402304 fio 11116 nvme2n1 W 360095080 4096 10.01 7.402301 fio 11114 nvme2n1 W 435595264 4096 10.03 7.404880 ? 0 R 0 0 13.05 7.405610 fio 11113 nvme2n1 W 404175704 4096 13.31 7.405615 fio 11115 nvme2n1 W 431294592 4096 13.30 7.405613 fio 11122 nvme2n1 W 431346928 4096 13.31 7.405622 fio 11110 nvme2n1 W 435752816 4096 13.29 7.405615 fio 11119 nvme2n1 W 341507776 4096 13.30 7.408257 ? 0 R 0 0 16.43 7.409044 fio 11117 nvme2n1 W 447283192 4096 16.68 7.409045 fio 11121 nvme2n1 W 416285840 4096 16.68 7.409039 fio 11109 nvme2n1 W 456646632 4096 16.68 7.409042 fio 11118 nvme2n1 W 390444152 4096 16.70 7.411630 ? 0 R 0 0 19.80 7.412301 fio 11111 nvme2n1 W 399623312 4096 19.92 7.412306 fio 11107 nvme2n1 W 428312656 4096 19.91 7.412304 fio 11108 nvme2n1 W 429840536 4096 19.91 7.412306 fio 11116 nvme2n1 W 367495600 4096 19.90 7.412298 fio 11106 nvme2n1 W 439576208 4096 19.92 7.412303 fio 11114 nvme2n1 W 440410104 4096 19.92 7.414895 ? 0 R 0 0 23.03 7.415619 fio 11119 nvme2n1 W 414765000 4096 23.20 7.415617 fio 11113 nvme2n1 W 393851488 4096 23.21 7.415623 fio 11109 nvme2n1 W 445463896 4096 23.18 7.415623 fio 11118 nvme2n1 W 447224928 4096 23.19 7.415617 fio 11122 nvme2n1 W 350260888 4096 23.20 7.415625 fio 11110 nvme2n1 W 432564080 4096 23.20 7.418341 ? 0 R 0 0 26.48 7.419133 fio 11111 nvme2n1 W 375273112 4096 26.68 7.419129 fio 11117 nvme2n1 W 452462600 4096 26.68 7.419130 fio 11121 nvme2n1 W 412156552 4096 26.68 7.419136 fio 11107 nvme2n1 W 415248712 4096 26.66 7.419135 fio 11108 nvme2n1 W 444485640 4096 26.67 7.419134 fio 11114 nvme2n1 W 408588896 4096 26.67 7.421849 ? 0 R 0 0 29.99 7.422636 fio 11119 nvme2n1 W 404934712 4096 30.14 7.422632 fio 11113 nvme2n1 W 343318280 4096 30.15 7.422632 fio 11116 nvme2n1 W 371521224 4096 30.15 7.422640 fio 11118 nvme2n1 W 389212872 4096 30.13 7.422634 fio 11122 nvme2n1 W 382614408 4096 30.14 7.422641 fio 11110 nvme2n1 W 379936424 4096 30.14 7.425223 ? 0 R 0 0 33.36 7.425896 fio 11111 nvme2n1 W 455174560 4096 33.36 7.425894 fio 11117 nvme2n1 W 401707528 4096 33.37 7.425893 fio 11121 nvme2n1 W 426255640 4096 33.36 7.425890 fio 11109 nvme2n1 W 365486784 4096 33.37 7.425898 fio 11114 nvme2n1 W 377539544 4096 33.36 7.425900 fio 11108 nvme2n1 W 382761808 4096 33.35 7.428480 ? 0 R 0 0 36.61 7.429218 fio 11110 nvme2n1 W 406829528 4096 36.63 7.429214 fio 11119 nvme2n1 W 393946664 4096 36.64 7.429207 fio 11107 nvme2n1 W 415408032 4096 36.65 7.429210 fio 11113 nvme2n1 W 385973176 4096 36.64 7.429210 fio 11116 nvme2n1 W 422185208 4096 36.65 7.429211 fio 11122 nvme2n1 W 378197592 4096 36.64 7.431799 ? 0 R 0 0 39.93 7.432525 fio 11111 nvme2n1 W 367365664 4096 39.91 7.432525 fio 11117 nvme2n1 W 444802872 4096 39.92 7.432526 fio 11121 nvme2n1 W 437397920 4096 39.92 7.432523 fio 11109 nvme2n1 W 430033784 4096 39.92 7.432522 fio 11118 nvme2n1 W 456897664 4096 39.93 7.435114 ? 0 R 0 0 43.24 7.435842 fio 11107 nvme2n1 W 367326304 4096 43.21 7.435839 fio 11108 nvme2n1 W 420955128 4096 43.21 7.435846 fio 11113 nvme2n1 W 447898968 4096 43.20 7.435846 fio 11116 nvme2n1 W 386840000 4096 43.21 7.435838 fio 11114 nvme2n1 W 369948512 4096 43.22 7.435848 fio 11122 nvme2n1 W 350021472 4096 43.20 7.438435 ? 0 R 0 0 46.53 7.439102 fio 11110 nvme2n1 W 347904664 4096 46.44 7.439109 fio 11111 nvme2n1 W 348698608 4096 46.41 7.439108 fio 11117 nvme2n1 W 415497048 4096 46.42 7.439109 fio 11121 nvme2n1 W 402081448 4096 46.42

Two conclusions

  1. O_SYNC benefits async I/O a lot.
  2. Device is sensitive to I/O size, in Samsung 980 Pro, the IOPS are the same over different I/O sizes while the bandwidth increases as the size increases. In WD SN850, both IOPS and bandwidth have "crest" and "trough" as the I/O size increases.

Comment From: TimHe95

Device cache is always disabled:

> sudo nvme set-feature -f 6 -v 0 /dev/nvme2n1
set-feature:06 (Volatile Write Cache), value:00000000

Comment From: TimHe95

With blktrace, it seems that the slowness is not in OS when --bs=4k --fsync=1 --ioengine=libaio --iodepth=32:

==================== Device Overhead ====================

       DEV |       Q2G       G2I       Q2M       I2D       D2C <------ time the I/O is “active” in the driver and on the device
---------- | --------- --------- --------- --------- ---------
 (259, 12) |   0.0158%   0.0000%   0.0007%   0.0000%  92.6055%
---------- | --------- --------- --------- --------- ---------
   Overall |   0.0158%   0.0000%   0.0007%   0.0000%  92.6055%

Comment From: uvletter

fio --filename=/dev/nvme2n1 --size=50g --ioengine=[sync/libaio] --iodepth=[1/32] --numjobs=16 --rw=randwrite --buffered=0 --direct=1 --fsync=[1/0] --bs=[4k/128k] --sync=[none/sync]

I think the configuration may not exactly match the workload of AOF, several points:

  • AOF is written by a single thread, so maybe numjobs should be 1
  • AOF is append only, --rw=write is more appropriate
  • AOF use the buffered IO now, so maybe direct should be 0

Besides, ioengine of libaio and sync cannot compare directly, libaio can occupy the queue depth, in contrast sync only leverage 1, one wait the former one. I think you could design test case for the workloads redis works now, as well as a way redis could promote(e.g. libaio O_SYNC).

Note that here will result in many small write requests (i.e., 4KB < 32KB), which seems not very friendly with my NVMe SSD.

Normally the block size is 4KB for SSD, the generic block layer would plug and merge the bio request and emit to the block driver in the unit of block size. A larger block size setting in fio usually means a better performance, but in reality applications maybe not have so many huge write, it just for testing the capacity of block device.