v1.27.0

[romange]

Dragonfly v1.27.0 - a Valentine Edition 💘

Important features and changes:

1. Fixes around JSON+SEARCH: fix(search_family): Support multiple fields in SORTBY option in the FT.AGGREGATE command. SECOND PR #4232 fix(search_family): Support boolean and nullable types in indexes #4314
2. Support for memcached meta protocol feat: add support for meta memcache commands #4362 feat: support deletions with meta protocol #4398 feat: support memcache meta responses #4366
3. Fixes in tiering code fix: properly clear tiered stashes upon expirations #4395 fix: ExternalAllocator::Free with large sizes #4388 fix: ExtentTree use of invalidated iterator #4416
4. Added LMPOP fix: LMPOP implementation was added #4504 and ZMPOP feat:Adding support for ZMPOP command #4385

What's Changed

• fix: test_network_disconnect_during_migration by @BorysTheDev in fix: test_network_disconnect_during_migration #4345
• refactor(rdb_saver): Add SnapshotDataConsumer to SliceSnapshot by @BagritsevichStepan in refactor(rdb_saver): Add SnapshotDataConsumer to SliceSnapshot #4287
• chore(rax_tree): Introduce raxFreeWithCallback call in RaxTreeMap destructor by @BagritsevichStepan in chore(rax_tree): Introduce raxFreeWithCallback call in RaxTreeMap destructor #4255
• refactor(search_family): Add Aggregator class by @BagritsevichStepan in refactor(search_family): Add Aggregator class #4290
• fix: avoid on stack allocation of lz4 compression context by @romange in fix: avoid on stack allocation of lz4 compression context #4322
• chore: update helio and improve our stack overflow resiliency by @romange in chore: update helio and improve our stack overflow resiliency #4349
• fix: configure jsonconst double parser to use std::from_chars by @romange in fix: configure jsonconst double parser to use std::from_chars #4360
• fix(search_family): Support boolean and nullable types in indexes by @BagritsevichStepan in fix(search_family): Support boolean and nullable types in indexes #4314
• fix(search_family): Support multiple fields in SORTBY option in the FT.AGGREGATE command. SECOND PR by @BagritsevichStepan in fix(search_family): Support multiple fields in SORTBY option in the FT.AGGREGATE command. SECOND PR #4232
• test: skip test_network_disconnect_during_migration by @BorysTheDev in test: skip test_network_disconnect_during_migration #4359
• fix(json_family): Remove std::endl during logging in ParseJsonPath method by @BagritsevichStepan in fix(json_family): Remove std::endl during logging in ParseJsonPath method #4363
• feat: add support for meta memcache commands by @romange in feat: add support for meta memcache commands #4362
• chore(test): disable failing test untill fixed by @adiholden in chore(test): disable failing test untill fixed #4367
• chore(deps): bump the actions group across 1 directory with 2 updates by @dependabot in chore(deps): bump the actions group across 1 directory with 2 updates #4361
• feat: support memcache meta responses by @romange in feat: support memcache meta responses #4366
• fix(pytest): fix failing test test_replication_timeout_on_full_sync by @adiholden in fix(pytest): fix failing test test_replication_timeout_on_full_sync #4370
• chore(json_family): Remove redundant JSON value copying by @BagritsevichStepan in chore(json_family): Remove redundant JSON value copying #4375
• feat(rdb_saver): Support big value serialization for stream by @BagritsevichStepan in feat(rdb_saver): Support big value serialization for stream #4376
• chore(snapshot): Small cleanup in Snapshot code by @BagritsevichStepan in chore(snapshot): Small cleanup in Snapshot code #4377
• fix: stream memory counting durin snapshot loading by @BorysTheDev in fix: stream memory counting durin snapshot loading #4346
• fix: return value for DflyMigrateAck by @BorysTheDev in fix: return value for DflyMigrateAck #4379
• chore(server): cleanup replication shard sync execution by @adiholden in chore(server): cleanup replication shard sync execution #4374
• test: fix test_network_disconnect_during_migration by @BorysTheDev in test: fix test_network_disconnect_during_migration #4378
• fix(snapshot_test): Fix test_big_value_serialization_memory_limit after adding streams support by @BagritsevichStepan in fix(snapshot_test): Fix test_big_value_serialization_memory_limit after adding streams support #4383
• fix: ExternalAllocator::Free with large sizes by @romange in fix: ExternalAllocator::Free with large sizes #4388
• fix(server): debug populate consume less memory by @adiholden in fix(server): debug populate consume less memory #4384
• refactor: slot_set don't use stack memory anymore by @BorysTheDev in refactor: slot_set don't use stack memory anymore #4386
• feat:Adding support for ZMPOP command by @guyzilla in feat:Adding support for ZMPOP command #4385
• fix(replication): do not log to journal on callback fail by @adiholden in fix(replication): do not log to journal on callback fail #4392
• chore: minor refactorings around dense_set deletions by @romange in chore: minor refactorings around dense_set deletions #4390
• feat(server): refactor allow preempt on journal record by @adiholden in feat(server): refactor allow preempt on journal record #4393
• fix: allow cluster node load snapshot bigger than maxmemory by @BorysTheDev in fix: allow cluster node load snapshot bigger than maxmemory #4394
• fix: properly clear tiered stashes upon expirations by @romange in fix: properly clear tiered stashes upon expirations #4395
• feat: support deletions with meta protocol by @romange in feat: support deletions with meta protocol #4398
• fix(tests): check cluster big snapshot in unit test by @adiholden in fix(tests): check cluster big snapshot in unit test #4403
• chore(pytest): add timeout per test by @adiholden in chore(pytest): add timeout per test #4404
• fix: provide resp3 option to CapturingReplyBuilder by @BorysTheDev in fix: provide resp3 option to CapturingReplyBuilder #4400
• chore: deprecate unneeded runtime flags by @romange in chore: deprecate unneeded runtime flags #4405
• feat: Yield inside huge values migration serialization by @chakaz in feat: Yield inside huge values migration serialization #4197
• chore(pytest): add check for rss grow in replicaiton big values by @adiholden in chore(pytest): add check for rss grow in replicaiton big values #4406
• feat: allow querying of json objects stored as strings by @romange in feat: allow querying of json objects stored as strings #4399
• test: move ReplicaofRejectOnLoad test from pytest into unit tests by @BorysTheDev in test: move ReplicaofRejectOnLoad test from pytest into unit tests #4410
• chore: Increase migration wait time in test_cluster_migration_huge_container by @chakaz in chore: Increase migration wait time in test_cluster_migration_huge_container #4409
• chore: Introduce small buffer in redis parser by @romange in chore: Introduce small buffer in redis parser #4076
• fix: invalid iterator case in Qlist::Erase by @romange in fix: invalid iterator case in Qlist::Erase #4414
• test: add test migration in cache mode by @BorysTheDev in test: add test migration in cache mode #4413
• fix: ExtentTree use of invalidated iterator by @kostasrim in fix: ExtentTree use of invalidated iterator #4416
• chore: fix build on FreeBSD by @chakaz in chore: fix build on FreeBSD #4412
• chore: reduce number of logs in cluster tests by @BorysTheDev in chore: reduce number of logs in cluster tests #4419
• fix: geosearch and georadius response format by @kostasrim in fix: geosearch and georadius response format #4420
• chore: split geo and zset families by @kostasrim in chore: split geo and zset families #4421
• fix: switch to SHUTTING_DOWN state unconditionally by @romange in fix: switch to SHUTTING_DOWN state unconditionally #4408
• chore: split zset and geo family tests by @kostasrim in chore: split zset and geo family tests #4425
• refactor: remove transaction lib on cluster code dependency by @BorysTheDev in refactor: remove transaction lib on cluster code dependency #4417
• chore: Increase migration wait time in test_cluster_migration_huge_container by @chakaz in chore: Increase migration wait time in test_cluster_migration_huge_container #4427
• chore: update jsoncons version to 0.178 by @romange in chore: update jsoncons version to 0.178 #4368
• fix: improve error propagation with RESTORE commands by @romange in fix: improve error propagation with RESTORE commands #4428
• fix: show managed stuff on privileged port by @romange in fix: show managed stuff on privileged port #4433
• chore: dragonfly connection refactorings by @romange in chore: dragonfly connection refactorings #4434
• fix: reply_builder should properly serialize bulk strings endings. by @romange in fix: reply_builder should properly serialize bulk strings endings. #4441
• chore: pull latest helio by @romange in chore: pull latest helio #4446
• chore: Remove template Callbacks by @BagritsevichStepan in chore: Remove template Callbacks #4431
• fix: resp inline parsing correctly resets itself by @romange in fix: resp inline parsing correctly resets itself #4458
• fix(ci): sanitizers build by @kostasrim in fix(ci): sanitizers build #4457
• chore: Add stats print for slot migrations by @chakaz in chore: Add stats print for slot migrations #4456
• fix: slot calculation during transaction squashing by @BorysTheDev in fix: slot calculation during transaction squashing #4460
• fix(sanitizers): Set stack size to 64 KB during tests for Sanitizers by @BagritsevichStepan in fix(sanitizers): Set stack size to 64 KB during tests for Sanitizers #4454
• Fix test hypothesis by @cunla in Fix test hypothesis #4380
• feat(dev): allow monitoring a valkey server on localhost by @romange in feat(dev): allow monitoring a valkey server on localhost #4467
• chore: add Dash::Prefetch function by @romange in chore: add Dash::Prefetch function #4476
• chore: dfly_bench now accepts data and score template names by @romange in chore: dfly_bench now accepts __data__ and __score__ template names #4477
• chore: replace deprecated HELIO_STACK_CHECK by @kostasrim in chore: replace deprecated HELIO_STACK_CHECK #4465
• chore: Fix all clang build warnings by @chakaz in chore: Fix all clang build warnings #4475
• fix: hiredis requires df to report version >7.4 by @kostasrim in fix: hiredis requires df to report version >7.4 #4474
• chore: make per-thread QueueBackpressure objects global by @romange in chore: make per-thread QueueBackpressure objects global #4482
• chore: OpManagerTest fixes by @romange in chore: OpManagerTest fixes #4480
• chore: run replication reg tests with epoll by @kostasrim in chore: run replication reg tests with epoll #4426
• chore: remove pubsub semantics from Connection::WeakRef by @romange in chore: remove pubsub semantics from Connection::WeakRef #4483
• Fix typo in outgoing_slot_migration.cc by @moredure in Fix typo in outgoing_slot_migration.cc #4484
• feat: allow finish and start migration in the same config by @BorysTheDev in feat: allow finish and start migration in the same config #4486
• feat: add support for noreply and great options by @romange in feat: add support for noreply and great options #4485
• feat(json_family): Add json_nesting_depth_limit flag by @BagritsevichStepan in feat(json_family): Add json_nesting_depth_limit flag #4444
• fix (stream): XRANGE incorrectly interprets the end parameter by @adiholden in fix (stream): XRANGE incorrectly interprets the end parameter #4443
• feat: implement connection timeout functionality by @romange in feat: implement connection timeout functionality #4407
• feat: add 'testing_time' limit option to dfly_bench by @romange in feat: add 'testing_time' limit option to dfly_bench #4487
• fix: test_acl_cat_commands_multi_exec_squash by @kostasrim in fix: test_acl_cat_commands_multi_exec_squash #4492
• chore: add sanitizers to ci workflow by @kostasrim in chore: add sanitizers to ci workflow #4462
• chore: improve test_timeout robustness by @romange in chore: improve test_timeout robustness #4494
• fix(cluster): crash in cluster migration by @adiholden in fix(cluster): crash in cluster migration #4495
• skip test_migration_one_after_another on epoll by @adiholden in skip test_migration_one_after_another on epoll #4499
• fix: several bugs by @romange in fix: several bugs #4500
• chore: new docker build pipeline by @romange in chore: new docker build pipeline #4503
• fix: rpm package setup by @romange in fix: rpm package setup #4506
• fix: Do not bump elements during RDB load by @chakaz in fix: Do not bump elements during RDB load #4507
• fix cluster: migration crash fix by @adiholden in fix cluster: migration crash fix #4508
• chore: split epoll/uring reg test flows by @kostasrim in chore: split epoll/uring reg test flows #4512
• fix(test): fix rdb random fail on OOM by @adiholden in fix(test): fix rdb random fail on OOM #4509
• chore: add defrag for StringSet by @kostasrim in chore: add defrag for StringSet #4308
• chore: add benchmarking of ScanOpts::Matches by @romange in chore: add benchmarking of ScanOpts::Matches #4511
• fix: crash for delconsumer during stream reading by @BorysTheDev in fix: crash for delconsumer during stream reading #4513
• chore(deps): bump github/codeql-action from 3.28.0 to 3.28.5 in the actions group across 1 directory by @dependabot in chore(deps): bump github/codeql-action from 3.28.0 to 3.28.5 in the actions group across 1 directory #4514
• chore: remove unused code by @romange in chore: remove unused code #4516
• fix: LMPOP implementation was added by @vyavdoshenko in fix: LMPOP implementation was added #4504
• chore: add reflex matcher to the benchmarks by @romange in chore: add reflex matcher to the benchmarks #4520
• chore: introduce GlobMatcher by @romange in chore: introduce GlobMatcher #4521
• chore: Add Lua force atomicity flag by @chakaz in chore: Add Lua force atomicity flag #4523
• chore: add more logs to the op_manager code by @romange in chore: add more logs to the op_manager code #4527
• fix(search_family): Add options test for the FT.SEARCH command by @BagritsevichStepan in fix(search_family): Add options test for the FT.SEARCH command #4478
• chore(ci): report proactor kind on reg test failure by @kostasrim in chore(ci): report proactor kind on reg test failure #4531
• chore: update release build for arm by @BorysTheDev in chore: update release build for arm #4519
• chore: refactoring around glob matching by @romange in chore: refactoring around glob matching #4540
• chore: clean up of deprecated flags by @romange in chore: clean up of deprecated flags #4545
• chore: move QList::Node definition into dragonfly codebase by @romange in chore: move QList::Node definition into dragonfly codebase #4547
• chore: Add benchmarks of qlist compression paths by @romange in chore: Add benchmarks of qlist compression paths #4543
• chore: remove clang install step from ci workflow by @kostasrim in chore: remove clang install step from ci workflow #4550
• fix: test_snapshoting_during_migration by @BorysTheDev in fix: test_snapshoting_during_migration #4555
• fix: Bug GEQ range query bug by @romange in fix: Bug GEQ range query bug #4557
• fix: consume parser error on dfly load by @kostasrim in fix: consume parser error on dfly load #4556
• chore: GlobMatcher uses now reflex::Matcher regex engine by @romange in chore: GlobMatcher uses now reflex::Matcher regex engine #4528
• chore: Add FakeRedis capture/compare to more cluster tests and seed during migration by @chakaz in chore: Add FakeRedis capture/compare to more cluster tests and seed during migration #4542
• Fix BullMQ tests workflow in GitHub actions by @chakaz in Fix BullMQ tests workflow in GitHub actions #4565
• fix(stream_family): Fix journaling in the XADD and XTRIM commands by @BagritsevichStepan in fix(stream_family): Fix journaling in the XADD and XTRIM commands #4448
• refactor: rename Context into ExecutionState by @BorysTheDev in refactor: rename Context into ExecutionState #4562
• chore: pull helio epoll fix and tests by @kostasrim in chore: pull helio epoll fix and tests #4559
• fix server: fix deadlock blmove does not conclude on error by @adiholden in fix server: fix deadlock blmove does not conclude on error #4561
• feat: introduce more options for traffic logger by @romange in feat: introduce more options for traffic logger #4571
• fix: a regex bug with escaping inside of character classes by @romange in fix: a regex bug with escaping inside of character classes #4575
• docs: the broken link was fixed by @vyavdoshenko in docs: the broken link was fixed #4579
• fix: a rare deadlock with blmove and multi-db by @romange in fix: a rare deadlock with blmove and multi-db #4568
• server: enable big value serialization by default by @adiholden in server: enable big value serialization by default #4581
• chore: minor simplification around WaitWatch by @romange in chore: minor simplification around WaitWatch #4580
• fix: ascii logo was added by @vyavdoshenko in fix: ascii logo was added #4524
• fix: migration finalization process by @BorysTheDev in fix: migration finalization process #4576
• fix: incorrect passing of db_index in EVAL transactions by @romange in fix: incorrect passing of db_index in EVAL transactions #4586
• test: add ClusterEvalCrossslot by @BorysTheDev in test: add ClusterEvalCrossslot #4587
• fix: huge TTL in RESTORE cmd is rounded to kMaxExpireDeadlineMs by @BorysTheDev in fix: huge TTL in RESTORE cmd is rounded to kMaxExpireDeadlineMs #4589

Huge thanks to all the contributors! ❤️

New Contributors

• @guyzilla made their first contribution in feat:Adding support for ZMPOP command #4385
• @moredure made their first contribution in Fix typo in outgoing_slot_migration.cc #4484
• @vyavdoshenko made their first contribution in fix: LMPOP implementation was added #4504

Full Changelog: v1.26.0...v1.27.0

---

This discussion was created from the release v1.27.0.

[romange]