Skip to content

feat: add namespace and epoch for enhancement. #2581

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

AminMortezaie
Copy link

Add Namespace and Epoch Support to KV Store

Overview

This PR introduces namespace and epoch support to AutoMQ's KV store system through a composable design pattern, allowing for backward compatibility while providing new functionality for applications that need namespace isolation and epoch-based versioning.

Key Changes

  1. Added a new NamespacedControllerKVClient that extends the base ControllerKVClient
  2. Introduced a feature flag namespacedKVEnabled in Config for opt-in functionality
  3. Modified DefaultS3Client to support both namespaced and non-namespaced modes

Implementation Details

  • Namespace Support: Keys are prefixed with their namespace (e.g., "namespace:key") for isolation
  • Epoch Management: Each namespace maintains its own epoch counter for versioning
  • Backward Compatibility:
    • Default behavior remains unchanged (namespacedKVEnabled = false)
    • Existing applications continue to work without modification
    • New applications can opt-in to namespaced functionality

Usage

To enable namespaced KV operations:

Config config = new Config()
    .namespacedKVEnabled(true);

When enabled, applications can:

  • Use namespace isolation for keys
  • Maintain epoch versioning per namespace
  • Prevent cross-namespace conflicts
  • Maintain data isolation between different parts of the system

Migration Path

  1. Deploy the changes with namespacedKVEnabled = false (default)
  2. Applications can gradually opt-in to namespaced functionality
  3. No immediate action required for existing applications

Performance Impact

  • Minimal overhead for non-namespaced operations (no changes to existing path)
  • Small additional overhead for namespaced operations:
    • Key prefix addition
    • Epoch checks
    • Additional epoch storage per namespace

Future Considerations

  1. Namespace cleanup utilities
  2. Epoch management tools
  3. Namespace statistics and monitoring
  4. Cross-namespace operations (if needed)

Related Issue

@CLAassistant
Copy link

CLAassistant commented May 20, 2025

CLA assistant check
All committers have signed the CLA.

@superhx
Copy link
Collaborator

superhx commented May 22, 2025

@AminMortezaie We sincerely appreciate your contributions. However, it appears that there might be a misunderstanding regarding the requirement. Instead of manipulating the key field, what we actually need is to extend the KVRecord by adding two new fields: namespace and epoch.

  • With namespace field, we can fast filter the KVs in certain namespace instead of matching them by prefix.

@AminMortezaie
Copy link
Author

Great, I will change to the real requirements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants