Implement a fingerprinting mechanism to track compaction states in a more efficient manner #18844

capistrant · 2025-12-15T20:16:25Z

Description

Add new functionality to Compaction Supervisors. Instead of storing compaction state for segments individually, centralize the state storage in a new metadata table. Individual segments will store a computed fingerprint that references a compaction state in the new metadata table. Since many segments will eventually end up sharing common compaction states, this should greatly reduce duplication in metadata storage.

note: This applies only to Compaction Supervisors. Scheduled compaction on the coordinator.

Compaction State Fingerprinting

Instead of storing CompactionState as the lastCompactionState field in every compaction segment, generate a fingerprint for a CompactionState and attach that to compacted segments. Add new centralized storage for CompactionState where individual states can be looked up by the aforementioned fingerprint. Since it is common for many segments in a data source to share a single CompactionState, this greatly reduces the metadata storage overhead for storing compaction states.

Metadata Store Changes

`druid_segments`

Add new column compaction_state_fingerprint that stores the fingerprint representation of the segments current compaction state. It can be null if no compaction has taken place.

`druid_compactionStates`

New metadata table that stores the full CompactionState associated with a fingerprint. Segments can look up their full compaction state here by using the compaction_state_fingerprint that they are associated with.

`CompactionStateManager`

The CompactionStateManager is responsible for managing the persistence and lifecycle of compaction states. It stores unique compaction configurations (identified by fingerprints) in the metadata database. The manager tracks which compaction states are actively referenced by segments, marking unreferenced states as unused and periodically cleaning up old unused states. This fingerprinting approach allows Druid to efficiently store and retrieve compaction metadata without duplicating identical compaction configurations across multiple segments.

`OnHeapCompactionStateManager`

Meant to serve as a mechanism for testing and simulations where metadata persistence may not be available/needed

`CompactionStateCache`

CompactionStateCache is a new component of the HeapMemorySegmentMetadataCache. It is modeled strongly after the existing datasource schema cache. This is where the existing compaction states are cached for reference by compaction supervisors.

CompactSegments Coordinator Duty Roadmap

This PR does not add support for compaction state fingerprinting to the coordinator based scheduled compaction that is carried out by the CompactSegments coordinator duty. This is because the Druid roadmap is to move all scheduled compaction to compaction supervisors. Making the decision to forgo compaction state fingerprint support for the legacy duty based compaction is a conscious choice we are making to help drive usage of supervisors and limit changes to the legacy duty based compaction code. Another PR should be spun up to officially deprecate legacy scheduled compaction on the coordinator.

Legacy `lastCompactionState` Roadmap

This PR implements no automatic transition to fingerprints for segments who are compacted and store CompactionState in their lastCompactionState field. Instead this PR aims to continue supporting lastCompactionState in Compaction decision making for segments compacted before fingerprinting. This means that legacy segments will not have to be re-compacted simply because they are not fingerprinted, as long as they have the proper CompactionState as specified by the compaction configuration for the data source in question.

This PR also continues to write both the new fingerprint as well as the legacy lastCompactionState by default. This allows normal rolling upgrade order as well as Druid version rollback without un-needed re-compaction. An operator can disable writing lastCompactionState by updating the cluster compaction config, after the Druid upgrade completes. Eventually, Druid code base will cease writing lastCompactionState at all and instead force using fingerprinting going forward. I think this should be done in the Druid version following the first version that this new feature is seen in. Even at this point, lastCompactionState will need to continue to be supported for already written segments, unless we want to devise an automated migration plan that can run in the background of a cluster to get all compacted segments migrated to fingerprinting.

Release note

coming soon

Upgrade Note

Metadata store changes are required for this upgrade. If you already have druid.metadata.storage.connector.createTables set to true no action is needed. If you have this feature disabled, you will need to alter the segments table and create the compactionStates table. Postgres DDL is provided below as a guide. You will have to adapt the syntax to your metadata store backend as well as use proper table naming depending on your configured table prefix and database.

-- create the compaction states lookup table and associated indices
CREATE TABLE druid_compactionStates (
    id BIGSERIAL NOT NULL,
    created_date VARCHAR(255) NOT NULL,
    datasource VARCHAR(255) NOT NULL,
    fingerprint VARCHAR(255) NOT NULL,
    payload BYTEA NOT NULL,
    used BOOLEAN NOT NULL,
    used_status_last_updated VARCHAR(255) NOT NULL,
    PRIMARY KEY (id),
    UNIQUE (fingerprint)
  );

  CREATE INDEX idx_druid_compactionStates_fingerprint ON druid_compactionStates(fingerprint);
  CREATE INDEX idx_druid_compactionStates_used ON druid_compactionStates(used, used_status_last_updated);

-- modify druid_segments table to have a column for storing compaction state fingerprints
ALTER TABLE druid_segments ADD COLUMN compaction_state_fingerprint VARCHAR(255);

Key changed/added classes in this PR

CompactionStatus
CompactionConfigBasedJobTemplate
CompactionState
SQLMetadataConnector
CompactionStateManager
CompactionStateCache
CompactSegments
KillUnreferencedCompactionState

This PR has:

… storage configurable

processing/src/main/java/org/apache/druid/timeline/CompactionState.java

processing/src/test/java/org/apache/druid/timeline/DataSegmentTest.java

...ervice/src/main/java/org/apache/druid/indexing/compact/CompactionConfigBasedJobTemplate.java

processing/src/main/java/org/apache/druid/timeline/CompactionState.java

server/src/main/java/org/apache/druid/server/coordinator/duty/CompactSegments.java

server/src/main/java/org/apache/druid/server/compaction/CompactionStatus.java

server/src/main/java/org/apache/druid/segment/metadata/CompactionStateManager.java

capistrant · 2025-12-15T22:28:29Z

server/src/main/java/org/apache/druid/segment/metadata/CompactionStateManager.java

+  @LifecycleStop
+  public void stop()
+  {
+    fingerprintCache.invalidateAll();


does this cache object need any other lifecycle cleanup?

server/src/main/java/org/apache/druid/segment/metadata/CompactionStateManager.java

capistrant · 2025-12-15T22:30:45Z

server/src/main/java/org/apache/druid/segment/metadata/CompactionStateManager.java

what about if the operator has create tables disabled and does not properly create the table before upgrading?

server/src/main/java/org/apache/druid/segment/metadata/CompactionStateManager.java

kfaraz

Thanks for the feature, @capistrant !

I have started going through the PR, leaving a partial review here.
I am yet to go through several changes, such as the ones made in CompactionStatus, DatasourceCompactibleSegmentIterator, etc.

kfaraz · 2025-12-16T09:01:14Z

server/src/main/java/org/apache/druid/segment/metadata/HeapMemoryCompactionStateManager.java

+ * <p>
+ * Useful for simulations and unit tests where database persistence is not needed.
+ */
+public class HeapMemoryCompactionStateManager extends CompactionStateManager


Might be cleaner to let CompactionStateManager be an interface, and let both the heap-based and the concrete class implement it.

kfaraz · 2025-12-16T09:01:49Z

server/src/main/java/org/apache/druid/segment/metadata/HeapMemoryCompactionStateManager.java

+ * In-memory implementation of {@link CompactionStateManager} that stores
+ * compaction state fingerprints in heap memory without requiring a database.
+ * <p>
+ * Useful for simulations and unit tests where database persistence is not needed.


If this is used only in tests, we should probably put it in the test source root src/test/java.

That is where I originally put it, but then I tried to use it in a simulation class which is in the app code, not test. Let me review this though, maybe I am mistaken on how it is all working with the simulations

Oh, I see. Are you referring to CoordinatorSimulationBuilder or some other class?

no CompactionRunSimulator, https://github.com/apache/druid/pull/18844/files#diff-b8a4fdf52e09ff26fa6f5610c021d196b9fa99673b83051de794ed07257be13b ... It creates CompactSegments instance, which as of now requires a CompactionStateManager. But I guess if we go the route of not supporting fingerprinting in the coordinator duty led compaction, this may not be a problem and it can be moved to the test space.

kfaraz · 2025-12-16T12:04:44Z

docs/configuration/index.md

+|`druid.manager.compactionState.cacheSize`|The maximum number of compaction state fingerprints to cache in memory on the coordinator and overlord. Compaction state fingerprints are used to track the compaction configuration applied to segments. Consider increasing this value if you have a large number of datasources with compaction configurations.|`100`|
+|`druid.manager.compactionState.prewarmSize`|The number of most recently used compaction state fingerprints to load into cache on Coordinator startup. This pre-warms the cache to improve performance immediately after startup.|`100`|


Both Coordinator and Overlord (with segment metadata caching enabled) already keep all used segments in memory, including the respective (interned) CompactionState objects as well.
I don't think the number of distinct CompactState objects that we keep in memory will increase after this patch.

Do we still need to worry about the cache size of these objects?
Does a cache miss trigger a fetch from metadata store?

kfaraz · 2025-12-16T12:11:34Z

processing/src/main/java/org/apache/druid/timeline/CompactionState.java

 {
+
+  /**
+   * Lazy initialization holder for deterministic ObjectMapper.


I wonder if we shouldn't just inject this mapper annotated with @Sorted or @Deterministic as a lazy singleton. It may be injected into CompactionStateManager and fingerprints will always be created by that class rather than using a static utility method.

processing/src/main/java/org/apache/druid/timeline/DataSegment.java

kfaraz · 2025-12-16T13:51:14Z

...ervice/src/main/java/org/apache/druid/indexing/compact/CompactionConfigBasedJobTemplate.java

+    if (segmentIterator.hasNext()) {
+      // If we are going to create compaction jobs for this compaction state, we need to persist the fingerprint -> state
+      // mapping so compacted segments from these jobs can reference a valid compaction state.
+      params.getCompactionStateManager().persistCompactionState(


The templates should only perform lightweight (i.e. non-IO) read-only operations as createCompactionJobs may be called frequently.
We should not do any persistence here.
Instead, the params can hold some mapping where we can add this compaction state and perform persistence on-demand (perhaps in the CompactionJobQueue).

thank you for the guidance. Will work on how to get this out of hot path

kfaraz · 2025-12-16T14:27:01Z

multi-stage-query/src/main/java/org/apache/druid/msq/exec/ControllerImpl.java

    }
  }

+  private static Function<Set<DataSegment>, Set<DataSegment>> addCompactionStateFingerprintToSegments(String compactionStateFingerprint)


Let's re-use the static function from AbstractTask itself?

sure! I didn't know if it was bad form to reach into that class from MSQ. But I like having just one impl

I think it is fine to use AbstractTask in the MSQ code. Alternatively, you can put the method in IndexTaskUtils too.

kfaraz · 2025-12-16T14:27:28Z

multi-stage-query/src/main/java/org/apache/druid/msq/exec/ControllerImpl.java

              Tasks.DEFAULT_STORE_COMPACTION_STATE
          );

+      String compactionStateFingerprint = querySpec.getContext()


Suggested change

String compactionStateFingerprint = querySpec.getContext()

final String compactionStateFingerprint = querySpec.getContext()

kfaraz · 2025-12-16T14:29:42Z

website/.spelling

 pre-compute
 pre-computed
 pre-computing
+pre-dates


predates need not be hyphenated.

sometimes my inability to spell, compounded by my inability to google how to spell, is embarrassing. this is one of those times. will fix

kfaraz · 2025-12-16T14:38:43Z

server/src/main/java/org/apache/druid/segment/metadata/CompactionStateManager.java

+ * </p>
+ */
+@ManageLifecycle
+public class CompactionStateManager


I don't feel that pre-warming the cache is really necessary. The fingerprint needs to be retrieved only when running the CompactionJobQueue on Overlord or CompactSegments on Coordinator.

Let's always keep all the compaction states in memory. We are already keeping all the used segments in memory. The distinct CompactionState objects will be fairly small in number and size.

The states can be cached in HeapMemorySegmentMetadataCache which already serves as a cache for used segments, pending segments and schemas.

If possible, let's support the fingerprint flow only with compaction supervisors and not the Coordinator-based CompactSegments duty. That would simplify the new flow and be another motivation for users to migrate to using compaction supervisors.

If possible, let's support the fingerprint flow only with compaction supervisors and not the Coordinator-based CompactSegments duty. That would simplify the new flow and be another motivation for users to migrate to using compaction supervisors.

would we want to deprecate CompactSegments compaction on the coordinator in this case? so we aren't forever supporting compaction without fingerprints + compaction with fingerprints?

Yes, the plan was to deprecate CompactSegments once compaction supervisors took off. I don't fully recall if compaction supervisors is already marked GA or not. They would also have to be made the default, if we want to start deprecation of CompactSegments.

But I feel all of this should be out of scope for the current PR.

If supporting the fingerprint logic in CompactSegments is not additional work and does not complicate the flow, we can leave it as is.

My only concern is that there should be just one service that is responsible for persisting new fingerprints. I would prefer that to be the Overlord, so that it always has a consistent cache state. So we either just don't support fingerprints on the Coordinator or we handle persistence by calling an Overlord API.

(I am yet to go through the whole PR to identify all the call sites that may persist a compaction state. I have only found the one in CompactionConfigBasedJobTemplate so far.)

… out of hot path

…red up

…d heap impl

processing/src/main/java/org/apache/druid/guice/DruidSecondaryModule.java

+  {
+    final ObjectMapper sortedMapper = new DefaultObjectMapper();
+    sortedMapper.configure(SerializationFeature.ORDER_MAP_ENTRIES_BY_KEYS, true);
+    sortedMapper.configure(MapperFeature.SORT_PROPERTIES_ALPHABETICALLY, true);


server/src/test/java/org/apache/druid/segment/metadata/HeapMemoryCompactionStateManager.java

+  {
+    ObjectMapper mapper = new DefaultObjectMapper();
+    mapper.configure(SerializationFeature.ORDER_MAP_ENTRIES_BY_KEYS, true);
+    mapper.configure(MapperFeature.SORT_PROPERTIES_ALPHABETICALLY, true);


server/src/test/java/org/apache/druid/segment/metadata/PersistedCompactionStateManagerTest.java

+  {
+    ObjectMapper mapper = new DefaultObjectMapper();
+    mapper.configure(SerializationFeature.ORDER_MAP_ENTRIES_BY_KEYS, true);
+    mapper.configure(MapperFeature.SORT_PROPERTIES_ALPHABETICALLY, true);


server/src/test/java/org/apache/druid/server/compaction/CompactionStatusTest.java

.../test/java/org/apache/druid/server/coordinator/duty/KillUnreferencedCompactionStateTest.java

+  {
+    ObjectMapper mapper = new DefaultObjectMapper();
+    mapper.configure(SerializationFeature.ORDER_MAP_ENTRIES_BY_KEYS, true);
+    mapper.configure(MapperFeature.SORT_PROPERTIES_ALPHABETICALLY, true);


server/src/test/java/org/apache/druid/server/compaction/CompactionStatusTest.java

+        .withGranularitySpec(new UserCompactionTaskGranularityConfig(Granularities.DAY, null, null))
+        .build();
+
+    CompactionState expectedState = CompactionStatus.createCompactionStateFromConfig(compactionConfig);


capistrant · 2026-01-05T01:25:59Z

@kfaraz I pushed up changes addressing the review you had to date. I ended up going with the idea of dropping support for fingerprinting from the CompactSegments duty. I also did my best to wrap my head around the segment metadata caching and tying in compaction state caching to that instead of having the caching as a one off in CompactionStateManager. now the manager just handles persists and then the lifecycle management of states in the database as they go unused and age out. I also took a crack at moving when the compaction states are persisted, per your suggestion. They no longer happen in the template code, but rather will happen when tasks are about to be ran.

I have talked briefly with @clintropolis and I think he will be submitting a review soon with thoughts on changing the naming in the metastore to drop the "compaction" from the names since we are already do more than just compaction and are trending towards adding even more functionality than pure compaction. If you have any thoughts on table and column naming, I'd love to hear it. If we do refactor the naming, I think it will be up for debate how much we change naming in the app code in this PR vs another refactoring PR that focuses on making the naming more generic in the app code when it comes to compaction supervisors.

kfaraz · 2026-01-05T03:45:07Z

Thanks for the update, @capistrant !
I will try to finish a review of the latest changes today.

techdocsmith · 2026-01-07T17:42:59Z

A couple of questions re metadata naming. Forgive me if I don't understand the naming conventions well. Take or leave as you will.

druid_segments
Is druid necessary here? What other segments migth we distinguish the Druid segments from?

druid_compactionStates
Suggest using similar underscore/all lowercase instead of underscore + camelcase: compaction_states. Same question about needing to specify druid

compaction_state_fingerprint
If this is a column within the druid_compactionStates, could it just be called fingerprint ?

kfaraz · 2026-01-07T17:51:26Z

@techdocsmith , I have tried to answer your queries below.

druid_segments
Is druid necessary here? What other segments migth we distinguish the Druid segments from?

This is an existing convention in the Druid codebase. The default names of the metadata tables use the prefix druid_ and use a camel-cased name. Howevers, users can always configure their cluster to use some other name.

compaction_state_fingerprint
If this is a column within the druid_compactionStates, could it just be called fingerprint ?

This is the foreign key column in the druid_segments table which refers to the fingerprint column in the druid_compactionStates table.

capistrant added 14 commits December 15, 2025 13:50

meatadata store bits part 1

d8490b0

annotate segments with compaction fingerprint before persist

3d2d423

Add ability to generate compaction state fingerprint

48854f4

add fingerprint to task context and make legacy last compaction state…

c6a3367

… storage configurable

update embedded tests for compaction supervisors to flex fingerprints

f3b706e

checkpoint with persisting compaction states

46fb807

add duty to clean up unused compaction states

0fef358

take fingerprints into account in CompactionStatus

edeaf30

Add and improve tests

97daf3f

get rid of some todo comments

dbcdfcf

fix checkstyle

38f6d15

cleanup some more TODO

4cf1197

Add some docs

ba269bd

update web console

f168bc9

github-actions bot added Area - Documentation Area - Batch Ingestion Area - Web Console Area - Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels Dec 15, 2025

capistrant added the Area - Compaction label Dec 15, 2025

github-advanced-security bot found potential problems Dec 15, 2025

View reviewed changes

capistrant added 3 commits December 15, 2025 15:19

make cache size configurable and fix some spelling

2292b15

fixup use of deprecated builder

74c8ebc

fix checktyle

adac5ec

capistrant commented Dec 15, 2025

View reviewed changes

capistrant added 2 commits December 15, 2025 17:20

fix coordinator compactsegments duty and respond to self review comments

4fb3a9c

fix spellchecker

708c6f8

kfaraz reviewed Dec 16, 2025

View reviewed changes

capistrant added 2 commits December 16, 2025 09:58

predates is a word

03bb14a

improve some javadocs

a262f79

capistrant added 13 commits December 16, 2025 10:05

simplify some test assertions based on review

6126e22

better naming

b78ec13

controller impl cleanup

78f115e

For compaction supervisors, take persisting pending compaction states…

f06d715

… out of hot path

use Configs.valueOrDefault helper in data segment

d571e43

Refactor where fingerprinting happens and how the object mapper is wi…

07afc2f

…red up

refactor CompactionStateManager into an interface with a persisted an…

12ea741

…d heap impl

Merge branch 'master' into compaction-fingerprinting

f57527a

remove fingerprinting support from the coordinator compact segments duty

858cbd3

Move on heap compaction state manager to test sources

9afab2f

CompactionStateManager is now overlord only

34a8a11

Refactor how the compaction state fingerprint cache is wired up

7214418

Merge branch 'master' into compaction-fingerprinting

6a9743a

github-advanced-security bot found potential problems Jan 2, 2026

View reviewed changes

capistrant added 2 commits January 4, 2026 10:26

prettify

cd55b0e

small changes after self-review

58724cc

github-advanced-security bot found potential problems Jan 4, 2026

View reviewed changes

		\|`druid.manager.compactionState.cacheSize`\|The maximum number of compaction state fingerprints to cache in memory on the coordinator and overlord. Compaction state fingerprints are used to track the compaction configuration applied to segments. Consider increasing this value if you have a large number of datasources with compaction configurations.\|`100`\|
		\|`druid.manager.compactionState.prewarmSize`\|The number of most recently used compaction state fingerprints to load into cache on Coordinator startup. This pre-warms the cache to improve performance immediately after startup.\|`100`\|

	String compactionStateFingerprint = querySpec.getContext()
	final String compactionStateFingerprint = querySpec.getContext()

Implement a fingerprinting mechanism to track compaction states in a more efficient manner #18844

Are you sure you want to change the base?

Implement a fingerprinting mechanism to track compaction states in a more efficient manner #18844

Uh oh!

Conversation

capistrant commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Compaction State Fingerprinting

Metadata Store Changes

druid_segments

druid_compactionStates

CompactionStateManager

OnHeapCompactionStateManager

CompactionStateCache

CompactSegments Coordinator Duty Roadmap

Legacy lastCompactionState Roadmap

Release note

Upgrade Note

Key changed/added classes in this PR

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kfaraz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Check notice

Uh oh!

Check notice

capistrant commented Dec 15, 2025 •

edited

Loading

`druid_segments`

`druid_compactionStates`

`CompactionStateManager`

`OnHeapCompactionStateManager`

`CompactionStateCache`

Legacy `lastCompactionState` Roadmap