Skip to content

Conversation

@rafiss
Copy link
Collaborator

@rafiss rafiss commented Dec 24, 2025

Previously, INSPECT jobs without a historical AS OF SYSTEM TIME clause would not create protected timestamp records, but still used an AOST clause with the current timestamp. If span processing took a long time (especially with BulkLowQoS admission control), garbage collection could occur before the query completed, resulting in "batch timestamp must be after replica GC threshold" errors.

This change adds per-span protected timestamp protection when INSPECT uses "now" as the AOST. It uses TryToProtectBeforeGC which waits until 80% of the GC TTL has elapsed before creating a PTS, avoiding unnecessary PTS creation for quick operations. The PTS is cleaned up when span processing completes.

Resolves: #159866
Epic: None

Release note: None

Previously, INSPECT jobs without a historical AS OF SYSTEM TIME clause
would not create protected timestamp records, but still used an AOST
clause with the current timestamp. If span processing took a long time
(especially with BulkLowQoS admission control), garbage collection could
occur before the query completed, resulting in "batch timestamp must be
after replica GC threshold" errors.

This change adds per-span protected timestamp protection when INSPECT
uses "now" as the AOST. It uses TryToProtectBeforeGC which waits until
80% of the GC TTL has elapsed before creating a PTS, avoiding
unnecessary PTS creation for quick operations. The PTS is cleaned up
when span processing completes.

Resolves: cockroachdb#159866
Epic: None

Release note: None
@rafiss rafiss requested a review from spilchen December 24, 2025 22:14
@rafiss rafiss added the backport-26.1.x Flags PRs that need to be backported to 26.1 label Dec 24, 2025
@rafiss rafiss requested a review from a team as a code owner December 24, 2025 22:14
@cockroach-teamcity
Copy link
Member

This change is Reviewable

Copy link
Collaborator

@fqazi fqazi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm_strong:

@fqazi reviewed 3 files and all commit messages, and made 1 comment.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @spilchen).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-26.1.x Flags PRs that need to be backported to 26.1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

inspect: GC threshold errors during scale test

3 participants