Add recovery slot advancement and enhanced rescue workflow procedures. #269
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 3 commits
November 20, 2025 18:15
This commit extends the rescue subscription lifecycle with recovery slot advancement functionality and comprehensive recovery orchestration procedures. The changes enable more robust disaster recovery workflows by providing precheck validation, multi-phase recovery execution, and automated slot management. The recovery slot management enhancements include a new function to advance recovery slots to minimum position across all slots, improving progress tracking and validation. Recovery workflow procedures now include spock.recover_precheck() for validation before recovery, extended spock.recover_run() with OUT parameter for cloned slot name, and comprehensive error handling with verbose logging throughout. Recovery SQL procedures support stop_lsn, skip_lsn, and stop_timestamp parameters for flexible recovery scenarios. Sample scripts and test files are included for complete workflow examples and validation. Code changes update spock_recovery.c with slot advancement logic, extend spock_functions.c with new SQL function wrappers, enhance spock_manager.c for rescue subscription lifecycle, and improve error handling in spock_apply.c and progress tracking in spock_output_plugin.c. Migration scripts are updated to support the new rescue subscription features. These changes build upon SPOC-137 rescue subscription features and enable production-ready disaster recovery workflows for multi-node Spock clusters.
rasifr
reviewed
Nov 25, 2025
| * - Otherwise, check if this specific origin should be forwarded | ||
| */ | ||
| ret = list_length(data->forward_origins) == 0; | ||
| if (list_length(data->forward_origins) == 0) |
Member
There was a problem hiding this comment.
It looks like the current logic is technically correct as implemented, because for
list_length(data->forward_origins) == 1 the code always sets ret = false regardless of whether the element is "all" or something else OR perhaps something else was intended?
Comment on lines
+616
to
+627
| if (current_slot_lsn == InvalidXLogRecPtr || min_remote_lsn > current_slot_lsn) | ||
| { | ||
| SpinLockAcquire(&recovery_slot->mutex); | ||
| recovery_slot->data.confirmed_flush = min_remote_lsn; | ||
| recovery_slot->data.restart_lsn = min_remote_lsn; | ||
| SpinLockRelease(&recovery_slot->mutex); | ||
|
|
||
| ReplicationSlotMarkDirty(); | ||
| ReplicationSlotSave(); | ||
|
|
||
| elog(LOG, "advanced recovery slot '%s' to minimum peer position %X/%X", | ||
| slot_name, LSN_FORMAT_ARGS(min_remote_lsn)); |
Member
There was a problem hiding this comment.
Perhaps pg_replication_slot_advance can be used?
Comment on lines
+581
to
+585
| /* Get remote LSN from origin status */ | ||
| replorigin_session_setup(origin_id, 0); | ||
| replorigin_session_reset(); | ||
| remote_lsn = replorigin_session_get_progress(false); | ||
|
|
Member
There was a problem hiding this comment.
Is it possible that session_setup/reset here can messup with apply manager's session? Maybe getting the lsn from pg_replication_origin_status or progress table can be more safe?
Comment on lines
1144
to
1148
| /* Create the cloned logical slot using the Spock output plugin */ | ||
| ReplicationSlotCreate(cloned_slot_name, true, RS_PERSISTENT, | ||
| false, true, false); | ||
| cloned_slot = MyReplicationSlot; | ||
| if (cloned_slot == NULL) |
Member
There was a problem hiding this comment.
Can pg_copy_logical_replication_slot be used?
added 9 commits
November 28, 2025 16:47
This commit adds automatic linking against the gcov library when coverage-related compiler flags are detected in CFLAGS. The Makefile now checks for common coverage flags such as --coverage, -fprofile-arcs, and -ftest-coverage, and links the gcov library accordingly to support code coverage instrumentation during testing.
This commit adds conditional compilation support for the checkpoint hook feature, allowing Spock to compile on PostgreSQL versions that do not provide the checkpoint hook interface. When the hook is not available, the code defines a compatible type and skips hook registration, ensuring backward compatibility across different PostgreSQL versions.
This commit changes the default replication set used for automatic DDL replication from DEFAULT_INSONLY_REPSET_NAME to DDL_SQL_REPSET_NAME. This ensures that DDL statements are properly replicated using the dedicated DDL replication set, maintaining consistency with the intended replication behavior for schema changes.
This commit disables the delta_apply_function attribute option feature by commenting out the implementation code. The feature requires the pg18-015-attoptions patch to be applied to PostgreSQL source code. When the patch is not available, the code is disabled to prevent compilation errors, with a note indicating that the patch must be applied to enable this functionality.
This commit modifies the WAL redo handler to skip all APPLY_PROGRESS records during recovery to prevent hash table corruption. If the hash table is corrupted during recovery, hash_search will call elog(PANIC) which cannot be caught and terminates the process. By skipping these records during recovery, the hash table will be rebuilt from scratch when recovery completes and shared memory is reinitialized, avoiding potential corruption issues.
This commit adds comprehensive support for rescue subscriptions throughout the apply worker system. Rescue subscriptions do not use the hash table for progress tracking, instead tracking progress via pg_replication_origin_status. The group progress tracking system now properly handles rescue subscriptions, validating and fixing inconsistent LSN values during WAL recovery instead of asserting. The apply worker handles cases where apply_group is NULL for rescue subscriptions, skipping hash table attachment and progress updates. Periodic stop LSN checking is added for rescue subscriptions during timeout periods. The get_apply_group_progress function is extended to include rescue subscriptions by querying pg_replication_origin_status separately and merging results. Additional safeguards ensure the hash table is initialized before access and handle cases where MyProc is not available during shmem startup.
This commit enhances recovery slot creation across both the recovery module and manager worker to ensure proper transaction state initialization before attempting to create replication slots. ReplicationSlotCreate may need to wait for locks, which requires a fully initialized PGPROC structure. The code now verifies that MyProc exists before creating slots and improves error handling by switching to TopMemoryContext when copying error data to avoid ErrorContext assertion failures. The subscription resumption logic is improved to enable regular subscriptions that were disabled but not rescue-suspended during recovery operations. Version-specific ReplicationSlotCreate calls are handled for PostgreSQL 17 and later versions.
This commit fixes the shared memory startup sequence to properly release AddinShmemInitLock after recovery slots shared memory initialization completes. The lock was previously held but not released, which could cause deadlocks or prevent other processes from initializing shared memory structures. The lock is now explicitly released after all shared memory initialization is complete.
This commit significantly refactors both the recovery SQL procedures and the Python cluster management script to improve validation, error handling, and monitoring capabilities. The recovery procedures now perform read-only validation with detailed INFO messages and check lag_tracker LSNs to determine if recovery is actually needed. The procedures handle existing rescue subscriptions more gracefully, checking their cleanup_pending and failed status before proceeding. LSN comparison logic ensures recovery only proceeds when the source node is ahead of the target node. The phase clone procedure now uses lag_tracker for more accurate LSN determination and includes verification steps to ensure cloned slots and rescue subscriptions are actually created. The cluster management script is enhanced with improved replication status monitoring, better feedback during recovery operations, and enhanced exception handling for interrupted operations. Error messages are improved throughout with consistent formatting and better context.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.