Comparing changes

…fest.postgresql.org/patch/5438/ merged in single commit. it is required for stability of stress tests.

Introduce stress tests for concurrent index operations: - test concurrent inserts/updates during CREATE/REINDEX INDEX CONCURRENTLY - cover various index types (btree, gin, gist, brin, hash, spgist) - test unique and non-unique indexes - test with expressions and predicates - test both parallel and non-parallel operations These tests verify the behavior of the following commits.

…dex builds Long-living snapshots used by CREATE INDEX CONCURRENTLY and REINDEX CONCURRENTLY can hold back the global xmin horizon. Commit d9d0762 attempted to allow VACUUM to ignore such snapshots to mitigate this problem. However, this was reverted in commit e28bb88 because it could cause indexes to miss heap tuples that were HOT-updated and HOT-pruned during the index creation, leading to index corruption. This patch introduces an alternative by periodically resetting the snapshot used during the first phase. By resetting the snapshot every N pages during the heap scan, it allows the xmin horizon to advance. Currently, this technique is applied to: - only during the first scan of the heap: The second scan during index validation still uses a single snapshot to ensure index correctness - non-parallel index builds: Parallel index builds are not yet supported and will be addressed in a following commits - non-unique indexes: Unique index builds still require a consistent snapshot to enforce uniqueness constraints, will be addressed in a following commits A new scan option SO_RESET_SNAPSHOT is introduced. When set, it causes the snapshot to be reset "between" every SO_RESET_SNAPSHOT_EACH_N_PAGE pages during the scan. The heap scan code is adjusted to support this option, and the index build code is modified to use it for applicable concurrent index builds that are not on system catalogs and not using parallel workers.

Extend periodic snapshot reset support to parallel builds, previously limited to non-parallel operations. This allows the xmin horizon to advance during parallel concurrent index builds as well. The main limitation of applying that technic to parallel builds was a requirement to wait until workers processes restore their initial snapshot from leader. To address this, following changes applied: - add infrastructure to track snapshot restoration in parallel workers - extend parallel scan initialization to support periodic snapshot resets - wait for parallel workers to restore their initial snapshots before proceeding with scan - relax limitation for parallel worker to call GetLatestSnapshot

Previously, concurrent builds if unique index used a fixed snapshot for the entire scan to ensure proper uniqueness checks. Now reset snapshots periodically during concurrent unique index builds, while still maintaining uniqueness by: - ignoring SnapshotSelf dead tuples during uniqueness checks in tuplesort as not a guarantee, but a fail-fast mechanics - adding a uniqueness check in _bt_load that detects multiple alive tuples with the same key values as a guarantee of correctness Tuples are SnapshotSelf tested only in the case of equal index key values, overwise _bt_load works like before.

This patch provides infrastructure for following enhancements to concurrent index builds by: - ii_Auxiliary in IndexInfo: indicates that an index is an auxiliary index used during concurrent index build - validate_index in IndexVacuumInfo: set if index_bulk_delete called during the validation phase of concurrent index build - STIR(Short-Term Index Replacement) access method is introduced, intended solely for short-lived, auxiliary usage STIR functions designed as an ephemeral helper during concurrent index builds, temporarily storing TIDs without providing the full features of a typical access method. As such, it raises warnings or errors when accessed outside its specialized usage path. Planned to be used in following commits.

Extend tuplestore to store individual Datum values: - fixed-length datatypes: store raw bytes without a length header - variable-length datatypes: include a length header and padding - by-value types: store inline This support enables usages tuplestore for non-tuple data (TIDs) in the next commit.

Replace the second table full scan in concurrent index builds with an auxiliary index approach: - create a STIR auxiliary index with the same predicate (if exists) as in main index - use it to track tuples inserted during the first phase - merge auxiliary index with main index during validation to catch up new index with any tuples missed during the first phase - automatically drop auxiliary when main index is ready To merge main and auxiliary indexes: - index_bulk_delete called for both, TIDs put into tuplesort - both tuplesort are being sorted - both tuplesort scanned with two pointers looking for the TIDs present in auxiliary index, but absent in main one - all such TIDs are put into tuplestore - all TIDs in tuplestore are fetched using the stream, tuplestore used in heapam_index_validate_scan_read_stream_next to provide the next page to prefetch - if fetched tuple is alive - it is inserted into the main index This eliminates the need for a second full table scan during validation, improving performance especially for large tables. Affects both CREATE INDEX CONCURRENTLY and REINDEX INDEX CONCURRENTLY operations.

During concurrent index operations, auxiliary indexes may be left as orphaned objects when errors occur (junk auxiliary indexes). This patch improves the handling of such auxiliary indexes: - add auxiliaryForIndexId parameter to index_create() to track dependencies between main and auxiliary indexes - automatically drop auxiliary indexes when the main index is dropped - delete junk auxiliary indexes properly during REINDEX operations

Skip unnecessary computations for auxiliary indices by: - in the index‐insert path, detect auxiliary indexes and bypass Datum value computation - set indexUnchanged=false for auxiliary indices to avoid redundant checks These optimizations reduce overhead during concurrent index operations.

Enhances validation phase of concurrently built indexes by periodically refreshing snapshots rather than using a single reference snapshot. This addresses issues with xmin propagation during long-running validations. The validation now takes a fresh snapshot every few pages, allowing the xmin horizon to advance. This restores feature of commit d9d0762, which was reverted in commit e28bb88. New STIR-based approach is not depends on single reference snapshot anymore.

This optimization allowed concurrent index builds to ignore other indexes without expressions or predicates. With the new snapshot handling approach that periodically refreshes snapshots, this optimization is no longer necessary. The change simplifies concurrent index build code by: - removing the PROC_IN_SAFE_IC process status flag - eliminating set_indexsafe_procflags() calls and related logic - removing special case handling in GetCurrentVirtualXIDs() - removing related test cases and injection points

…and short-term resetting shapshots This branch was automatically generated by a robot using patches from an email thread registered at: https://commitfest.postgresql.org/patch/4971 The branch will be overwritten each time a new patch version is posted to the thread, and also periodically to check for bitrot caused by changes on the master branch. Patch(es): https://www.postgresql.org/message-id/CADzfLwW5bDWSxjHK7mqX8Lewki3+5FBydBC+nVcxg4xMGKscyw@mail.gmail.com Author(s): Michail Nikolaev, Mihail Nikalayeu

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Comparing changes

Open a pull request

Commits on May 29, 2025

This comparison is taking too long to generate.

Uh oh!