Estuary has released several updates to its PostgreSQL change data capture (CDC) connector focused on backfill recovery, discovery at scale, and schema-change detection.
PostgreSQL capture bindings now support an Additional Backfill Filter, allowing a SQL expression to be applied to backfill queries for an individual table.
This can reduce the amount of data that needs to be re-read when only a subset of a large PostgreSQL table needs to be recovered or re-backfilled. For example, teams can use an indexed identifier or timestamp to limit the rows included in a recovery backfill.
The filter affects backfills only. Ongoing PostgreSQL CDC continues to capture new changes independently of the filter.
Because the SQL expression is applied directly to backfill queries, this is an advanced option and should be configured carefully. Changing the filter requires the affected binding to be backfilled again.
Estuary has also added PostgreSQL discovery filters that allow captures to control which tables are surfaced during connector discovery.
Captures can now:
These controls are useful for PostgreSQL environments with large numbers of schemas or tables where only part of the database should participate in a CDC pipeline.
Table metadata discovery is also processed in smaller groups to improve reliability when working with databases containing large numbers of tables.
PostgreSQL captures can now configure how frequently Estuary re-runs discovery while a capture is active.
Rediscovery is used to detect changes such as newly added tables and schema updates. The default interval is 15 minutes and can be adjusted for environments that require faster or less frequent discovery.
This gives PostgreSQL operators more control over the trade-off between schema-change detection latency and catalog-query activity.
Estuary's PostgreSQL source connector uses logical replication to capture inserts, updates, and deletes from the PostgreSQL write-ahead log (WAL).
When a new capture starts, Estuary can first backfill the existing contents of selected tables and then transition to ongoing CDC. The new backfill and discovery controls provide additional options for operating this process on large PostgreSQL environments and during recovery scenarios.
The PostgreSQL CDC connector supports PostgreSQL 10.0 and later across self-hosted PostgreSQL and managed PostgreSQL services.