November 21, 2024: PostgreSQL 17.2, 16.6, 15.10, 14.15, 13.18, and 12.22 Released!

[ANNOUNCE] SynchDB 1.0 Released - PostgreSQL Extension for Real-Time Database Synchronization

Posted on 2024-12-27 by Hornetlabs Technology Inc
Related Open Source

SynchDB 1.0 Release Notes

Release Date: December 24, 2024

We're excited to announce the release of SynchDB 1.0! This PostgreSQL extension enables seamless data synchronization from multiple heterogeneous databases (MySQL, MS SQL Server) directly to PostgreSQL. SynchDB handles all data synchronization natively without middleware, providing an efficient solution for real-time data replication and integration.

This release addresses performance and resource issues identified in the 1.0 beta1 release and introduces several new utilities for fine-tuning SynchDB's behavior and performance.

New Features

Performance Improvements

  • Implemented data caching in DML parsing stage to reduce PostgreSQL catalog access
  • Added throttle control in Debezium runner to prevent JVM memory buildup
  • Significantly increased processing speed by handling batches in single PostgreSQL transactions

New Functions and Views

  • synchdb_start_engine_bgw(name, mode): Added custom snapshot mode parameter
  • synchdb_log_jvm_meminfo(name): Debug function for JVM heap memory usage monitoring
  • synchdb_stats_view: New view for connector statistics
  • synchdb_reset_stats(name): Function to clear connector statistics

Configuration Enhancements

  • Added new GUCs for Debezium Runner performance tuning
  • Made JVM maximum heap memory allocation configurable
  • Enabled connector-specific SSL parameter configuration in rule files
  • Made maximum number of connector background workers configurable

Changes

  • Enhanced synchdb_state_view():
  • Added new 'stage' field showing connector status
  • Limited display to valid connectors only
  • Removed partial batch completion notifications
  • Updated SQLServer's default char type size mapping from 0 to -1

Bug Fixes

  • Resolved memory leaks in both SynchDB and Debezium runner components
  • Improved memory context handling in SynchDB
  • Corrected heap memory release during change event processing
  • Fixed SPI memory usage during DML processing

Known Issues

  1. Automatic connector launcher limitation:
  2. Only launches connector workers created under the default postgres database

  3. Issue tracking: github.com/Hornetlabs/synchdb/issues/71

  4. ALTER TABLE ALTER COLUMN limitations:

  5. Cannot handle complex data type changes (e.g., TEXT to INT)
  6. Does not support column index changes
  7. Cannot process renamed columns

  8. Issue tracking: github.com/Hornetlabs/synchdb/issues/77

  9. Connector state persistence:

  10. Restarting a paused connector resumes operation instead of maintaining paused state
  11. Issue tracking: github.com/Hornetlabs/synchdb/issues/80

Resources

About SynchDB

SynchDB is an open-source PostgreSQL extension developed by Hornetlabs Technology. It enables direct data replication from heterogeneous databases (MySQL, MS SQLServer, Oracle, etc.) to PostgreSQL with high performance and reliability. The extension handles all synchronization needs without requiring middleware or third-party software.