0.3.0 (2025-12-18)

Release of Data.SyncMaster 0.3.0 brings up support for Iceberg, Spark-on-K8s and Spark-on-Yarn.

Note

Currently Spark-on-K8s and Spark-on-Yarn do not support FTP, FTPS, SFTP, Samba and WebDAV.

Breaking Changes

• Worker container command should be changed from --queues 123-myqueue to worker --queues 123-myqueue (#295).

• Application should be configured via config.yml file (#289).

  It's still possible to use environment variables instead. But it is not recommended for security reasons, as docker/k8s envs can be read by other users.

  Other notable changes:

  • Environment variable SYNCMASTER__ENTRYPOINT__SUPERUSERS is renamed to SYNCMASTER__SUPERUSERS.
  • Logging format is configured explicitly via config.yml instead of having few predefined configuration files.

• Moved server.session middleware settings to auth block (#304). Also rename some fields in auth.keycloak settings block.

  Before vs after

  Before:

  auth:
      provider: ...
      keycloak:
          server_url: ...
          redirect_url: ...
  
  server:
      session:
          enabled: true
          secret_key: ...

  Now:

  auth:
      provider:
      keycloak:
          api_url: ...
          ui_callback_url: ...
      cookie:
          secret_key: ...

Features

• Added Iceberg support (#282, #284, #294, #297).

  Iceberg connection currently supports only Iceberg REST Catalog with S3 warehouse.

• Allow using SyncMaster worker image as spark.kubernetes.container.image. (#295)

• Allow passing default Spark session config via worker settings (#291):

  Example

  worker:
      spark_session_default_config:
          spark.master: local
          spark.driver.host: 127.0.0.1
          spark.driver.bindAddress: 0.0.0.0
          spark.sql.pyspark.jvmStacktrace.enabled: true
          spark.ui.enabled: false

• Added OAuth2GatewayProvider (#283).

  This allows using Data.SyncMaster under OAuth2 Gateway. Implementation is similar to DummyAuthProvider.

• Allow disabling SessionMiddleware, as it only required by KeycloakAuthProvider.

• Add hooks support to worker classes (TransferController, Handler) (#279).

• Pass transfer name and group name to Handlers (#308).

Improvements

• Make S3 connection region a mandatory option, to prevent possible errors.
• Hide database_name from Clickhouse and MySQL connection pages.
• Frontend: add placeholders to connection params, like host, port and so on.
• Sync frontend and backend checks for some field patterns, e.g. table name should be in format schema.table.
• Improve OpenAPI schema fields description.

Bug Fixes

Fix some file format options were ignored by SyncMaster worker:

• XML: root_tag, row_tag
• Excel start_cell, include_header
• CSV include_header, line_sep
• JSON, JSONLine: line_sep

0.2.5 (2025-10-10)

Features

• Implement Keycloak login page on frontend. (#128)
• Implement GET /v1/auth/logout endpoint for KeycloakAuthProvider. (#275)

Improvements

• Improved full-text search for technical fields such as hostnames, table names, and directory paths. (#255)
• Replace 307 redirect to Keycloak auth page with 401 response, due to browser restrictions for redirect + CORS + localhost. (#274)

Bug Fixes

• Replace sync methods of Keycloak client with async ones. (#177)

  Previously interaction with Keycloak could block asyncio event loop.

0.2.4 (2025-07-09)

Improvements

Include all required jars from Maven to worker image. This increases image size, but drastically reduces time of Spark session startup.

0.2.3 (2025-04-11)

Bug fixes

• Fix Worker not updating Run status and ended_at fields after executing a very long ETL process.

Improvements

• Change Celery log level from DEBUG to INFO.

0.2.2 (2025-04-11)

Breaking

• Use PUT /v1/qroups/:id instead of PATCH /v1/qroups/:id.
• Use PUT /v1/qroups/:id/users/:id instead of PATCH /v1/qroups/:id/users/:id.
• Use PUT /v1/queues/:id instead of PATCH /v1/queues/:id.
• Now allowed names length should be in 3..128 symbols range, not 1..inf.

Improvements

• Now queue name can include any ASCII printable characters.
• Queue slug is always lowercase. Spaces, hyphens and underscores are replaced with - symbol.

Bug fixes

• Call kinit before starting Spark session conecting to Hive cluster. (#225)
• Fix HDFS connection was trying to use anonymous auth instead of user/password. (#225)
• Fix updating queue ignored name and didn't reset description.

0.2.1 (2025-04-07)

Improvements

• Change docker image user from root to syncmaster, to improve security.
• Move server healthcheck to Docker image.
• SBOM file is generated on release.

Bug fixes

• Fix missing Swagger docs in prod image.

0.2.0 (2025-04-04)

TL;DR

• Completely new UI.
• Add support for FileSystem connections.
• Add support for simple tranformations within transfer - filter files, filter rows, change columns.
• Add support for incremental read strategy.
• Add support for running transfers by schedule.
• Add support for changing SparkSession resource limits.

Huge thanks to @dmitry-pedchenko, @maxim-lixakov, @IlyasDevelopment, @Zabilsya.

Breaking Changes

• Implement a single error handling format to improve consistency (#95)
• Change response format for GET /v1/groups - add current user role for each group (#97)
• Change response format for GET /v1/groups/:id - add current user role for group (#109)
• Now migrations are executed in a dedicated one-off container, instead of being run as a part of backend container. (#163)
• Delete transfers, queues and groups records instead of marking them as deleted (#168)
• Move the type field from nested connection_data field to the root level of the Connection response. (#169)
• Decouple auth_data.type from the connection type (e.g. postgres) and link it to the authentication type (e.g. basic). (#169)
• Add deletion of connections records instead of marking them as deleted (#170)
• Use PUT instead of PATCH for Connection and Transfer models (#215)
• Return new connection and transfer object in POST /v1/transfer/copy and POST /v1/connection/copy endpoints.
• Change response status from 200 to 204 for all DELETE endpoints.

Features

• Add hive to known types (#67)
• Allow search for groups, users, connections, transfers, queues (#92, #94, #99, #100, #101, #103)
• Add filters for connections, transfers and runs (#94, #102, #106)
• Implement a scheduler to run celery tasks on a schedule. This can be done by setting Transfer.is_scheduled=True and Transfer.schedule="..." (cron-like expression). The Run model now has a type field with options MANUAL and SCHEDULED. (#114)
• Add GET v1/monitoring/metrics endpoint to provide basic HTTP server metrics in Prometheus format (#121)
• Implemented KeycloakAuthProvider for Single Sign-On (SSO) authentication.(#123)
• Implemented DummyAuthProvider for development and testing environments. (#123)
• Add API schemas for new DB sources - Clickhouse, MSSQL, MySQL (#124, #125, #126, #160)
• Add logic for handling FTP, FTPS, SFTP, Samba, WebDAV transfers (#189, #191, #192, #194)
• Add API schemas for file sources - SFTP, FTP, FTPS, WebDAV, Samba (#187)
• Add API schemas for file formats - Excel, XML, ORC, Parquet (#140, #142, #143, #144)
• Add compression options to file formats CSV, JSON, JSONLine, Excel, ORC, Parquet, XML (#159, #161)
• Add transformations for Transfers with dataframe row filtering (#184)
• Add transformations for Transfers with dataframe column filtering (#186)
• Add transformations for Transfers with file filtering (#198)
• Add increment_by field to strategy_params (#202)
• Implement increment strategy for transfers with file sources (#209)
• Implement increment strategy for transfers with database sources (#211)
• Add resources field to Transfer. (#214)
• Add file_name_template field to target_params (#196, #201)

Improvements

• Updated User model to include email, first_name, middle_name, and last_name fields, all optional. (#123)
• Read env variable SYNCMASTER__ENTRYPOINT__SUPERUSERS to promote users to SUPERUSER role during server startup. (#137)
• Enabled dynamic selection of authentication provider via environment variable SYNCMASTER__AUTH__PROVIDER. (#123)
• Enable parallel reading from JDBC sources. (#219)
• Reset HWM when changing strategy from incremental to full. (#217)
• Grant read-only permissions for the previous group owner when ownership is transferred (#135)

Bug Fixes

• Use Hadoop AWS magic commiter only if transfer target is S3. (#46)
• Check that service_name and sid are mutually exclusive when editing Oracle connection. (#52)
• Queue name is unique within a group, new field slug is globally-unique. (#54, #119)
• Prohibit updating connection type it if there is a transfer associated with this connection. (#55)
• Fix error when is_scheduled field value was ignored. (#57)
• Group without any users assigned was missing in groups list. (#62)
• Dump connection credentials while starting a transfer. (#63)

0.1.5 (2024-04-22)

Breaking Changes

• Pass current Run to CREATE_SPARK_SESSION_FUNCTION. This allows using run/transfer/group information for Spark session options, like appName or custom ones. (#38)

Improvements

• Reduce backend image size (#44)

Bug Fixes

• Fix 500 error in case of PATCH v1/connections/:id request with passed auth_data.password field value (#39)
• Do not use asyncio.gather with SQLAlchemy requests (#40)
• Fix 500 error while creating HDFS connection (#41)
• Fix missing options field from Transfer params with hdfs and s3 type

0.1.4 (2024-04-15)

Bug Fixes

• Fix missing backend factory for uvicorn.
• Fix missing kinit executable in worker image.

0.1.3 (2024-04-15)

Bug Fixes

Fix backend image entrypoint.