Comment on page

Examples

Introduction
Below, you can see sample configuration files to describe different workloads. To see all the available settings, please see the Options page.
Postgres options
Postgres --> Snowflake
outputSource: snowflake_stage
kafka:
  bootstrapServer: kafka:9092
  groupID: group_abc
  topicConfigs:
    - db: shop
      tableName: customers
      schema: inventory
      topic: "dbserver1.inventory.customers"
      cdcFormat: debezium.postgres.wal2json
      cdcKeyFormat: org.apache.kafka.connect.storage.StringConverter
snowflake:
  account: ACCOUNT_ID
  username: USER
  password: PASSWORD
  warehouse: DWH_NAME
  region: us-east-2.aws
Postgres --> BigQuery
outputSource: bigquery
kafka:
  bootstrapServer: kafka:9092
  groupID: group_abc
  topicConfigs:
    - db: shop
      tableName: customers
      schema: inventory
      topic: "dbserver1.inventory.customers"
      cdcFormat: debezium.postgres.wal2json
      cdcKeyFormat: org.apache.kafka.connect.storage.StringConverter
bigquery:
  pathToCredentials: PATH_TO_CREDENTIALS
  projectID: PROJECT_ID
  defaultDataset: DEFAULT_DATASET
MongoDB options
MongoDB --> Snowflake
outputSource: snowflake_stage
kafka:
  bootstrapServer: kafka:9092
  groupID: group_abc
  topicConfigs:
    - db: shop
      tableName: customers
      schema: inventory
      topic: "dbserver1.inventory.customers"
      cdcFormat: debezium.mongodb
      cdcKeyFormat: org.apache.kafka.connect.storage.StringConverter
snowflake:
  account: ACCOUNT_ID
  username: USER
  password: PASSWORD
  warehouse: DWH_NAME
  region: us-east-2.aws
MongoDB --> BigQuery
outputSource: bigquery
kafka:
  bootstrapServer: kafka:9092
  groupID: group_abc
  topicConfigs:
    - db: shop
      tableName: customers
      schema: inventory
      topic: "dbserver1.inventory.customers"
      cdcFormat: debezium.mongodb
      cdcKeyFormat: org.apache.kafka.connect.storage.StringConverter
bigquery:
  pathToCredentials: PATH_TO_CREDENTIALS
  projectID: PROJECT_ID
  defaultDataset: DEFAULT_DATASET
Optional Blocks
Error Reporting to Sentry
If you provide your Sentry DSN; Artie Transfer will automatically report any errors during processing into your Sentry project. 
Visit this link to see how you can find your Sentry DSN.
reporting:
  sentry:
    dsn: https://docs.sentry.io/product/sentry-basics/dsn-explainer/    
Transfer is using a vendor neutral logging library, so file a feature request if you use Rollbar, or another vendor! 
Telemetry
Visit the Telemetry page to see all the metrics that Transfer emits.
telemetry:
  metrics:
    provider: datadog
    settings:
      tags:
       - env:production
       - customer:artie.so
      namespace: "transfer."
      addr: "127.0.0.1:8125"
​

Last modified 4mo ago