Comment on page
In this section, we will go over how to gather your credentials and prepare your PostgreSQL to start providing CDC logs.
This is necessary so that we are able to run a Debezium deployment to read from your PostgreSQL server's replication slot and publish them onto Kafka. To see additional configuration properties, please click here to see Debezium's documentation.
We will need the following:
Using Amazon RDS? RDS has its own internal permissioning model. Run this command instead of
ALTER USER REPLICATION!
GRANT rds_replication to username;
CREATE USER artie_transfer WITH PASSWORD 'password';
-- (optional) If the schema is not public, you will need this additional line
GRANT USAGE ON SCHEMA schema_name TO artie_transfer;
-- Grant read-only access to future tables
ALTER DEFAULT PRIVILEGES IN SCHEMA schema_name GRANT SELECT ON TABLES TO artie_transfer;
-- Grant access to existing tables
GRANT SELECT ON ALL TABLES IN SCHEMA schema_name TO artie_transfer;
-- The replication role does not have enough permissions to create publications.
-- So you will need to create this as well.
CREATE PUBLICATION dbz_publication FOR ALL TABLES;
-- Add the replication role to your user (not needed for Amazon RDS)
ALTER USER artie_transfer REPLICATION;
TIMESTAMP WITH TIME ZONE
TIME WITH TIME ZONE
INTERVAL [P], we store this in microseconds.
PostGIS data types
- More coming soon!
To set up your PostgreSQL database for CDC-based replication, you will need to enable replication slots. When this is done incorrectly, it could potentially cause a replication slot overflow and bring your production database down.
PostgreSQL Watcher provides additional guardrails around your database replication, and will do the following:
- 📊 Regularly check and monitor your replication slot size in 15-minute intervals and notify if the slot exceeds a certain threshold.
- 💓 Heartbeats verification. For folks that are leveraging Heartbeats, PostgreSQL Watcher will also check to make sure table permissions are updated and our service account has access to run Heartbeats. Watcher will notify you if the verification fails.
PostgreSQL Watcher is available to all Artie Cloud customers using PostgreSQL as a data source.
These considerations are automatically handled for you if Artie Transfer is running a Debezium connector for you.
We are also actively working on reducing the amount of considerations required to support every possible configuration.
- Debezium will automatically create a replication slot for you.
value.convertermust be set to
value.converter.schemas.enablemust be set to
Last modified 25d ago