-
Notifications
You must be signed in to change notification settings - Fork 9
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add before/after columns, use GIN index, document Destination DB
- Loading branch information
Showing
10 changed files
with
164 additions
and
34 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
41 changes: 41 additions & 0 deletions
41
core/src/migrations/Migration20240208205640_before_and_after.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| import { Migration } from '@mikro-orm/migrations'; | ||
|
|
||
| export class Migration20240208205640_before_and_after extends Migration { | ||
|
|
||
| async up(): Promise<void> { | ||
| // before - new column | ||
| this.addSql('alter table "changes" add column "before" jsonb not null default \'{}\';'); | ||
| this.addSql('create index "changes_before_index" on "changes" using GIN ("before" jsonb_path_ops);'); | ||
|
|
||
| // after - rename | ||
| this.addSql('drop index "changes_values_index";'); | ||
| this.addSql('alter table "changes" rename column "values" to "after";'); | ||
| this.addSql('create index "changes_after_index" on "changes" using GIN ("after" jsonb_path_ops);'); | ||
|
|
||
| // context - new index | ||
| this.addSql('drop index "changes_context_index";'); | ||
| this.addSql('create index "changes_context_index" on "changes" using GIN ("context" jsonb_path_ops);'); | ||
|
|
||
| // unique constraint | ||
| this.addSql('alter table "changes" drop constraint "changes_position_database_schema_table_values_unique";'); | ||
| this.addSql('alter table "changes" add constraint "changes_position_operation_table_schema_database_unique" unique ("position", "operation", "table", "schema", "database");'); | ||
| } | ||
|
|
||
| async down(): Promise<void> { | ||
| // before - drop column | ||
| this.addSql('alter table "changes" drop column "before";'); | ||
|
|
||
| // after - rename | ||
| this.addSql('drop index "changes_after_index";'); | ||
| this.addSql('alter table "changes" rename column "after" to "values";'); | ||
| this.addSql('create index "changes_values_index" on "changes" ("values");'); | ||
|
|
||
| // context - new index | ||
| this.addSql('drop index "changes_context_index";'); | ||
| this.addSql('create index "changes_context_index" on "changes" ("context");'); | ||
|
|
||
| // unique constraint | ||
| this.addSql('alter table "changes" drop constraint "changes_position_operation_table_schema_database_unique";'); | ||
| this.addSql('alter table "changes" add constraint "changes_position_database_schema_table_values_unique" unique ("position", "database", "schema", "table", "values");'); | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| # Destination Database | ||
|
|
||
| Bemi automatically provisions a cloud-hosted PostgreSQL destination database to store all changes made in a source database. | ||
| You have full control over this database which comes with additional features: | ||
|
|
||
| * Autoscaling, managed table partitioning and index optimization to improve performance | ||
| * Automatic failover, read-replica, and backups for high availability and fault tolerance | ||
| * Automatic data retention: 15 days, 30 days, or unlimited | ||
| * Automatic PostgreSQL version upgrades | ||
| * Standard cloud support | ||
| * Control plane and monitoring through Bemi Dashboard (coming soon) | ||
|
|
||
| ## Data Structure | ||
|
|
||
| Changes performed by creating, updating, or deleting each row are stored in a table called `changes` and have the following structure: | ||
|
|
||
| | Column | Type | Description | | ||
| | ---------------- | ---------------- | ------------------------------------------------------------ | | ||
| | `id` | `uuid` | A unique identifier of the change record | | ||
| | `database` | `varchar(255)` | Database name where the changed record was stored | | ||
| | `schema` | `varchar(255)` | Schema name where the changed record was stored | | ||
| | `table` | `varchar(255)` | Table name where the changed record was stored | | ||
| | `primary_key` | `varchar(255)` | A unique identifier of the changed record (optional) | | ||
| | `operation` | `text` | Enum that can be either `CREATE`, `UPDATE`, or `DELETE` | | ||
| | `before` | `jsonb` | Record's values before the change | | ||
| | `after` | `jsonb` | Record's values after the change | | ||
| | `context` | `jsonb` | App context passed by using our recommended [ORM packages](/#supported-nodejs-orms) | | ||
| | `committed_at` | `timestamptz(0)` | When the record was changed | | ||
| | `queued_at` | `timestamptz(0)` | When the changed record was ingested from WAL | | ||
| | `created_at` | `timestamptz(0)` | When the change record was stored in the database | | ||
| | `transaction_id` | `integer` | PostgreSQL transaction ID | | ||
| | `position` | `bigint` | PostgreSQL WAL position | | ||
|
|
||
| ## Querying Changes | ||
|
|
||
| You can query changes by using our [ORM packages](/#supported-nodejs-orms) or by directly connecting and executing SQL queries. | ||
| For example, if you need to find when and how a user record with ID `b7267340-5011-40f4-ab9a-902b68fc5b25` had its email updated to `new@example.com` in the last 3 months: | ||
|
|
||
| ```sql | ||
| SELECT * | ||
| FROM "changes" | ||
| WHERE | ||
| "database" = 'postgres' AND "schema" = 'public' AND "table" = 'users' AND | ||
| "primary_key" = 'b7267340-5011-40f4-ab9a-902b68fc5b25' AND "operation" = 'UPDATE' AND | ||
| "after" @> '{"email": "new@example.com"}' AND NOT ("before" @> '{"email": "new@example.com"}') AND | ||
| "committed_at" BETWEEN (NOW() - INTERVAL '3 months') AND NOW() | ||
| LIMIT 1;` | ||
| ``` | ||
|
|
||
| The JSONB columns are indexed with [GIN Index](https://www.postgresql.org/docs/current/indexes-types.html#INDEXES-TYPES-GIN) with `jsonb_path_ops` operator class that is perfomance-optimized for operators like: | ||
|
|
||
| * `jsonb @> '{"key": value}'` to check if a key/value pair matches JSONB | ||
| * `jsonb @? '$.key'` to check if a key exists in JSONB |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters