Comment on page
S3
This page will go over how to configure and run our S3 destination.
To run our S3 connector, you will need to specify the following:
- 1.Parent bucket
- 3.Desired output format
- 4.(Optional) Prefix
Artie Transfer will save table data in this particular format:
/{{bucketName}}/{{optionalPrefix}}/{{fullyQualifiedTableName}}/{{YYYY-MM-DD}}
Example: /artie/foo/db.schema.tableName/2023-08-06
Upon each flush, there will be a new file created within this folder, the filename is:
{{unix_timestamp}}_{{randomString(4)}}.parquet.gz- Unix Timestamp is the latest timestamp of row processed
- Random string is created to allow parallelism
Artie Type | Parquet Type |
|---|---|
Float | Float |
Integer | Integer |
Numeric | DECIMAL(p,s) |
Boolean | Boolean |
String | String |
Struct | JSON string |
Array<any> | Array<string> |
Timestamp | Int64, Unix timestamp (in ms) |
Time | String |
Date | String |
Last modified 3mo ago