Comment on page
Artie's typing library
Curious how Artie's typing library works? You've come to the right place! Here, we will discuss how Artie's internal typing library works and how we ensure source-data integrity.
At a high level, Transfer will detect missing columns and automatically add them to the destination and set the correct types. When we get the first
NOT NULLvalue, we will run this through our typing library so that we can infer the correct data type. Transfer strives to maintain source-data integrity and as a result, we do not apply any transforms. The second objective with our typing library is also performance, as a slow typing library may incur additional overhead for row processing. We are happy to report that our typing library is 2x faster than Go's Reflect library.
Our schema conforms directly to the source (Postgres, MySQL and MongoDB), so we will set the data type (whether it be a FLOAT or INT) correctly to our destination.
What happens if the value is
The first one,
5, will be casted as a
INTand the second one will be casted a
A common question that we get is whether or not we apply any sort of transform such as JSON object flattening.
As part of our values to preserve data integrity, we will provide the JSON object back out to the destination in the way that it was received. See below for an example in Snowflake.
JSON object preservation in Snowflake example
How do you know if a value is a JSON object?
Our typing library will try to run values through our JSON parser and label this value as a JSON object if it passes our parsing test.
Arrays also have first-class support and we support the following:
- Normal arrays
- Nested arrays
Array with nested objects
Similar to JSON objects, if you do not want the Typing library to infer your string value as a
TIME, then simply pass the preferred data type as part of the optional schema. This is automatically supported with all of our sources.
Last modified 4mo ago