Hi all, I am attempting to process data synced fro...
# ask-community
y
Hi all, I am attempting to process data synced from Fivetran into an S3 data lake. However, I am stuck on how to proceed after setting up the fivetran assets, which seem to yield only the AssetKey as an output. Followed the example in the documentation. Received the following
Yielded output "schema_table_name" of type "Nothing".
STEP_FAILURE - Execution of step "fivetran_sync_connector_id" failed.
dagster._core.errors.DagsterExecutionHandleOutputError: Error occurred while handling output "schema_table_name" of step "fivetran_sync_connector_id"::
TypeError: 'NoneType' object is not subscriptable
schema_table_name
and
fivetran_sync_connector_id
have been renamed for this post, NoneType error is thrown because my IO manager is trying to output based on the previous step. The ideal flow should go like this: Fivetran syncs from DynamoDB to parquet files on S3, an IO manager should be able to pick up the newly created files and perform transformation steps. How do I trigger the downstream assets to process only the incrementally created files in S3?