YH
05/30/2023, 4:19 AMYielded output "schema_table_name" of type "Nothing".
STEP_FAILURE - Execution of step "fivetran_sync_connector_id" failed.
dagster._core.errors.DagsterExecutionHandleOutputError: Error occurred while handling output "schema_table_name" of step "fivetran_sync_connector_id"::
TypeError: 'NoneType' object is not subscriptable
schema_table_name
and fivetran_sync_connector_id
have been renamed for this post, NoneType error is thrown because my IO manager is trying to output based on the previous step.
The ideal flow should go like this: Fivetran syncs from DynamoDB to parquet files on S3, an IO manager should be able to pick up the newly created files and perform transformation steps.
How do I trigger the downstream assets to process only the incrementally created files in S3?