https://dagster.io/ logo
Powered by
Title
b

Bojan

01/26/2023, 11:24 AM
I think i’ve identified a bug with dagster-fivetran connector. When the table that is being synced returns 0 rows the asset will fail to get materialized in dagster. log from fivetran: 
"message" : "Completed extracting records for table: public.accounting_attributes. Total rows extracted: 0. Finished extraction for 4 out of 160 tables"
log on dagster: 
dagster._core.errors.DagsterStepOutputNotFoundError: Core compute for op "fivetran_sync_foo_id did not return an output for non-optional output "public_table_name_bar"
y

yuhan

01/27/2023, 5:41 PM
cc @ben
b

ben

01/27/2023, 6:06 PM
Hi Bojan, thanks for identifying this issue, let me see if I can replicate it
Would you be able to share which connector you are using & the snippet of your Dagster code you are using to generate the assets?
b

Bojan

01/30/2023, 9:12 AM
Sure thing Ben, sorry for a late reply. I wasn’t near a laptop over the weekend.
I think this is an issue with fivetrans api. Redacted the table names but this is pretty much what i do 
db = build_fivetran_assets(
    connector_id="sensibly_censor",
    asset_key_prefix=["fivetran", "snowflake", "db"],
    destination_tables=[
        "public.foo",
        "public.baz",
        "bar.boo",
    ],
)
the step that fails yields the following output. 
dagster._core.errors.DagsterStepOutputNotFoundError: Core compute for op "fivetran_sync_sensibly_censor" did not return an output for non-optional output "fivetran_snowflake_db_public_foo"
note that other tables will sync w/o issues
b

ben

01/30/2023, 10:49 PM
Hi Bojan, I believe this is being caused by Fivetran’s API not returning metadata for the table which has no rows synced in the API response. I’m going to add a config option to 
build_fivetran_assets
which will force Dagster to create an asset materialization for all tables listed in 
destination_tables
so that 
public.foo
will still be output
https://github.com/dagster-io/dagster/pull/11972
b

Bojan

01/31/2023, 12:46 PM
Thanks @ben! I’ve reached out to fivetran’s support around this, left their current official response. https://github.com/dagster-io/dagster/pull/11972#issuecomment-1410284396
@ben any chance this goes out in the next release or should i just fork and patch this for us until it goes out
b

ben

02/01/2023, 6:58 PM
Hi Bojan, I’ll try to get this merged for tomorrow’s release
❤️ 1
:next-level-daggy: 1
b

Bojan

02/01/2023, 7:22 PM
awesome thank you very much
@ben thanks for taking care of this, i’ve just tested it this morning and we’re good !
#dagster-supportJoin Slack
Powered by