I think i ve identified a bug with dagster fivetran connecto dagster #ask-community

I think i’ve identified a bug with dagster-fivetra...

Bojan

01/26/2023, 11:24 AM

I think i’ve identified a bug with dagster-fivetran connector. When the table that is being synced returns 0 rows the asset will fail to get materialized in dagster. log from fivetran:

Copy code

"message" : "Completed extracting records for table: public.accounting_attributes. Total rows extracted: 0. Finished extraction for 4 out of 160 tables"

log on dagster:

Copy code

dagster._core.errors.DagsterStepOutputNotFoundError: Core compute for op "fivetran_sync_foo_id did not return an output for non-optional output "public_table_name_bar"

yuhan

01/27/2023, 5:41 PM

cc @ben

ben

01/27/2023, 6:06 PM

Hi Bojan, thanks for identifying this issue, let me see if I can replicate it

ben

01/27/2023, 6:35 PM

Would you be able to share which connector you are using & the snippet of your Dagster code you are using to generate the assets?

Bojan

01/30/2023, 9:12 AM

Sure thing Ben, sorry for a late reply. I wasn’t near a laptop over the weekend.

Bojan

01/30/2023, 9:14 AM

I think this is an issue with fivetrans api. Redacted the table names but this is pretty much what i do

Copy code

db = build_fivetran_assets(
    connector_id="sensibly_censor",
    asset_key_prefix=["fivetran", "snowflake", "db"],
    destination_tables=[
        "public.foo",
        "public.baz",
        "bar.boo",
    ],
)

Bojan

01/30/2023, 9:14 AM

the step that fails yields the following output.

Copy code

dagster._core.errors.DagsterStepOutputNotFoundError: Core compute for op "fivetran_sync_sensibly_censor" did not return an output for non-optional output "fivetran_snowflake_db_public_foo"

Bojan

01/30/2023, 9:37 AM

note that other tables will sync w/o issues

ben

01/30/2023, 10:49 PM

Hi Bojan, I believe this is being caused by Fivetran’s API not returning metadata for the table which has no rows synced in the API response. I’m going to add a config option to

build_fivetran_assets

which will force Dagster to create an asset materialization for all tables listed in

destination_tables

so that

public.foo

will still be output

ben

01/30/2023, 10:53 PM

https://github.com/dagster-io/dagster/pull/11972

Bojan

01/31/2023, 12:46 PM

Thanks @ben! I’ve reached out to fivetran’s support around this, left their current official response. https://github.com/dagster-io/dagster/pull/11972#issuecomment-1410284396

Bojan

02/01/2023, 8:46 AM

@ben any chance this goes out in the next release or should i just fork and patch this for us until it goes out

ben

02/01/2023, 6:58 PM

Hi Bojan, I’ll try to get this merged for tomorrow’s release

❤️ 1

👌 1

Bojan

02/01/2023, 7:22 PM

awesome thank you very much

Bojan

02/03/2023, 10:32 AM

@ben thanks for taking care of this, i’ve just tested it this morning and we’re good !

4 Views

Open in Slack

Previous Next