https://dagster.io/ logo
#ask-community
Title
# ask-community
p

Pavlo Levin

09/04/2023, 1:23 PM
Hi! Have a problem with airbyte and auto-materialization. I have an airbyte asset with no other upstream or downstream assets and for this asset setted freshness policy and lazy auto-materialization policy. Most of the time all works correctly and auto-materialize job runs to meet the freshness policy but sometimes dagster launches a second auto-materializing job when the first run isn't finished yet, which obviously causes an error because airbyte job which was started by the previous run is still going on. Materialization conditions for both of these runs are the same "Required to meet this or downstream asset's freshness policy". What can cause such a strange scheduling?
Auto-materialize history page looks like this
And an asset definitions looks like this
Copy code
ronda_airbyte_assets = build_airbyte_assets(
    connection_id="b8628356-0bf3-4c58-b99c-b1ced92d11c0",
    destination_tables=[
        "applicant",
        "project",
        "webhook_data",
    ],
    asset_key_prefix=["dbt", "source", "dbt_analytics", "ronda"],
    group_name="ronda",
    freshness_policy=FreshnessPolicy(maximum_lag_minutes=60 * 2),
)
o

owen

09/08/2023, 9:45 PM
Hi @Pavlo Levin -- just confirming that we've seen this, and are working on a consistent way to reproduce this error (it seems to happen under some specific combination of conditions which are a bit hard to nail down)
just a quick question -- are there any assets downstream of your airbyte assets which have freshness policies?
p

Pavlo Levin

09/09/2023, 8:35 AM
Hi @owen No, there is nothing downstream for this airbyte connection
Also, I have a situation with another airbyte connection that has downstream assets. At the same time, this airbyte asset doesn't have a freshness policy and auto-materializes only because it's required by the downstream asset. First job was launched at 08:03 and succeeded at 08:17. During this time two another job was launched as you can see on screenshot below
o

owen

09/12/2023, 10:52 PM
Just confirming that we're still investigating this isssue
🤝 1