19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hi!
Have a problem with airbyte and auto-materiali...
# ask-communityp
Pavlo Levin
09/04/2023, 1:23 PMHi!
Have a problem with airbyte and auto-materialization. I have an airbyte asset with no other upstream or downstream assets and for this asset setted freshness policy and lazy auto-materialization policy. Most of the time all works correctly and auto-materialize job runs to meet the freshness policy but sometimes dagster launches a second auto-materializing job when the first run isn't finished yet, which obviously causes an error because airbyte job which was started by the previous run is still going on. Materialization conditions for both of these runs are the same "Required to meet this or downstream asset's freshness policy". What can cause such a strange scheduling?
Pavlo Levin
09/04/2023, 1:24 PMAuto-materialize history page looks like this
Pavlo Levin
09/04/2023, 1:25 PMAnd an asset definitions looks like this
Copy code
ronda_airbyte_assets = build_airbyte_assets(
connection_id="b8628356-0bf3-4c58-b99c-b1ced92d11c0",
destination_tables=[
"applicant",
"project",
"webhook_data",
],
asset_key_prefix=["dbt", "source", "dbt_analytics", "ronda"],
group_name="ronda",
freshness_policy=FreshnessPolicy(maximum_lag_minutes=60 * 2),
)o
owen
09/08/2023, 9:45 PMHi @Pavlo Levin -- just confirming that we've seen this, and are working on a consistent way to reproduce this error (it seems to happen under some specific combination of conditions which are a bit hard to nail down)
owen
09/08/2023, 11:20 PMjust a quick question -- are there any assets downstream of your airbyte assets which have freshness policies?
p
Pavlo Levin
09/09/2023, 8:35 AMHi @owen
No, there is nothing downstream for this airbyte connection
Pavlo Levin
09/09/2023, 8:37 AMAlso, I have a situation with another airbyte connection that has downstream assets. At the same time, this airbyte asset doesn't have a freshness policy and auto-materializes only because it's required by the downstream asset. First job was launched at 08:03 and succeeded at 08:17. During this time two another job was launched as you can see on screenshot below
o
owen
09/12/2023, 10:52 PMJust confirming that we're still investigating this isssue
🤝 1