https://dagster.io/ logo
#integration-airbyte
Title
# integration-airbyte
a

Arman S

03/24/2023, 10:38 AM
Hi guys, I am trying to use AIrbyte integration as code functionality and I understand that you need to do:
Copy code
dagster-airbyte check --module my_python_module.my_submodule:reconciler
dagster-airbyte apply --module my_python_module.my_submodule:reconciler
In order to check and apply changes. However, I would like to do this from within the code. Which is fine I can do
if not reconciler.check().is_empty(): reconciler.apply()
This works ok if you just run
dagit
But it breaks if you run
dagster dev
. Because the code is being executed both by
dagster
dev at the start time and
dagit
which causes memory leak. Is there a recommended way of doing this?
b

ben

03/27/2023, 3:13 PM
Hi Arman, I would not recommend running this code as part of loading your Dagster project since it will run the (potentially computationally expensive) reconciliation every time you start a run or reload your code location. I would instead recommend either running the CLI or a Python script to run
reconciler.apply()
as part of CI or as a standalone step.
1
🙏 1
2 Views