Hi! I’m considering making the switch to ingestion...
# integration-airbyte
Hi! I’m considering making the switch to ingestion-as-code. How are people running the
dagster-airbyte check/apply
commands in their deployed containers? As part of their CI/CD? As a dagster job? Not really sure how best to get this set up so it can run on infrastructure with access to the env vars and db connections it needs
Also struggling with this one. Right now we have a manual deployment process, just for simplicity's sake. The fact it runs as a module from inside our Dagster project module means it needs access to all of the environment variables that the whole module requires, which is a bit of a bear. I don't think running it as a dagster job will ever work, since building the dagster environment will fail if your Airbyte connections are out of sync with their ingestion-as-code imported asset definitions. So your Airbyte instance has to be updated before your build step
Yeah I think you’re right about the job not working, that was poorly thought out by me on that one 😅 I’ll let you know what we end up going with if we get it going
Appreciate your insight, thanks for sharing ty spinny
You got it! It's funny, solving this exact problem is what I had on my roadmap for today. When you mentioned running it as a Dagster job I got very excited at first 😂. I'm thinking the best way to get it to work is to have a pre-build step with a custom container you install the module and it's required env variables into.
blob rofl 1
I think the big issue for me is that I can only access our warehouse and airbyte deployment from within our self-hosted infrastructure - so including it in a container build phase is a no go as far as I can tell crycat
Hi, how we can run apply from python code?
I am using Freshdesk as a source and Postgres as a destination. but I want to apply the configuration from Python, not from the command window.