Hi! I’m considering making the switch to ingestion-as-code. How are people running the

dagster-airbyte check/apply

commands in their deployed containers? As part of their CI/CD? As a dagster job? Not really sure how best to get this set up so it can run on infrastructure with access to the env vars and db connections it needs

Also struggling with this one. Right now we have a manual deployment process, just for simplicity's sake. The fact it runs as a module from inside our Dagster project module means it needs access to all of the environment variables that the whole module requires, which is a bit of a bear. I don't think running it as a dagster job will ever work, since building the dagster environment will fail if your Airbyte connections are out of sync with their ingestion-as-code imported asset definitions. So your Airbyte instance has to be updated before your build step

Yeah I think you’re right about the job not working, that was poorly thought out by me on that one 😅 I’ll let you know what we end up going with if we get it going

You got it! It's funny, solving this exact problem is what I had on my roadmap for today. When you mentioned running it as a Dagster job I got very excited at first 😂. I'm thinking the best way to get it to work is to have a pre-build step with a custom container you install the module and it's required env variables into.

I think the big issue for me is that I can only access our warehouse and airbyte deployment from within our self-hosted infrastructure - so including it in a container build phase is a no go as far as I can tell crycat

Hi, how we can run apply from python code?