The following is more of a design question: I'm f...
# ask-community
The following is more of a design question: I'm following along the example here (dagster-airbyte). I was able to get connection id (connection ids in my case). Now, I need to programmatically pass that connection id to an op. And, I'm wondering what's the best way to do so? That example ends with:
so you'll need to update
with this value.
But, ideally, I don't want to hard-code etc anything. My deployment is multi-container docker. Once I get connection id(s), I want to keep re-using those connections instead of creating new connection ids. So, I guess that setup code cannot go in a resource or op etc as a resource/op will be reinvoked/reinstantiated? In other words, in dagster pipeline, where is ideal place for a function (with a return value passable to an op) that needs to execute only once? A workaround that occurred to me is maybe that setup script can be invoked at docker-compose entry point (or something)
Copy code
python -m modern_data_stack_assets.setup_airbyte
and store that connection id somewhere (db, config etc)? I think there is a neater dagster's way to do so that I'm overlooking.