Hamza Liaqat03/23/2022, 2:11 PM
so you'll need to updateBut, ideally, I don't want to hard-code etc anything. My deployment is multi-container docker. Once I get connection id(s), I want to keep re-using those connections instead of creating new connection ids. So, I guess that setup code cannot go in a resource or op etc as a resource/op will be reinvoked/reinstantiated? In other words, in dagster pipeline, where is ideal place for a function (with a return value passable to an op) that needs to execute only once? A workaround that occurred to me is maybe that setup script can be invoked at docker-compose entry point (or something)with this value.
and store that connection id somewhere (db, config etc)? I think there is a neater dagster's way to do so that I'm overlooking.
python -m modern_data_stack_assets.setup_airbyte