There has been some discussion in an issue related to this here:
https://github.com/dagster-io/dagster/issues/19210
A developer from Dagster has commented saying it is on their roadmap to address in the upcoming months.
The general idea from users is to pivot to using terraform to stand everything up.