The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

for our instance we would need to have a fix for <https://github.com/dagster-io/dagster/issues/4861> . so i tried to get started (<https://github.com/Kapernikov/dagster/commit/a92b81a96a5452711144e479ec9d7324a7d2ec28>) but i'm not sure if i'm even remotely on the right track. the problem is that dagster uses alembic for table creation (but not always, sometimes it also uses sqlalchemy meta) and also that there is an abstraction layer.

• alembic migrations are supposed to be static, and best practises say you should include the schema in the migrations itself. but off course that's not feasible here as the schema should be configurable

• i don't think the schema support can be db-independent since not all dbs even support schemas

so i really wonder if its worth to continue the route i took

So, our alembic migrations aren’t really db-agnostic.   We have different alembic paths across our dbs, in part because we diverged early on and never reconciled the alembic hashes across mysql/postgres/sqlite.

I *think* you’re on the right path… we could alter all of our postgres-alembic scripts to take an optional schema argument into its ops.

in the end, we went for a much simpler solution, might be interesting for others too: in postgres you can create a specific user with a default schema. and other than i was expecting, when you do that, then dagster nicely creates all its stuff in that schema.