Hi all! I'm using Dagster in a GKE kubernetes environment with user deployments and just started looking into schedulers.
What would be a good way of filling solid (or resources) configurations for schedules through k8s configmaps?
Would I need some sort of factory for my schedule?
03/15/2021, 4:07 PM
you'd need some schedule factory to query your configmaps and convert them into schedule config for a pipeline
what's your use case for this? why not just set your schedule/solid configuration in your user deployment?
03/15/2021, 4:13 PM
do you mean into the user deployment section of the helm chart? (we're using helm to deploy)
03/15/2021, 4:14 PM
i mean in your pipeline code, in the image that your user deployment is using
03/15/2021, 4:15 PM
I would prefer not to have separate docker images for different environments (test, production, etc), to separate config from code
I could probably specify parameters through env vars / envConfigMaps .. and then have the parameters picked up during deployment and
03/15/2021, 4:17 PM
you can have all your environments in one docker image, using modes
maybe that's a solution. My usual approach would still be to separate config from docker image though .. let's say dagster just triggets a service exposed on an internal http endpoint running on the same k8s cluster, and you want to change the url. In an ideal scenario you would just update a config rather than re-building a docker image.
On the other hand I understand that rebuilding a docker image for the user deployment is a quite common task with dagster anyways when adjusting pipelines etc, so the overhead is not as big