https://dagster.io/ logo
#announcements
Title
# announcements
g

Gaetan DELBART

06/18/2020, 12:30 PM
blob wave Hello ! First of all, thank you for your amazing work on dagster 0.8 (and thanks for the webinar BTW 😍) I've successfully migrated our pipelines to dagster 0.8 and also, I've updated the helm chart that is deployed to our k8s cluster. I've several remarks on the
dagster-k8s helm
. 1. In all the deployments, you use an init container using
image: postgres:9.6.16
But, in production envs, we use an external database, using postgres
11.6
so, I had to manually change that to
image: postgres:11.6
. Could we use a variable for the tag of this image ? 2. We don't use celery at all in our productions envs, and in the
Values.yaml
file it is possible to disable celery
Copy code
####################################################################################################
# Celery
####################################################################################################
celery:
  # The Celery workers can be deployed with a fixed image (no user code included)
  image:
    repository: ""
    tag: ""
    pullPolicy: Always

  enabled: true <- I've change this to false
So I added an
{{ if .Values.celery.enabled }}
in
deployment-celery.yaml
& to
deployment-celery-extras.yaml
to prevent deployment of celery from happening 3. Since we don't use celery, I had to change the
configmap-instance.yaml
, specifically the
run_launcher
part. In fact, the class
class: CeleryK8sRunLauncher
is hardcoded, and cannot be change, so I had to change this manually to
class: K8sRunLauncher
& tweek the config a bit to be able to run pipeline directly in k8s-job. Maybe we could have a section in the
values.yaml
to let the user choose what he want to use ? Finally, we use traefik as our cluster router. and I've added some template to the helm release, to add routes to the dagit ui. Would you be interested in a PR to implement traefik in your helm repository ?
kubernetes 2
🎉 2
❤️ 1
Also, let me know if you are interested for some PR about this, I will look into it 🙂
f

Fran Sanchez

06/18/2020, 1:18 PM
Hi Gaetan, I'd be interested to experiment with something like this, as I don't want to use Celery right now. Could you share a branch with these changes to try out? Cheers!
n

nate

06/18/2020, 3:27 PM
hi Gaetan, thanks for the feedback! some quick thoughts: For 1-3, yes all of these should be configurable, we’ll take a look and make these updates. For Traefik, what did you have to add to get this working? Is this just another resource we could put behind a boolean on the Helm chart?
r

Robert Lancer

06/18/2020, 4:38 PM
Could the Traefik part also be done with ingress annotations?
g

Gaetan DELBART

06/24/2020, 12:39 PM
Hey, sorry, I forgot to check this thread on dagster slack 🤦 I'll try to setup a repo with the changes I did on helm ! @nate Yes, I juste had to add some varabiels in the
values.yaml
=> In order to use traefik, I had to add some values in the
values.yaml
Copy code
traefik:
  enabled: true
  domain: <http://my-domain.com|my-domain.com>
  ui_auth: user-pwd
Then I had to add a IngressRoute ( custom ressource for traefik)
Copy code
{{- if .Values.traefik.enabled -}}
apiVersion: <http://traefik.containo.us/v1alpha1|traefik.containo.us/v1alpha1>
kind: IngressRoute
metadata:
  name: {{ template "dagster.fullname" . }}-dagit
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`dagit.{{ .Values.traefik.domain }}`) && PathPrefix(`/`)
      kind: Rule
      priority: 1
      middlewares:
        - name: traefik-auth
      services:
        - name: {{ include "dagster.dagit.fullname" . }}
          port: 80

{{end}}
And a middleware ( for basic auth)
Copy code
{{- if .Values.traefik.enabled -}}
apiVersion: <http://traefik.containo.us/v1alpha1|traefik.containo.us/v1alpha1>
kind: Middleware
metadata:
  name: traefik-auth
spec:
  basicAuth:
    secret: traefik-credentials
    removeHeader: true
{{end}}
@Robert Lancer Not sure about that, since traefik 2.0 use "Ingress Route" which are Custom Ressource Definiton
a

Adrian

01/07/2021, 6:57 PM
Did the PR for the helm release go out?
I don't see anything for traefik in
values.yml
4 Views