https://dagster.io/ logo
#ask-community
Title
# ask-community
l

Laszlo Bencze

03/02/2023, 12:51 PM
Hi Team, We have already deployed the Dagster application into Kubernetes via Helm without any problem, however our client would like to use the whisper for managing the secrets in the K8s. The expected operation is that every pod creation would contain the init container spec that calls the whisper client and the container command would contains a source command that initialize the required environment variables. My assumption is that the K8sLuncher would contain this kind of information, right? However, we can give env variables in the user repository that’s why the user code helm chart would contains the init container tag in the spec. I have checked helm chart template and currently the user code spec does not provide opportunities for setting the init container and executing user defined commands. It would be very useful if we were able to handle this situation via helm? Would it be possible to implement this feature in the helm chart?
d

daniel

03/03/2023, 2:31 AM
Hi Laszlo - the part of this that i'm not sure about is the part where you override the container command. Right now you can override the docker ENTRYPOINT (what k8s calls the command) but dagster always injects the docker CMD with the command that it needs to set to launch the run (what k8s calls the args). Were you imagining here that you would override the ENTRYPOINT in some way where the dagster CMD (something like "dagster api execute-run" would still work?)
l

Laszlo Bencze

03/03/2023, 7:34 AM
Hi Daniel, I set the raw config in the k8sluncher this ways:
Copy code
runK8sConfig:
        containerConfig:
          - args: ['. path; . path]
I would thing that when a job is constructed, my custom args would be merged with the dagster api call. However, I can’t see my custom args in dagster-run pod only the dagster api execute-run. It is interesting, because the dagster api call contains my custom args. My assumption would be like similar one:
Copy code
args:
  - . path; . path
  - dagster
  - api
  - execute-run
  - etc
If I change the above to command I can override the container command, but the dagster call won’t be executed so it is not an option.
I see the attached picture in the main run’s pod args.
Let me summarize: 1, it is necessary to add init container spec in the user deployment helm chart -> feature 2, if I define custom args at level these args can’t appear in the pod’s args.
d

daniel

03/03/2023, 11:57 AM
Is “. path; . path” the actual thing you’re hoping to add to the front? I haven’t seen that syntax before - if it’s not do you have an example that’s closer to what you’re trying to do? Im just trying to verify whether prepending it to the front of the dagster commend will actually work
l

Laszlo Bencze

03/03/2023, 12:04 PM
sorry for my ambiguity. I would like to set env vars before the dagster execution, so . path -> source <path> .
d

daniel

03/03/2023, 12:11 PM
I'm still having a bit of trouble following. Any chance you could paste a full representative example of what you want to set it to?
l

Laszlo Bencze

03/03/2023, 12:27 PM
okey, the env is a k8s cluster. we would like to manage the secret via whisper in the cluster. We have to call a whisper client function in the init container part in order to reach the defined secret. This client gets the secret from the store and mount them in the predefined mount folder, for example /secret. In the folder, there will be a file that contains the secret settings for example export DB=sample_db. If my code uses DB as env vars this env vars has to be set before the usage. Hence I have to execute the source <path of the mounted file> .
Copy code
apiVersion: apps/v1
kind: Deployment
metadata:
  name: whisper-sample-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: whisper-sample-app
  strategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: whisper-sample-app
    spec:
      initContainers:
        - name: init-whisper-client
          imagePullPolicy: IfNotPresent
          # user and group should match securityContext setting
          command:
            - /bin/sh
            - -c
            - "whisper-client \
              --user 65534 \
              --group 65534 \
              --server <url> \
              --cache-file /secrets/whisper.cache \
              --certificate /tls-whisper/tls.crt \
              --private-key /tls-whisper/tls.key \
              --directory /secrets/app \
              --output-dir /secrets \
              --error-bubbling
              --namespace <namespace>"
          image: whisper-client/whisper-client:2.0.1
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          volumeMounts:
            # Secret will be cached here
            - mountPath: /secrets
              name: secrets
            # Client certificate will be loaded here
            - mountPath: /tls-whisper
              name: tls-whisper
              readOnly: true
      containers:
        - image: /library/busybox:1.33.0
          command: ['sh', '-c', 'source /secrets/app/postgres-creds; source /secrets/app/snowflake-creds; echo Container is Running ; env; sleep 360000']
          imagePullPolicy: IfNotPresent
          name: whisper-sample-app
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          volumeMounts:
            # Secret will be loaded here
            - mountPath: /secrets
              name: secrets
      volumes:
        - emptyDir: {}
          name: secrets
        - csi:
            driver: <name>
            readOnly: true
            volumeAttributes:
              []
          name: tls-whisper
      securityContext:
        runAsUser: 65534
        runAsNonRoot: true
        runAsGroup: 65534
        supplementalGroups:
          - 65534
        fsGroup: 65534
d

daniel

03/03/2023, 12:41 PM
OK, so you're imagining the args for the dagster run would end up something like
Copy code
args: ['sh', '-c', 'source /secrets/app/postgres-creds; source /secrets/app/snowflake-creds; dagster api execute-run ....']
Are you sure that setting the command to
['sh', '-c', 'source /secrets/app/postgres-creds; source /secrets/app/snowflake-creds;]
doesn't work? I was under the impression that it prepends the command (which you can set) to the args (which dagster replaces)
l

Laszlo Bencze

03/03/2023, 12:52 PM
yes, my imagenation is the first option. yes I tried to set the command, in this case, the command section was executed but the dagster was not called. If I add to the args these settings has not been merged.
d

daniel

03/03/2023, 1:01 PM
I wouldn't expect them to be merged, but I would expect to to run "command" + "args" together as a single command - i've seen people use that pattern before without issue so i'm a bit confused what's happening here
Do you have the raw pod spec available for the pod that you just screenshotted?
l

Laszlo Bencze

03/03/2023, 1:25 PM
d

daniel

03/03/2023, 1:35 PM
I think we might need to get a little more complex with the entrypoint here in order for this to work and pass through the arguments correctly - i.e. write your own shell script that runs the args like in the example here: https://dagster.slack.com/archives/C01U954MEER/p1677849763695939?thread_ts=1677845551.341539&amp;cid=C01U954MEER
I'll see if I can pull together an example - but that part at the end of the thread that Agon mentions where it runs "exec $@" is the part where it will correctly run the dagster args I think
and then there's the initContainers piece which it something we'll have to add support for in the Helm chart - we have a way to do it for runs but not for the user code deployments currently
l

Laszlo Bencze

03/03/2023, 1:52 PM
yes, I would like to add these custom settings via the k8sluncher’s runK8sConfig option. If I use this:
Copy code
containerConfig:
  command: ['sh', '-c', '. /secrets/app/postgres-creds; . /secrets/app/snowflake-creds;']
You see the out of pod template, it is in the command. If I use the args I won’t see in the args beside the dagster command.
yes initContainers in the user code helm chart would be very useful and be welcomed feature. But you will have to provide the possability to execute custom code in the command or args tag. for example:
Copy code
containers:
- args:
  command: ['sh', '-c', '. /secrets/app/postgres-creds; . /secrets/app/snowflake-creds; dagster api grpc ]
d

daniel

03/03/2023, 2:02 PM
I think what you’re going to need to do in the short term is writing a custom shell script to use as the command that correctly runs the args supplied by dagster in addition to the shell commands that you want to run - I can send an example later today
l

Laszlo Bencze

03/03/2023, 2:22 PM
thanks for your effort.:)
d

daniel

03/03/2023, 4:17 PM
I tried to answer this as a discussion here: https://github.com/dagster-io/dagster/discussions/12687 - So you could replace
Copy code
echo "Running additional command before CMD"
with your whisper-specific commands. Would that work here? (As an aside, I think in your examples you left out the
source
command? This isn't related to dagster, but shouldn't it be
source /secrets/app/postgres-creds
not just
. /secrets/app/postgres-creds`?` Maybe some syntax that i'm not familiar with)
2 Views