https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
r

rowan gaffney

03/08/2022, 3:16 PM
Is there a value in the helm chart for launching jobs into a specific namespace (it is dafualting to the default namespace)? I am using the CeleryK8sRunLauncher
d

daniel

03/08/2022, 3:44 PM
Hi rowan - right now if you're using celery-k8s that needs to be set in the config for the celery_k8s_job_executor, but it can be set there in the job_namespace field (and then it will launch all its k8s pods in that namespace)
This is a reasonable request though and I filed an issue for it here: https://github.com/dagster-io/dagster/issues/6988
r

rowan gaffney

03/08/2022, 6:06 PM
Thanks @daniel. Would I set this via the celery_k8s_job_executor like:
Copy code
@job(executor_def=celery_k8s_job_executor,
     config={'execution':{'config':{'job_namespace':'my_namespace','image_pull_policy': 'Always'}}})
d

daniel

03/08/2022, 6:18 PM
I believe so yeah
r

rowan gaffney

03/08/2022, 7:30 PM
hmm, still trying to launch in the default namespace
d

daniel

03/08/2022, 7:35 PM
Huh, how strange - I see it in the code right here pulling from the executor config: https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-celery-k8s/dagster_celery_k8s/launcher.py#L221 What version of dagster is this - if there's a line like 'Creating Kubernetes run worker job' in your event log, does it say what namespace it's using?
And you're sure that the run config of the launched run includes the 'job_namespace': 'my_namespace' part?
r

rowan gaffney

03/09/2022, 4:03 PM
if i understand your question correctly, the answer is yes. I have dagster running in "my_namespace", but it is trying to launch the jobs in the "default" namespace.
Copy code
cannot create resource \"jobs\" in API group \"batch\" in the namespace \"default\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
My job looks like:
Copy code
from dataflow.ops.hello import hello
from dagster_celery_k8s.executor import celery_k8s_job_executor

@job(executor_def=celery_k8s_job_executor,
     config={'execution':{'config':{'job_namespace':'rgaffney','image_pull_policy': 'Always'}}})
def say_hello_job():
    hello()
And repo looks like:
Copy code
from dagster import repository
from dataflow.jobs.say_hello import say_hello_job

@repository
def dataflow():
    jobs = [say_hello_job]
    return jobs
d

daniel

03/09/2022, 4:05 PM
And if you click on the page for the run that was launched in the default namespace, and press the "view tags and config" button in the upper right - does it show that it ran with that config?
also what version of dagster is this?
r

rowan gaffney

03/09/2022, 4:15 PM
version: 0.14.1 the config field in dagit (via the "view tags and config") is empty... I am using the dagster-user-deployments via grpc to load this flow... I assume if I update the image and click "reload" that is all I need to do to reload the updated image/configs correct?
d

daniel

03/09/2022, 4:16 PM
when you say "update the image", what steps are you taking exactly?
that involves upgrading the helm chart right?
there's an open issue here that may be relevant: https://github.com/dagster-io/dagster/issues/6522 if you open a new tab in the launchpad after changing the default job config, does it show up? you'll want to be sure that config is shown in the launchpad before you launch
r

rowan gaffney

03/09/2022, 4:20 PM
the image that has the jobs, ops, and repo definition is updated and pushed to the container regestry, defined here - https://github.com/dagster-io/dagster/blob/e0dc13bd7b4a16c137e902e43056dc18b1b85f39/helm/dagster/values.yaml#L229. Then I can just click "reload" in dagit?
but perhaps I need to redeploy the dagster-user-deployments chart?
d

daniel

03/09/2022, 4:34 PM
Yeah, the deployment in that chart needs to restart, we have plans to make ‘’reload” in dagit do that for you, but it does not yet do so. You can also write that config in the launchpad though if you just want to test that it works, it doesn't have to be defined in code
r

rowan gaffney

03/09/2022, 6:44 PM
is it just the dagster-user-deployments that has to restart, or other parts of dagster too?
d

daniel

03/09/2022, 6:45 PM
just the dagster-user-deployments
r

rowan gaffney

03/09/2022, 7:10 PM
ok, with the redeployment, when I go to the launchpad the default space is missing the configs, but If I click "add" a new "new run" appears and the run_configs are there?
d

daniel

03/09/2022, 7:10 PM
I think that's https://github.com/dagster-io/dagster/issues/6522, yeah. I'll make sure that issue gets bubbled up and prioritized.
r

rowan gaffney

03/09/2022, 8:18 PM
👍 - gotcha. Thanks for you help on this @daniel
2 Views