19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
announcements
dagster-airbyte
dagster-airflow
dagster-bigquery
dagster-cloud
dagster-cube
dagster-dask
dagster-dbt
dagster-de
dagster-ecs
dagster-feedback
dagster-kubernetes
dagster-machine-learning
dagster-noteable
dagster-releases
dagster-serverless
dagster-showcase
dagster-snowflake
dagster-spatial
dagster-support
dagster-wandb
dagstereo
data-platform-design
events
faq-read-me-before-posting
gigs-freelance
github-discussions
introductions
jobs
random
tools
豆瓣酱帮
Title
n
Navneet Sajwan
10/25/2021, 6:47 PMHi,
I am using user-code-deployments. I want to specify executable path for each repository.
My workspace.yaml and helm configurations are in the images attached. Seems like it is not working, it is not using the environment mentioned in the executable_path fields.
d
daniel
10/25/2021, 7:20 PMHi Navneet - the way that we recommend doing this when each user code deployment is in its own pod is to use different images/Dockerfiles for different environments
n
Navneet Sajwan
10/26/2021, 8:48 AMWe are planning not to use separate dockerfiles/images for each repository thereby trying to make it dynamic.
We have mounted efs to maintain our repositories and virtual environments in it. This environment will be available to the repository in the EFS before its creation. Is there a way to pass a 'executable path' as an argument in helm?
If not, do you plan to have this feature in the future?
d
daniel
10/26/2021, 2:08 PMit's not currently - is this something you'd possibly be interested in sending a PR for?
I believe it would just involve changing this: https://sourcegraph.com/github.com/dagster-io/dagster/-/blob/helm/dagster/charts/dagster-user-deployments/templates/deployment-user.yaml?L43-44
To make the command "python -m dagster.api.grpc" instead of "dagster api grpc", and then making the "python" the default, but overridable by the deployment
m
Mohammad Nazeeruddin
10/26/2021, 5:24 PMHi @daniel
We Added this code in deployment_user.yaml and it's working fine. So I hope this is correct approach.
PythonEnvPath passing value : /home/orchestrator/system-env/bin/python
command: ["{{ $deployment.pythonEnvPath }}","-m","dagster"]
args: ["api", "grpc", "-h", "0.0.0.0", "-p", "{{ $deployment.port }}", "{{- join "\",\"" $deployment.dagsterApiGrpcArgs }}"]d
daniel
10/26/2021, 5:30 PMthat looks right to me! I'd happily review that PR if you want to send it out 🙂 your call.
👍 3
m
Mohammad Nazeeruddin
10/27/2021, 5:40 AMThank you @daniel
d
daniel
10/27/2021, 1:21 PMHmmm I just realized something though - it won’t be using that executable path in the pod that it creates when it goes to launch the run (if you’re using the K8sRunLauncher)
m
Mohammad Nazeeruddin
10/27/2021, 1:36 PMYes, it's not using executable path.
Hi @daniel,
When we are creating pods using K8sRunLauncher in pipeline execution time. it's showing pod > status : NotReady. But pipeline executing successfully in this k8s pods i checked logs everything looks fine. we are deploying user-code-deployment helm chart with Argocd tool in this tool the k8s pods still in progress state only.
and when i am executing same pipeline it's creating a new pod for each execution. these pods will be delete for after sometime ?
d
daniel
10/27/2021, 2:14 PMThese docs touch on the pod deletion / TTL question: https://docs.dagster.io/getting-started#kubernetes-job-and-pod-ttl-management
I'm not sure about the other question, would it be possible to make a new post so that the rest of the team can see it
👍 1
Still testing, but I believe https://github.com/dagster-io/dagster/pull/5415 may cause it to launch the run using the correct virtual environment
m
Mohammad Nazeeruddin
10/28/2021, 5:53 AMThank you @daniel We are Happy with your support to resolve our issues.😊
p
Prateek Agarwal
10/28/2021, 7:15 AMI think the link is broken @daniel. This is the correct link - https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment