Hi everyone, Need help regarding the location of dagster_home folder. We have used kubernetes for dagster deployment and mounted efs volume to our pod. We are trying to move dagster_home from container to efs volume but it keeps giving graphql error.

Hi Navneet - could you give more context on what changes you made that produced the error?

Hi @daniel Initially, dagster_home was inside /opt/dagster. We moved it to efs which is mounted on /home . We made this change to our Dockerfile.

ENV DAGSTER_HOME=/opt/dagster/dagster_home

to

ENV DAGSTER_HOME=/home/dagster_home/

Which Dockerfile are you referring to here? I believe the helm chart has a top level dagsterHome key that can be used to change this. I was under the impression that the helm chart handles creating all the dagster home management for you

We are using our customised docker file. could you please share if there is any reference for dockerfile.

I don't totally follow, sorry. Which part of the system is the dockerfile used for?

I see. You shouldn't need to set DAGSTER_HOME in any of your containers if you're using the helm chart - it is set by the helm chart using configuration in your values.yaml . I would recommend using the dagsterHome: key in the helm chart to customize your dagster home location, rather than setting it in individual dockerfiles

kk got it.

That error looks like the run pod created by the run launcher doesn’t have access to your dagster_home directory. What additional config does the pod need in order to access dagster_home with your new setup? The run launcher may need to be configured to apply that config to the pod that it creates

I just compared the dagster pod and the dagster run pod and got a few differences. I wanted your take on that.

These are the options we let you set in the K8sRunLauncher: https://docs.dagster.io/_apidocs/libraries/dagster-k8s#dagster_k8s.K8sRunLauncher 1. I think overriding service_account_name to your own service account would let you change the role? 2. env_vars or env_secrets sounds right fro this yes 3. The mount is the one i'm not sure about actually... we may need to expose volume_mounts as a config option on K8sRunLauncher. One workaround would be setting the volume mounts as a tag on the pipeline as described here: https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#solid-or-pipeline-kubernetes-configuration - but that would require putting a tag on every pipeline which you may not want to do. 4, changing config maps sounds like you need to use the env_config_maps field?

Hi @daniel, We have resolved all of our issues except one. We need to mount a PVC(EFS) on this dagster run pod as the pipeline and repository files are on the EFS itself. You have mentioned to use tags. Do you have a sample on how we can use tags to provide volume mounts? Also, you mention you are not sure so is there anyone else we can refer who can help us with this?

I'll check with a couple other members of the team tomorrow for what our recommendation is here for how to handle this. Upon digging in a bit more I don't think the tag solution I gave you will work - we don't have a good comprehensive way to handle all the different volume types that are out there currently, I'll see what other options we have.

Hi @daniel Gm! Hope you have had a good weekend. Please do update us when you have something for us on above issue.

Will do - it’s possible the solution will require waiting for a release, is there any short term workaround for you that doesn’t require mounting a custom volume in the pod for the launched run?

Not sure but we might have to write some code to read from S3 directly instead of EFS. Mount would have been awesome. In case we wait for release what do you think could be an approx time frame?

One possibility would be for you guys to fork or subclass the K8sRunLauncher while we come up with a fully supported volume solution, that would not depend on a release. The timing here is somewhat unfortunate since most of the dagster team is meeting together for an event this week so isn’t quite as responsive as normal It’s possible that the next release with a potential fix wouldn’t be until Thursday 9/30

It think the former will be a bigger change for us. We are ok if this comes in the 09/30 release.

Thanks @daniel once again, we will wait for this release from dagster. We will delay our release by couple of weeks to incorporate this change. Let us know how we can track this issue.

I filed a feature request issue here: https://github.com/dagster-io/dagster/issues/4889 I think we should be able to have a solution for a PVC by the release on 09/30, yes.

Hi @daniel we are really excited about this feature and looking forward this week for Dagster release. Give us a shout in case anything is needed by our side. Thanks,!!

Hi all, the change to add volume_mounts and volumes to the run launcher just landed, you can see the expected format in the PR: https://github.com/dagster-io/dagster/pull/4872

Cool 😎. Thanks Dagster team and @daniel. Looking forward for tomorrow's release.

Hi @daniel We deployed user-code-deployment but we are getting below error in dagit :

rpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:

status = StatusCode.UNAVAILABLE details = "no healthy upstream" debug_error_string = "{"created":"@1634031954.613875382","description":"Error received from peer ipv4:192.168.125.148:3030","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"no healthy upstream","grpc_status":14}" This our workspace.yaml and gateway yaml files in helm. in gateway we configured Istio things to access our dagit ui. could you please help us to resolve this error.

Hi Mohammed could you make a new post for this to ensure that the rest of the team sees it?

yeah sure.

I don’t off hand - somebody else on the team may though