Hey, I'm trying to play around with supplying the ...
# deployment-kubernetes
d
Hey, I'm trying to play around with supplying the stock value file to helm and getting this error
helm install dag dagster/dagster -f ./values.yaml --namespace dagster --create-namespace
Error: Deployment.apps "dag-dagit" is invalid: [spec.template.spec.containers[0].livenessProbe.tcpSocket: Forbidden: may not specify more than 1 handler type, spec.template.spec.containers[0].startupProbe.tcpSocket: Forbidden: may not specify more than 1 handler type]
r
Could you paste your values.yaml and the version of the dagster helm chart that you鈥檙e installing?
d
The values.yaml is the same as on master right now
Everything else is just would be the latest release
I managed to get it working if I use the helm chart in master
Tried to turn on user-code-deployments now getting this
Copy code
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1608773546.796629200","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":4142,"referenced_errors":[{"created":"@1608773546.796625800","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":397,"grpc_status":14}]}"
>
@rex
r
@David O'Keeffe you鈥檒l need to specify the
dagsterApiGrpcArgs
to allow dagit to communicate with the user code container
try supplying this as the override values, and it should work!
Copy code
userDeployments:
  # Whether launching user deployments is enabled.
  enabled: false
  # List of unique deployments
  deployments:
    - name: "k8s-example-user-code-1"
      image:
        repository: "dagster/k8s-example"
        tag: latest
        # Change with caution! If you're using a fixed tag for pipeline run images, changing the
        # image pull policy to anything other than "Always" will use a cached/stale image, which is
        # almost certainly not what you want.
        pullPolicy: Always
      # Arguments to `dagster api grpc`.
      # Ex: "dagster api grpc -m dagster_test.test_project.test_pipelines.repo -a define_demo_execution_repo"
      # would translate to:
      # dagsterApiGrpcArgs:
      #   - "-m"
      #   - "dagster_test.test_project.test_pipelines.repo"
      #   - "-a"
      #   - "define_demo_execution_repo"
      dagsterApiGrpcArgs:
        - "-f"
        - "/example_project/example_repo/repo.py"
      port: 3030
      env:
        YOUR_ENV_VAR: ""
      env_config_maps:
        - ""
      env_secrets:
        - ""
      nodeSelector: {}
      affinity: {}
      tolerations: []
      podSecurityContext: {}
      securityContext: {}
      resources: {}
      replicaCount: 1
This is the important part, since this is missing in the default helm values. But thanks for pointing this out, I鈥檒l make a change to include it as part of the default values
Copy code
dagsterApiGrpcArgs:
        - "-f"
        - "/example_project/example_repo/repo.py"
d
Ahh yes
I uncommented the other ones
r
https://dagster.phacility.com/D5765 will land this on master, thanks for reporting this 馃檪
d
@rex same error still
Running kube on docker desktop windows WSL2
Copy code
(base) 位 kubectl get pods --namespace dagster
NAME                                                         READY   STATUS    RESTARTS   AGE
my-release-celery-workers-dagster-6757f5595c-gdq9b           1/1     Running   0          3m17s
my-release-celery-workers-dagster-6757f5595c-gw55b           1/1     Running   0          3m17s
my-release-dagit-5986bc8c77-gfdzs                            1/1     Running   0          3m17s
my-release-dagster-daemon-5cd674845b-hcs6b                   0/1     Running   0          3m17s
my-release-dagster-k8s-example-user-code-1-867d4955f-2gbwl   1/1     Running   0          3m17s
my-release-postgresql-0                                      1/1     Running   0          3m17s
my-release-rabbitmq-0                                        1/1     Running   0          3m17s
Copy code
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1608774937.846691700","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":4142,"referenced_errors":[{"created":"@1608774937.846670000","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":397,"grpc_status":14}]}"
r
did you try reloading the repository location after reinstalling with the new helm values?
d
Hah no
It works now
馃檶馃徏 1
now this
Copy code
dagster.core.errors.DagsterInvariantViolationError: Cannot specify job_image in executor config when loading pipeline from GRPC server.

dagster.core.errors.DagsterInvariantViolationError: Cannot specify job_image in executor config when loading pipeline from GRPC server.

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/core/instance/__init__.py", line 1116, in submit_run
    run, external_pipeline=external_pipeline
,  File "/usr/local/lib/python3.7/site-packages/dagster/core/run_coordinator/default_run_coordinator.py", line 45, in submit_run
    return self._instance.launch_run(pipeline_run.run_id, external_pipeline)
,  File "/usr/local/lib/python3.7/site-packages/dagster/core/instance/__init__.py", line 1149, in launch_run
    self, run, external_pipeline=external_pipeline
,  File "/usr/local/lib/python3.7/site-packages/dagster_celery_k8s/launcher.py", line 178, in launch_run
    "Cannot specify job_image in executor config when loading pipeline "
Removed the job image from the config in the playground
The job is hung - daemon looks to be restarting over and over
Copy code
(base) 位 kubectl get pods --namespace dagster
NAME                                                         READY   STATUS    RESTARTS   AGE
my-release-celery-workers-dagster-6757f5595c-gdq9b           1/1     Running   0          11m
my-release-celery-workers-dagster-6757f5595c-gw55b           1/1     Running   0          11m
my-release-dagit-5986bc8c77-gfdzs                            1/1     Running   0          11m
my-release-dagster-daemon-5cd674845b-hcs6b                   0/1     Running   3          11m
my-release-dagster-k8s-example-user-code-1-867d4955f-2gbwl   1/1     Running   0          11m
my-release-postgresql-0                                      1/1     Running   0          11m
my-release-rabbitmq-0                                        1/1     Running   0          11
r
did you specify your own s3 bucket + prefix for intermediate storage? I鈥檓 not sure if this will work out of the box directly without those specified
issue was resolved by disabling celery and enabling the k8s run launcher instead - but opened up https://github.com/dagster-io/dagster/issues/3476 to track the issue around error observability when using celery in helm