https://dagster.io/ logo
Powered by
Title
m

Mandi Alexander

01/05/2023, 9:13 PM
I'm working on a POC for Dagster and I keep running into the following error: 
% helm install dagster dagster/dagster               
Error: INSTALLATION FAILED: values don't meet the specifications of the schema(s) in the following chart(s):
dagster:
invalid character '<' looking for beginning of valuedagster-user-deployments:
invalid character '<' looking for beginning of value
I've tried using a local copy of 
values.yaml
. I've also tried different versions of the helm chart, but I get the same error. My best guess is that it's reaching out and getting a bad http response. Does anybody know what's causing this?
:dagster-bot-resolve: 1
I've been following this: https://docs.dagster.io/deployment/guides/kubernetes/deploying-with-helm The only part I didn't set up was S3. It doesn't seem like that would give this particular error at 
helm install
, but please let me know if I'm wrong.
(Actually, looks like the layout of that guide has changed since the last time I tried. I'll have to see if any of the content has changed...)
d

daniel

01/06/2023, 4:59 AM
Hi Mandi - is it possible to share a 
values.yaml
file that reproduces the error that you're seeing?
thanks - and what's the helm command that you're running to use it?
oh, you had it above - it's the helm install, looks like
Does the same error happen when you run this command? 
helm upgrade --install dagster dagster/dagster -f values.yaml
(in the folder where the valyues.yaml is)
m

Mandi Alexander

01/06/2023, 4:29 PM
It does.
I'm going through the updated installation doc now...
d

daniel

01/06/2023, 4:30 PM
That's very strange, what version of helm do you have installed?
I assume you did the step where you add the helm repo: 
helm repo add dagster <https://dagster-io.github.io/helm>
m

Mandi Alexander

01/06/2023, 4:31 PM
v3.10.3
dagster helm version: 1.1.7
A coworker tried it on our production cluster and got the same result. 🤷
d

daniel

01/06/2023, 4:34 PM
Have you been able to install any other helm charts in this environment? It seems like it might be having trouble finding the helm schema to check
that "invalid character '<' looking for beginning of value" might be the start of some kind of error message when it tries to fetch the helm schema
💯 1
m

Mandi Alexander

01/06/2023, 4:35 PM
This is the only helm I've tried locally. I know we've installed other charts on our production cluster though.
If you know a good chart to test with I'm happy to try it.
Otherwise I'll poke around for one.
I'm pretty motivated to solve this. 😆
d

daniel

01/06/2023, 4:37 PM
What if you run "helm template dagster dagster/dagster -f values.yaml" instead? same error?
m

Mandi Alexander

01/06/2023, 4:42 PM
Same error.
I'm able to install the bitnami/spark helm chart.
(dagster_venv) mandi.alexander@VU-C02FT2ELML85 helm_test % helm install myspark bitnami/spark
NAME: myspark
LAST DEPLOYED: Fri Jan  6 10:45:14 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: spark
CHART VERSION: 6.3.13
APP VERSION: 3.3.1
d

daniel

01/06/2023, 4:46 PM
I know you said its happening in prod too, but where is your local k8s cluster running?
docker-desktop?
m

Mandi Alexander

01/06/2023, 4:46 PM
(dagster_venv) helm_test % helm install myspark bitnami/spark
NAME: myspark
LAST DEPLOYED: Fri Jan  6 10:45:14 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: spark
CHART VERSION: 6.3.13
APP VERSION: 3.3.1
Yes, Docker desktop on a Mac.
d

daniel

01/06/2023, 4:47 PM
and what k8s version?
trying it myself on docker desktop to see if i can reproduce with that values.yaml file
m

Mandi Alexander

01/06/2023, 4:51 PM
You'll have to forgive my ignorance--I'm new to Docker and Kubernetes. 
kubectl version
gives me this: 
Client Version: v1.25.2
Kustomize Version: v4.5.7
Server Version: v1.25.2
d

daniel

01/06/2023, 4:51 PM
no problem, that works
m

Mandi Alexander

01/06/2023, 4:51 PM
docker version
gives me this: 
Client:
 Cloud integration: v1.0.29
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        baeda1f
 Built:             Tue Oct 25 18:01:18 2022
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.15.0 (93002)
 Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208
  Built:            Tue Oct 25 18:00:19 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.10
  GitCommit:        770bd0108c32f3fb5c73ae1264f7e503fe7b2661
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
I have to step away for a bit, but running through the current deploy guide in a new directory gives me the same result.
d

daniel

01/06/2023, 5:01 PM
very odd... this is what I get when I run `helm template dagster/dagster on your values.yaml file`: https://gist.github.com/gibsondan/3fbd922d3571c7b0695d0787285352f3
🤔 1
helm template
doesn't even need a cluster to be running, so I don't think its related to your kubernetes cluster
m

Mandi Alexander

01/06/2023, 5:03 PM
I might try setting it up on my personal computer when I get back. Maybe it's something to do with the vpn or our security setup.
d

daniel

01/06/2023, 5:08 PM
the other thing you could try is some other chart that uses a schema for validation (I know bitnami/rabbitmq does for example)
that would rule out some general issue related to helm + schema validation in your environment
m

Mandi Alexander

01/06/2023, 6:41 PM
bitnami/rabbitmq seems to install ok: 
% helm install myrabbit bitnami/rabbitmq
NAME: myrabbit
LAST DEPLOYED: Fri Jan  6 12:40:13 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: rabbitmq
CHART VERSION: 11.3.1
APP VERSION: 3.11.6** Please be patient while the chart is being deployed **

Credentials:
    echo "Username      : user"
    echo "Password      : $(kubectl get secret --namespace default myrabbit-rabbitmq -o jsonpath="{.data.rabbitmq-password}" | base64 -d)"
    echo "ErLang Cookie : $(kubectl get secret --namespace default myrabbit-rabbitmq -o jsonpath="{.data.rabbitmq-erlang-cookie}" | base64 -d)"

Note that the credentials are saved in persistent volume claims and will not be changed upon upgrade or reinstallation unless the persistent volume claim has been deleted. If this is not the first installation of this chart, the credentials may not be valid.
This is applicable when no passwords are set and therefore the random password is autogenerated. In case of using a fixed password, you should specify it when upgrading.
More information about the credentials may be found at <https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/#credential-errors-while-upgrading-chart-releases>.

RabbitMQ can be accessed within the cluster on port 5672 at myrabbit-rabbitmq.default.svc.cluster.local

To access for outside the cluster, perform the following steps:

To Access the RabbitMQ AMQP port:

    echo "URL : <amqp://127.0.0.1:5672/>"
    kubectl port-forward --namespace default svc/myrabbit-rabbitmq 5672:5672

To Access the RabbitMQ Management interface:

    echo "URL : <http://127.0.0.1:15672/>"
    kubectl port-forward --namespace default svc/myrabbit-rabbitmq 15672:15672
I'm working on setting up my other computer between meetings. I'll update when I have more info. Thanks for all the help so far!
Yeah... it works on my private machine.
We found the blocker and got it working! 🎉 Thanks for all the help!
d

daniel

01/09/2023, 9:16 PM
nice! curious what it ended up beign?
m

Mandi Alexander

01/09/2023, 9:19 PM
A TLD block. So the process was getting an html error message when it tried to reach out to kubernetesjsonschema.dev
d

daniel

01/09/2023, 9:27 PM
Ahhh I see
#dagster-supportJoin Slack
Powered by