https://dagster.io/ logo
p

Peter B

04/14/2021, 1:42 PM
Hi, I wonder if anyone would taking a quick glance at this values.yaml helm file for k8 deploymnet per https://docs.dagster.io/deployment/guides/kubernetes/deploying-with-helm ? I am receiving this error on the cluster: Warning Unhealthy 17m (x3 over 17m) kubelet Startup probe failed: Get "<http://192.168.137.189:80/dagit_info%7Chttp://&lt;internal-Ip&gt;:80/dagit_info>": dial tcp <internal-ip>80 connect: connection refused All 3 pods stay in a crash loop. Is this due to an error in the .yaml configuration, or will it be k8 networking setup? Here are the high lvl details: • running on EKS/Fargate AWS on a VPC using AWS's std EKS VPC template • references an external postgres db • pulls user repo DockerFile from ECR (does this fine) • everything else defaults including ports (bar secrets that are setup). • I've removed data and put in brackets like <variable> where it was personal data Would really appreciate any help. It is my first time using Dagster. Thanks
a

alex

04/14/2021, 4:07 PM
are there no other logs for the dagit pod?
p

Peter B

04/14/2021, 11:33 PM
Hi, thanks for that. So, for Dagit, I get: dagster.core.errors.DagsterInstanceMigrationRequired: Instance is out of date and must be migrated (Postgres run storage requires migration). Database is at revision None, head is 7cba9eeaaf1d. Please run
dagster instance migrate
.
Original exception: Traceback (most recent call last): _File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1706, in _execute_context_ cursor, statement, parameters, context _File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 691, in do_execute_ cursor.execute(statement, parameters) psycopg2.errors.InvalidSchemaName: no schema has been selected to create in _LINE 2: CREATE TABLE secondary_indexes (_ ^ However, there is no option to specify a postgres schema in Dagster? And there doesn't appear to be a way to specify a PG connection string (where one could override the default postgres schema search path) in values.yaml, in contrast to non-k8 dagster setups ? Also - how can one run 'dagster instance migrate' (per the above) , whilst the container is crashing? ---------------------------------------------------------__ Then, for User Code repo, received this one (re. Celery). It looks straightforward, but in the documentation (see screenshot), it says to use the celery image (rather than dagster/dagster-k8s) even though it is the 'non-celery version' vs. what's pictured in the 'Helm Deployment - Advanced Version'. Therefore could I please confirm if I should instead be using the 'dagster/dagster-k8s' docker.io image if I don't need the parallel celery workers? https://docs.dagster.io/deployment/guides/kubernetes/deploying-with-helm Thanks heaps!!!!!!!! File "/usr/local/lib/python3.8/dist-packages/dagster/core/code_pointer.py", line 109, in load_python_file raise DagsterImportError( dagster.core.errors.DagsterImportError: Encountered ImportError:
No module named 'dagster_celery_k8s'
while importing module repo from file /lookout_repo/repo.py. Local modules were resolved using the working directory
/
. If another working directory should be used, please explicitly specify the appropriate path using the
-d
or
--working-directory
for CLI based targets or the
working_directory
configuration option for
python_file
-based workspace.yaml targets.
a

alex

04/15/2021, 2:43 PM
cc @prha since i would not expect this error to block dagit starting
how can one run ‘dagster instance migrate’
https://docs.dagster.io/deployment/guides/kubernetes/how-to-migrate-your-instance
could I please confirm if I should instead be using the ‘dagster/dagster-k8s’ docker.io image if I don’t need the parallel celery workers?
this is correct
3 Views