https://dagster.io/ logo
#dagster-serverless
Title
# dagster-serverless
d

Dagster Jarred

08/14/2022, 5:23 PM
Adding @Todd de Quincey here to talk about the deployment error he's running into on Serverless cc: @ben @alex
t

Todd de Quincey

08/16/2022, 6:26 AM
Hi all, Thanks for starting the thread, Jarred. So the reproduction steps are as follows: 1. Created a local dagster project using
project_fully_featured
(Note: the primary branch is set as
master
by default) 2. Changed the primary branch to
main
3. No other changes made at all to the local repo 4. Pushed the code to github (personal account, as this is a quick PoC) 5. Created a Dagster Cloud account and using the serverless option, attempted to link Dagster Cloud to my repo 6. Noted that the
dagster_cloud.yaml
file was missing, so I copied this in from the Quickstart example 7. Updated the yaml file as per the below (I think this correct?) 8. Github actions run, but are failing with the below error 9. Thought that perhaps this was simply a config related issue, so I cloned the Quickstart repo and attempted to hook Dagster cloud up to it, but the Dagster cloud UI is throwing errors, and I now can’t change back to the old repo either
Copy code
locations:
  - location_name: midnite_dagster
    code_source:
      package_name: project_fully_featured
Copy code
#13 1.970 Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/site-packages (from jupyter-client>=6.1.5->nbclient<0.6.0,>=0.5.0->nbconvert->dagit==1.0.1->-r requirements.txt (line 2)) (23.2.1)
#13 1.972 Requirement already satisfied: tornado>=6.0 in /usr/local/lib/python3.8/site-packages (from jupyter-client>=6.1.5->nbclient<0.6.0,>=0.5.0->nbconvert->dagit==1.0.1->-r requirements.txt (line 2)) (6.2)
#13 2.612 Installing collected packages: dagster, dagster-graphql, dagit
#13 2.613   Attempting uninstall: dagster
#13 2.614     Found existing installation: dagster 1.0.3
#13 2.771     Uninstalling dagster-1.0.3:
#13 2.919       Successfully uninstalled dagster-1.0.3
#13 3.671   Attempting uninstall: dagster-graphql
#13 3.672     Found existing installation: dagster-graphql 1.0.3
#13 3.703     Uninstalling dagster-graphql-1.0.3:
#13 3.733       Successfully uninstalled dagster-graphql-1.0.3
#13 3.855   Attempting uninstall: dagit
#13 3.857     Found existing installation: dagit 1.0.3
#13 3.878     Uninstalling dagit-1.0.3:
#13 3.923       Successfully uninstalled dagit-1.0.3
#13 4.145 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
#13 4.145 dagster-spark 0.16.3 requires dagster==1.0.3, but you have dagster 1.0.1 which is incompatible.
#13 4.145 dagster-slack 0.16.3 requires dagster==1.0.3, but you have dagster 1.0.1 which is incompatible.
#13 4.145 dagster-pyspark 0.16.3 requires dagster==1.0.3, but you have dagster 1.0.1 which is incompatible.
#13 4.145 dagster-postgres 0.16.3 requires dagster==1.0.3, but you have dagster 1.0.1 which is incompatible.
#13 4.145 dagster-pandas 0.16.3 requires dagster==1.0.3, but you have dagster 1.0.1 which is incompatible.
#13 4.145 dagster-dbt 0.16.3 requires dagster==1.0.3, but you have dagster 1.0.1 which is incompatible.
#13 4.145 dagster-aws 0.16.3 requires dagster==1.0.3, but you have dagster 1.0.1 which is incompatible.
#13 4.146 Successfully installed dagit-1.0.1 dagster-1.0.1 dagster-graphql-1.0.1
#13 4.146 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: <https://pip.pypa.io/warnings/venv>
#13 DONE 4.4s

#14 [8/8] RUN dagster-cloud --version
#0 0.064 /bin/sh: 1: dagster-cloud: not found
#14 ERROR: process "/bin/sh -c dagster-cloud --version" did not complete successfully: exit code: 127
------
 > [8/8] RUN dagster-cloud --version:
#0 0.064 /bin/sh: 1: dagster-cloud: not found
------
Dockerfile:21
--------------------
  19 |     
  20 |     # Make sure dagster-cloud is installed. Fail early here if not.
  21 | >>> RUN dagster-cloud --version
  22 |     
  23 |     # placeholder section for adding env vars below
--------------------
error: failed to solve: process "/bin/sh -c dagster-cloud --version" did not complete successfully: exit code: 127
Error: buildx failed with: error: failed to solve: process "/bin/sh -c dagster-cloud --version" did not complete successfully: exit code: 127
When I attempt to change back to the original repo now, there is no way to do this. Just errors out
a

alex

08/16/2022, 2:30 PM
Created a local dagster project using project_fully_featured (Note: the primary branch is set as master by default)
This was with
dagster project from-example
I assume? As per the build output, the
dagster-cloud
dependency is not present so needs to get added to
setup.py
or a
requirements.txt
cc @yuhan
Thought that perhaps this was simply a config related issue, so I cloned the Quickstart repo and attempted to hook Dagster cloud up to it, but the Dagster cloud UI is throwing errors, and I now can’t change back to the old repo either
This we’ll have to look in to cc @ben
1
t

Todd de Quincey

08/16/2022, 3:36 PM
This was with
dagster project from-example
I assume?
Correct presumption!
9 Views