Hey folks I could use some help getting my project deployed dagster #ask-community

Hey folks, I could use some help getting my projec...

Nick James

08/21/2023, 8:01 PM

Hey folks, I could use some help getting my project deployed to my dagster serverless account. I’ve been following the tutorial setup with the hackernews examples and am trying to deploy it via the github integration, but the github action keeps failing. The logs from the action say

Error: buildx failed with: ERROR: failed to solve: process "/bin/sh -c dagster-cloud --version" did not complete successfully: exit code: 127

I’ve tried having dagster-cloud in both the setup.py and requirements.txt files, but it doesn’t seem to install the library when the action builds its container. Hoping it’s just a simple config I’m missing, and I’m pointing in the wrong place. Here’s my

dagster_cloud.yml

file

Copy code

locations:
  - location_name: swingleft_dagster
    code_source:
      package_name: swingleft_dagster

and here’s my setup.py inside the dagster location:

Copy code

from setuptools import find_packages, setup

setup(
    name="swingleft_dagster",
    packages=find_packages(exclude=["swingleft_dagster_tests"]),
    install_requires=[
        "dagster",
        "dagster-cloud",
    ],
    extras_require={"dev": ["dagster-webserver", "pytest"]},
)

and here’s the directory structure for the project (this is my DBT project that I’m adding dagster to, but just including the dagster stuff here for now as that’s not relevant yet)

Copy code

.
├── README.md
└── swingleft-dagster
    ├── README.md
    ├── dagster_cloud.yaml
    ├── analytics.hackernews
    ├── pyproject.toml
    ├── setup.cfg
    ├── setup.py
    ├── swingleft_dagster
    │   ├── __init__.py
    │   ├── assets.py
    │   ├── requirements.txt
    │   └── resources
    │       ├── __init__.py
    ├── swingleft_dagster.egg-info
    │   ├── PKG-INFO
    │   ├── SOURCES.txt
    │   ├── dependency_links.txt
    │   ├── requires.txt
    │   └── top_level.txt
    └── swingleft_dagster_tests
        ├── __init__.py
        └── test_assets.py

Nick James

08/21/2023, 8:02 PM

i have also tried using the dagster-cloud cli locally and run into the same error:

Copy code

=> ERROR [9/9] RUN if ! dagster-cloud --version; then         echo "Could not find the   0.3s
------
 > [9/9] RUN if ! dagster-cloud --version; then         echo "Could not find the dagster-cloud package.  Make sure you include the dagster-cloud package in your setup.py or requirements.txt file to ensure that it is installed as part of your workspace.";         exit 1;     fi:
#14 0.245 /bin/sh: 1: dagster-cloud: not found
#14 0.246 Could not find the dagster-cloud package.  Make sure you include the dagster-cloud package in your setup.py or requirements.txt file to ensure that it is installed as part of your workspace.

ben

08/21/2023, 8:46 PM

Hi Nick, is

dagster-cloud

also found in the

requrements.txt

Nick James

08/21/2023, 8:46 PM

yes. i’ve tried with and without it

Nick James

08/21/2023, 8:47 PM

i actually just rearranged to put the

dagster_cloud.yaml

file into the actual dagster project so it’s all in the same place

Nick James

08/21/2023, 8:48 PM

i realized i wasn’t correctly copying the folder structure in the docs. i believe it is correctly installing

dagster-cloud

now, but now I’m seeing this:

Copy code

Exception: Error in location config:
Must specify a field at path root:code_source if more than one field is defined. Defined 
fields: ['module_name', 'package_name', 'python_file']

Nick James

08/21/2023, 8:55 PM

i have updated the configs/directory structure shared in my original message to reflect the changes i’ve made. It feels like progress.

Shalabh Chaturvedi

08/21/2023, 9:16 PM

Hi Nick - can you share the github workflow you are using? Is it based on this one? https://github.com/dagster-io/dagster-cloud-action/blob/main/github/serverless/deploy.yml

Nick James

08/21/2023, 9:17 PM

yep that’s the one

Nick James

08/21/2023, 9:17 PM

lately I’ve been tyring to use the CLI to deploy just because I don’t want to merge all these changes into my main branch

Nick James

08/21/2023, 9:22 PM

with the dagster_cloud.yaml file moved down a directory, the dagster UI yells at me even after setting the path variable for it

Nick James

08/21/2023, 9:33 PM

if i move dagster_cloud file back up i run back into the error where it fails to install dagster-cloud module

Shalabh Chaturvedi

08/21/2023, 10:11 PM

You can keep

dagster_cloud.yaml

right next to the

setup.py

. If you want to deploy with the cli, can you try this command:

Copy code

cd swingleft-dagster

dagster-cloud serverless deploy-python-executable --location-file=dagster_cloud.yaml --location-name=swingleft_dagster

Open in Slack

Previous Next