hi, basic question as i'm having trouble getting s...
# dagster-plus
v
hi, basic question as i'm having trouble getting set up with dagster cloud. i have the following directory structure, where assets.py depends on
shared
and
lib
(and other libraries within `~/dev`:
Copy code
~/dev/
├─ shared/
├─ lib/
├─ dagster_project/
│  ├─ dagster_project/
│  │  ├─ assets.py
│  │  ├─ __init__.py
│  ├─ dagster_cloud.yaml
│  ├─ setup.py
when i try to deploy from within
~/dev/dagster_project
, it fails because of the missing dependencies on
shared
and
lib
; when i deploy from
~/dev
, it fails because it can't find the correct
dagster_project
module. what's the correct command or process here?
s
hey @Vanessa Cai there's a bit of CICD magic to do here, and a few different approaches one approach is to make the shared packages and lib packages available as installable dependencies, this approach is discussed here: https://github.com/dagster-io/dagster/discussions/12340 another approach is to copy those libraries into the dagster_project as part of the CICD process, so that they remain top-level shared libraries locally and in your Git repo, but become sub-libraries from Dagster's perspective during deployment. an example of this approach is the
ml_project
in this repo: https://github.com/slopp/dagteam/tree/main. As documented in the bottom of that repo's readme, the steps are: (a) deploy from ~/dev, but with
dagster_cloud.yaml
in the root
/dev
directory (b) use the
build
argument within
dagster_cloud.yaml
to point at dagster_project (c) have the CICD process copy shared and lib into the dagster_project, and then install those libraries using an appropriate reference in
setup.py
or a
dagster_project/requirements.txt
file
❤️ 1
v
thank you!