Hi, I am new to dagster and are just getting started. When trying to follow along with the https://docs.dagster.io/examples/airline_demo airline tutorial I observe 2 problems: 1)

pip install -e .

is not installing the optional dependencies which are needed to run the tutorial. 2) https://github.com/dagster-io/dagster/tree/master/examples/airline_demo/airline_demo does not contain the

repository.yaml

file and thus executing a call to

dagit

fails. How can I get the tutorial to work? (for (1) I guess I have installed the necessary dependencies manually, but so far am stuck on (2).

Which repository.yaml do you mean?

Well this is the error I get:

dagit
on_any_event(self=<dagit.dagit.DagsterAutoRestartTrick object at 0x7fe130193c70>, event=<DirCreatedEvent: event_type=created, src_path='/private/var/folders/cx/tptc40fs3492df12b62dx_kr0000gn/T/tmp720xpex5', is_directory=True>)
on_any_event(self=<dagit.dagit.DagsterAutoRestartTrick object at 0x7fe130193c70>, event=<DirModifiedEvent: event_type=modified, src_path='/private/var/folders/cx/tptc40fs3492df12b62dx_kr0000gn/T', is_directory=True>)
Usage: dagit-cli [OPTIONS]

Error: Expected to use file "repository.yaml" to load repository but it does not exist. Verify your current working directory or CLI arguments.

What command are you using to start dagit?

dagit

simply as outlined in the tutorial - dagit only.

Try

dagit -w workspace.yaml

?

fails with:

Error: no such option: -w

Erm, did it get renamed when I wasn't looking? 🙂

dagit, version 0.6.9

Oh, that's very very old

but this is what I get from conda

I know - also check this. Strange.

Also, this tutorial doesn't use conda

😉 sure. But I install it into an empty conda env.

Sure, but conda isn't installing the packages

I have made the following observations:

conda create -y --name dagster-demo python=3.8
conda activate dagster-demo

pip install -e .

is not installing dagit. When using conda instead:

channels:
  - conda-forge
  - defaults
dependencies:
  - dagster
  - dagit
  - python>=3.8
  - pip

now I get 0.10.8 But it fails as:

ModuleNotFoundError: No module named 'dagster_aws'

is missing. Now when trying to fix it :

pip install -e .

it still fails as the optional dependencies (which seem to be required for this tutorial are not installed). When instead using a larger conda

environment.yml

file with contents of:

name: dagster-demo
channels:
  - conda-forge
  - defaults
dependencies:
  - dagster
  - dagit
  - python>=3.8
  - pip
  - dagstermill
  - dagster-aws
  - dagster-cron
  - dagster-postgres
  - dagster-pyspark
  - dagster-slack
  - dagster-snowflake
  - descartes
  - geopandas
  - seaborn
  - pandas
  - pyspark
  - sqlalchemy
  - sqlalchemy-redshift
  - sqlalchemy-utils

I get the outdated version:

0.6.9

I would recommend you to ditch conda. It does too many magic behind the scenes, and it doesn't play nice with other tools. But you could try installing dagster via pip instead of conda. It might pull the latest version from pypi

As you see, I have tried this. However,

pip install -e .

fails to install the necessary required (but marked as optional) dependencies. How could I get these installed as well?

Could you try in a new environment?

id did so already using:

conda create -y --name dagster-demo python=3.8

Hmmm I think airplane tutorial is a bit too complex. You could try the cereal example instead. It's under the tutorial section of the docs. It doesn't need extra depedencies aside from dagster and dagit I think

I know - but I already took a look there and would love to get the airline example going now.

I have this one already running.

https://docs.dagster.io/examples/airline_demo do you have all the pre reqs as stated here?

yes. But the optional dependencies i.e.

dagster-aws

and all the others are not installed when executing `

pip install -e .

If all else fails you could try using docker compose

do you have a link for that? Maybe this is indeed simpler.

Theres a link to gh repo from the docs link above

sure - but the docker-compose from there only starts postgres and not dagster.

Oh I didn't read it carefully. You keep it open tho for db instance.

this one https://github.com/dagster-io/dagster/tree/master/examples/airline_demo

good news is you’re not imagining things. the example is indeed outdated: https://github.com/dagster-io/dagster/issues/3577

many thanks.

altho on my machine it seems that I have to install

dagster_aws

manually. it then asks for

dagster_pyspark, dagstermill

and so on…..

awesome.

From looking at the code, you need

pip install -e '.[full]'

I think