What is the best way to create a new dagster project if I also want to use poetry? I'd like to use

dagster project scaffold

, but this depends on dagster already being installed, which I want to manage with poetry. If I try creating a new project using poetry first (

poetry new project-name

), then the

pyproject.toml

will have missing config required by dagster. Thanks in advance for helping this noob out!

from @Odette Harary that shows how to make the poetry env first, installing dagster into there, and then scaffolding the project afterwards. Would this help?

Thanks, I have been following this but I'm still a little confused. If I follow this approach, then

poetry run dagster dev

will complain about my

pyproject.toml

not being configured properly.

Hmm, glad you figured it out, but I'm curious because I just tried it out and this sequence of commands worked for me:

poetry new test
cd test
poetry add dagster dagit
dagster project scaffold --name dagster-poetry-test
cd dagster-poetry-test
dagster dev

Granted, I can see how if you bump the dagster code back up to the poetry root directory, that error would happen

Thanks for the follow up here. Let me try your example and see what happens.

The scaffold doesn't do anything special, so if you move the generated Dagster project back up to the parent directory, update your

pyproject.toml

to reflect that

[tool.dagster]

block you mentioned, then you should have both the scaffold and poetry on a flat(ter) directory structure, matching more of what you intended

This is very helpful. The flatter directory structure is where I was running into trouble. I was able to follow your example and things are running ok. I'll keep experimenting with getting things working in my preferred flat project structure. Thanks again!

👋 Hi I was running into this same issue of wanted to set up Dagster and Poetry from a commong

pyproject.toml

file. The code you shared for the nested directory structure worked just fine. But every time I try to “flatten” the structure and pull everything into one

pyproject.toml

file I get this error:

Could not load location check_providers_pipeline to check for sensors due to the following error: dagster._core.errors.DagsterInvariantViolationError: No repositories, jobs, pipelines, graphs, or asset definitions found in "check_providers_pipeline".

All I did was add this section to the

pyproject.toml

file

[tool.dagster]
module_name = "check_providers_pipeline"

Then a

check_providers_pipeline/assets.py

file with a single generic asset function. I’ve tried several variations and still run into the same error. Googling around, I can’t find anyone else running into this same error. Any ideas what I could be doing wrong?