https://dagster.io/ logo
#ask-community
Title
# ask-community
p

Philip Orlando

05/01/2023, 10:52 PM
What is the best way to create a new dagster project if I also want to use poetry? I'd like to use
dagster project scaffold
, but this depends on dagster already being installed, which I want to manage with poetry. If I try creating a new project using poetry first (
poetry new project-name
), then the
pyproject.toml
will have missing config required by dagster. Thanks in advance for helping this noob out!
2
t

Tim Castillo

05/01/2023, 10:58 PM

Here's a video example

from @Odette Harary that shows how to make the poetry env first, installing dagster into there, and then scaffolding the project afterwards. Would this help?
p

Philip Orlando

05/01/2023, 10:59 PM
Thanks, I have been following this but I'm still a little confused. If I follow this approach, then
poetry run dagster dev
will complain about my
pyproject.toml
not being configured properly.
Here's the specific error message that I'm seeing:
Adding
[tool.dagster]
to my
pyproject.toml
appears to have resolved the issue. Not sure if there is any additional config that I'm missing out on by not using
dagster project scaffold
though.
t

Tim Castillo

05/01/2023, 11:08 PM
Hmm, glad you figured it out, but I'm curious because I just tried it out and this sequence of commands worked for me:
Copy code
poetry new test
cd test
poetry add dagster dagit
dagster project scaffold --name dagster-poetry-test
cd dagster-poetry-test
dagster dev
Granted, I can see how if you bump the dagster code back up to the poetry root directory, that error would happen
p

Philip Orlando

05/01/2023, 11:10 PM
Thanks for the follow up here. Let me try your example and see what happens.
t

Tim Castillo

05/01/2023, 11:11 PM
The scaffold doesn't do anything special, so if you move the generated Dagster project back up to the parent directory, update your
pyproject.toml
to reflect that
[tool.dagster]
block you mentioned, then you should have both the scaffold and poetry on a flat(ter) directory structure, matching more of what you intended
2
p

Philip Orlando

05/01/2023, 11:13 PM
This is very helpful. The flatter directory structure is where I was running into trouble. I was able to follow your example and things are running ok. I'll keep experimenting with getting things working in my preferred flat project structure. Thanks again!
a

Alex V

05/26/2023, 3:52 AM
👋 Hi I was running into this same issue of wanted to set up Dagster and Poetry from a commong
pyproject.toml
file. The code you shared for the nested directory structure worked just fine. But every time I try to “flatten” the structure and pull everything into one
pyproject.toml
file I get this error:
Copy code
Could not load location check_providers_pipeline to check for sensors due to the following error: dagster._core.errors.DagsterInvariantViolationError: No repositories, jobs, pipelines, graphs, or asset definitions found in "check_providers_pipeline".
All I did was add this section to the
pyproject.toml
file
Copy code
[tool.dagster]
module_name = "check_providers_pipeline"
Then a
check_providers_pipeline/assets.py
file with a single generic asset function. I’ve tried several variations and still run into the same error. Googling around, I can’t find anyone else running into this same error. Any ideas what I could be doing wrong?
7 Views