https://dagster.io/ logo
Join Slack
Powered by
Can someone help clarify the new Repository thing ...
# announcements
e
Can someone help clarify the new Repository thing ? I'm getting very confused comparing the docs against the demos in the git repo. Throughout the beginning of the documentation it uses the regular "@pipeline" decorator with no parameters and can run dagit to view pipelines and yaml config files like normal. However, when comparing to the demo files the pipelines have "mode_defs" and "preset_defs" predefined (also noticed they have names and other properties like resources and config). When defining a repo I would expect to be able to simply import and list pipelines just like the pipeline_defs in the pandas_hello_world demo. However, when dagit is run it's throwing errors saying the pipeline doesn't have a "name" property set when only using the plain "@pipeline" decorator like the documentation. This is strange since viewing the definition of the "@pipeline" decorator it states the "name" parameter is optional ? Also, what's the purpose of the lambas in the "pipeline_dict" dictionaries for the bay_bikes and airline demos ? I understand what intent for Repositories and pipeline modes is for. However, during these early stages with simple tests I'm trying to avoid having to define local vs test vs prod pipeline modes. Perhaps a little clarification on which source to believe to be able to run dagit would be helpful.
Here is my repository.yaml:
Copy code
repository:
  file: repo.py
  fn: define_rsc_etl_repo
repo.py:
Copy code
from pipelines.hello_world import hello_world_pipeline

from dagster import RepositoryDefinition

def define_rsc_etl_repo():
    return RepositoryDefinition(
        name='rsc_etl_repo',
        pipeline_defs=[
            hello_world_pipeline
        ]
        # or this:
        # pipeline_dict={
        #    'hello_world_pipeline': lambda: hello_world_pipeline
        # }
    )
And hello_world_pipeline.py:
Copy code
from dagster import execute_pipeline, pipeline, solid

@solid
def greet(context):
    print("Hello World!")

@pipeline
def hello_world_pipeline():
    greet()
The "no name" attribute error message even though name is optional.
s
So this is because you are importing the hello_world_pipeline module and not the pipeline.
Copy code
from pipelines.hello_world.hello_world_pipeline
 import hello_world_pipeline
should resolve it from looking at your error message However our error message should be better here. Do you mind filing an issue?
e
What a stupid mistake for such a long post. Thanks for pointing that out Nick. I'll file that issue.
s
no worries! happens to all of us
Open in Slack
PreviousNext
Powered by