19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
#announcements
Title
e
Eric
01/25/2020, 5:48 AMCan someone help clarify the new Repository thing ? I'm getting very confused comparing the docs against the demos in the git repo. Throughout the beginning of the documentation it uses the regular "@pipeline" decorator with no parameters and can run dagit to view pipelines and yaml config files like normal.
However, when comparing to the demo files the pipelines have "mode_defs" and "preset_defs" predefined (also noticed they have names and other properties like resources and config). When defining a repo I would expect to be able to simply import and list pipelines just like the pipeline_defs in the pandas_hello_world demo. However, when dagit is run it's throwing errors saying the pipeline doesn't have a "name" property set when only using the plain "@pipeline" decorator like the documentation. This is strange since viewing the definition of the "@pipeline" decorator it states the "name" parameter is optional ? Also, what's the purpose of the lambas in the "pipeline_dict" dictionaries for the bay_bikes and airline demos ?
I understand what intent for Repositories and pipeline modes is for. However, during these early stages with simple tests I'm trying to avoid having to define local vs test vs prod pipeline modes. Perhaps a little clarification on which source to believe to be able to run dagit would be helpful.
Here is my repository.yaml:
Copy code
repository:
file: repo.py
fn: define_rsc_etl_reporepo.py:
Copy code
from pipelines.hello_world import hello_world_pipeline
from dagster import RepositoryDefinition
def define_rsc_etl_repo():
return RepositoryDefinition(
name='rsc_etl_repo',
pipeline_defs=[
hello_world_pipeline
]
# or this:
# pipeline_dict={
# 'hello_world_pipeline': lambda: hello_world_pipeline
# }
)And hello_world_pipeline.py:
Copy code
from dagster import execute_pipeline, pipeline, solid
@solid
def greet(context):
print("Hello World!")
@pipeline
def hello_world_pipeline():
greet()The "no name" attribute error message even though name is optional.
s
schrockn
01/25/2020, 4:42 PMSo this is because you are importing the hello_world_pipeline module and not the pipeline.
Copy code
from pipelines.hello_world.hello_world_pipeline
import hello_world_pipelinee
Eric
01/26/2020, 6:47 AMWhat a stupid mistake for such a long post. Thanks for pointing that out Nick. I'll file that issue.
s
schrockn
01/27/2020, 2:36 AMno worries! happens to all of us