https://dagster.io/ logo
#announcements
Title
# announcements
b

Basil V

03/25/2020, 6:50 PM
I found the docs at: https://docs.dagster.io/latest/api/apidocs/types/#dagster.Nothing this should be what I need I think? Thanks so much!
p

prha

03/25/2020, 6:51 PM
You could use a
Nothing
type as Nick suggests, or you could pass a newly constructed object with the table name and maybe the load time or something like that
b

Basil V

03/25/2020, 6:52 PM
Ok awesome—yeah that's what I initially had in mind was to pass some "dummy" value like the table name between them, but I was curious if you guys had a convention for this kind of scenario. It sounds like Nothing is the way to go.
Thanks
d

dwall

03/25/2020, 6:53 PM
yeah we do this very often in our own project
something like:
Copy code
@solid(
    description="A solid.",
    input_defs=[InputDefinition(name="start_after", dagster_type=Nothing)]
)
def solid(context):
    pass

...

@pipeline
def pipeline():
    solid(start_after=[other_solid(), another_solid()])
👍 1
p

prha

03/25/2020, 6:53 PM
Yup! FWIW, I tend to prefer the “dummy” value, because it’s still information and it makes things easy to trace through and inspect
b

Basil V

03/25/2020, 6:54 PM
That's a fair point
One other quick question—I'm just getting into composite solids. How does configuration look for a composite solid?
Like in my yaml config file for solids
p

prha

03/25/2020, 6:55 PM
you configure the composite as usual, but it has a config mapping function that can map the composite’s config into config for its individual component solids
b

Basil V

03/25/2020, 6:55 PM
Do I pass the config for each "sub-solid" to the main composite solid, or still define the indiviual config for each?
Ahh ok great
Is there a page about the mapping function in docs anywhere that you could point me to?
p

prha

03/25/2020, 6:57 PM
one sec, lemme check
b

Basil V

03/25/2020, 6:58 PM
Thanks
b

Basil V

03/25/2020, 7:03 PM
Awesome really appreciate it
p

prha

03/25/2020, 7:05 PM
no problem!
b

Basil V

03/25/2020, 7:23 PM
Does that pattern of using the Nothing type work with composite solids?
Copy code
dagster.core.errors.DagsterInvalidDefinitionError: @composite_solid 'load' decorated function does not have parameter(s) 'start_after', which are in solid's input_defs. Solid functions should only have keyword arguments that match input names and a first positional parameter named 'context'.
And the solid:
Copy code
@composite_solid(
    description='The L in ELT',
    input_defs=[InputDefinition(name='start_after', dagster_type=Nothing)]
)
def load() -> Nothing:
...
d

dwall

03/25/2020, 7:31 PM
@Basil V I think you need to supply
start_after
as an input to your
load()
function
b

Basil V

03/25/2020, 7:32 PM
Why do I need to do this for composite solids but not for regular solids?
d

dwall

03/25/2020, 7:32 PM
so
def load(start_after: Nothing) -> Nothing:
errr I'll let the dagster team answer but my understanding was that it had to do with the input mapping that happens in a composite solid
b

Basil V

03/25/2020, 7:33 PM
If I change my definition to:
Copy code
@solid(
    description='The L in ELT',
    input_defs=[InputDefinition(name='start_after', dagster_type=Nothing)]
)
def load(context) -> Nothing:
It doesn't give me the error
Hmm ok, thanks!
d

dwall

03/25/2020, 7:34 PM
here is an example from our project:
Copy code
@composite_solid(
    name="dbt_rpc_snapshot_freshness_and_wait",
    description="Invoke a dbt source snapshot-freshness process and wait for it to finish.",
    input_defs=[InputDefinition(name="start_after", dagster_type=Nothing,)],
    output_defs=[OutputDefinition(name="result", dagster_type=DbtRpcPollResult)],
    config={
        "select": Field(
            config=Noneable(Array(String)),
            default_value=None,
            is_required=False,
            description="The dbt sources to snapshot freshness for.",
        ),
        "interval": Field(
            config=Int,
            is_required=False,
            default_value=10,
            description="The interval (in seconds) at which to poll the dbt rpc process.",
        ),
        "logs": Field(config=Bool, is_required=False, default_value=True),
    },
    config_fn=lambda cfg: {
        "dbt_rpc_snapshot_freshness": {"config": {"select": cfg["select"]}},
        "dbt_rpc_poll": {"config": {"logs": cfg["logs"], "interval": cfg["interval"]}},
    },
)
def dbt_rpc_snapshot_freshness_and_wait(start_after: Nothing) -> DbtRpcPollResult:
    return dbt_rpc_poll(request_token=dbt_rpc_snapshot_freshness(start_after=start_after))
a

alex

03/25/2020, 7:34 PM
ya the “inputs” to
@composite_solid
s are input mappings, and while
Nothing
has no value so is not useful in the compute body of a
@solid
, the
@composite
still needs to route where the
Nothing
is supposed to go to
b

Basil V

03/25/2020, 7:37 PM
Oh so my first "solid" in a composite solid will also need the "Nothing" input definition?
👍 1
Refering to
Copy code
dbt_rpc_snapshot_freshness(start_after=start_after)
d

dwall

03/25/2020, 7:39 PM
in that example, the
dbt_rpc_snapshot_freshness
solid has a
Nothing
input, so all the composite solid is doing is mapping the
Nothing
it recieved as an input to the
Nothing
input of the
dbt_rpc_snapshot_freshness
solid
but in theory it doesnt have to be the first solid I don't think. It can map
Nothing
to any child solid that has
Nothing
as an input
but in the instance that we are referring to where we just want a solid to start after another solid without an explicit input/output dependency, I guess it will often be the first child solid in the composite getting the
Nothing
input
and yes, here is the
dbt_rpc_snapshot_freshness
solid def:
Copy code
@solid(
    description="A solid to invoke dbt source snapshot-freshness.",
    input_defs=[InputDefinition(name="start_after", dagster_type=Nothing)],
    output_defs=[
        OutputDefinition(
            name="request_token", dagster_type=String, description="The request token of the invoked dbt snapshot."
        )
    ],
    config={
        "select": Field(
            config=Noneable(Array(String)),
            default_value=None,
            is_required=False,
            description="The dbt sources to snapshot-freshness for.",
        )
    },
    required_resource_keys={"rpc"},
    tags={"kind": "dbt"},
)
def dbt_rpc_snapshot_freshness(context) -> String:
    resp = context.resources.rpc.snapshot_freshness(select=context.solid_config["select"])
    return resp.json().get("result").get("request_token")
b

Basil V

03/25/2020, 7:41 PM
Yeah exactly. Ok but there has to be at least one solid within the composite that has the
start_after
param? I mean there isn't way to simply define the
start_after
in the composite and handle it there without having to pass it to any of the solids within?
d

dwall

03/25/2020, 7:42 PM
yeah I believe it needs to be mapped. Again, I'll let the dagster team confirm
b

Basil V

03/25/2020, 7:44 PM
It feels like this logic for defining the dependency structure is "leaking" into the sub-solids when it ideally wouldn't have to. But maybe the way dagster is built doesn't allow for that
Thanks for the info much appreciated
👍 1
a

alex

03/25/2020, 7:52 PM
yeah - everything in dagster is data dependencies so
Nothing
is just a hacked up data dependency that can be used to create hacky sequencing dependencies. Its cumbersome nature reflects this.
b

Basil V

03/25/2020, 7:57 PM
Ok thanks
2 Views