https://dagster.io/ logo
Join Slack
Powered by
I found the docs at: <https://docs.dagster.io/late...
# announcements
b
I found the docs at: https://docs.dagster.io/latest/api/apidocs/types/#dagster.Nothing this should be what I need I think? Thanks so much!
p
You could use a 
Nothing
type as Nick suggests, or you could pass a newly constructed object with the table name and maybe the load time or something like that
b
Ok awesome—yeah that's what I initially had in mind was to pass some "dummy" value like the table name between them, but I was curious if you guys had a convention for this kind of scenario. It sounds like Nothing is the way to go.
Thanks
d
yeah we do this very often in our own project
something like:
Copy code
@solid(
    description="A solid.",
    input_defs=[InputDefinition(name="start_after", dagster_type=Nothing)]
)
def solid(context):
    pass

...

@pipeline
def pipeline():
    solid(start_after=[other_solid(), another_solid()])
👍 1
p
Yup! FWIW, I tend to prefer the “dummy” value, because it’s still information and it makes things easy to trace through and inspect
b
That's a fair point
One other quick question—I'm just getting into composite solids. How does configuration look for a composite solid?
Like in my yaml config file for solids
p
you configure the composite as usual, but it has a config mapping function that can map the composite’s config into config for its individual component solids
b
Do I pass the config for each "sub-solid" to the main composite solid, or still define the indiviual config for each?
Ahh ok great
Is there a page about the mapping function in docs anywhere that you could point me to?
p
one sec, lemme check
b
Thanks
p
b
Awesome really appreciate it
p
no problem!
b
Does that pattern of using the Nothing type work with composite solids?
Copy code
dagster.core.errors.DagsterInvalidDefinitionError: @composite_solid 'load' decorated function does not have parameter(s) 'start_after', which are in solid's input_defs. Solid functions should only have keyword arguments that match input names and a first positional parameter named 'context'.
And the solid:
Copy code
@composite_solid(
    description='The L in ELT',
    input_defs=[InputDefinition(name='start_after', dagster_type=Nothing)]
)
def load() -> Nothing:
...
d
@Basil V I think you need to supply 
start_after
as an input to your 
load()
function
b
Why do I need to do this for composite solids but not for regular solids?
d
so 
def load(start_after: Nothing) -> Nothing:
errr I'll let the dagster team answer but my understanding was that it had to do with the input mapping that happens in a composite solid
b
If I change my definition to:
Copy code
@solid(
    description='The L in ELT',
    input_defs=[InputDefinition(name='start_after', dagster_type=Nothing)]
)
def load(context) -> Nothing:
It doesn't give me the error
Hmm ok, thanks!
d
here is an example from our project:
Copy code
@composite_solid(
    name="dbt_rpc_snapshot_freshness_and_wait",
    description="Invoke a dbt source snapshot-freshness process and wait for it to finish.",
    input_defs=[InputDefinition(name="start_after", dagster_type=Nothing,)],
    output_defs=[OutputDefinition(name="result", dagster_type=DbtRpcPollResult)],
    config={
        "select": Field(
            config=Noneable(Array(String)),
            default_value=None,
            is_required=False,
            description="The dbt sources to snapshot freshness for.",
        ),
        "interval": Field(
            config=Int,
            is_required=False,
            default_value=10,
            description="The interval (in seconds) at which to poll the dbt rpc process.",
        ),
        "logs": Field(config=Bool, is_required=False, default_value=True),
    },
    config_fn=lambda cfg: {
        "dbt_rpc_snapshot_freshness": {"config": {"select": cfg["select"]}},
        "dbt_rpc_poll": {"config": {"logs": cfg["logs"], "interval": cfg["interval"]}},
    },
)
def dbt_rpc_snapshot_freshness_and_wait(start_after: Nothing) -> DbtRpcPollResult:
    return dbt_rpc_poll(request_token=dbt_rpc_snapshot_freshness(start_after=start_after))
a
ya the “inputs” to 
@composite_solid
s are input mappings, and while 
Nothing
has no value so is not useful in the compute body of a 
@solid
, the 
@composite
still needs to route where the 
Nothing
is supposed to go to
b
Oh so my first "solid" in a composite solid will also need the "Nothing" input definition?
👍 1
Refering to
Copy code
dbt_rpc_snapshot_freshness(start_after=start_after)
d
in that example, the 
dbt_rpc_snapshot_freshness
solid has a 
Nothing
input, so all the composite solid is doing is mapping the 
Nothing
it recieved as an input to the 
Nothing
input of the 
dbt_rpc_snapshot_freshness
solid
but in theory it doesnt have to be the first solid I don't think. It can map 
Nothing
to any child solid that has 
Nothing
as an input
but in the instance that we are referring to where we just want a solid to start after another solid without an explicit input/output dependency, I guess it will often be the first child solid in the composite getting the 
Nothing
input
and yes, here is the 
dbt_rpc_snapshot_freshness
solid def:
Copy code
@solid(
    description="A solid to invoke dbt source snapshot-freshness.",
    input_defs=[InputDefinition(name="start_after", dagster_type=Nothing)],
    output_defs=[
        OutputDefinition(
            name="request_token", dagster_type=String, description="The request token of the invoked dbt snapshot."
        )
    ],
    config={
        "select": Field(
            config=Noneable(Array(String)),
            default_value=None,
            is_required=False,
            description="The dbt sources to snapshot-freshness for.",
        )
    },
    required_resource_keys={"rpc"},
    tags={"kind": "dbt"},
)
def dbt_rpc_snapshot_freshness(context) -> String:
    resp = context.resources.rpc.snapshot_freshness(select=context.solid_config["select"])
    return resp.json().get("result").get("request_token")
b
Yeah exactly. Ok but there has to be at least one solid within the composite that has the 
start_after
param? I mean there isn't way to simply define the 
start_after
in the composite and handle it there without having to pass it to any of the solids within?
d
yeah I believe it needs to be mapped. Again, I'll let the dagster team confirm
b
It feels like this logic for defining the dependency structure is "leaking" into the sub-solids when it ideally wouldn't have to. But maybe the way dagster is built doesn't allow for that
Thanks for the info much appreciated
👍 1
a
yeah - everything in dagster is data dependencies so 
Nothing
is just a hacked up data dependency that can be used to create hacky sequencing dependencies. Its cumbersome nature reflects this.
b
Ok thanks
2 Views
Open in Slack
PreviousNext
Powered by