https://dagster.io/ logo
Join Slack
Powered by
I really like the `asset_reconciliation_sensor` wi...
# dagster-feedback
v
I really like the 
asset_reconciliation_sensor
with SLAs, but this makes the lack of configuration options in assets and multi-assets even more relevant than it was before. While we can set the configs on the job level, it’s impossible to configure the assets themselves. Maybe I’m still stuck in dagster anti-patterns, but I’d like to keep my code as static as possible and configure assets with YAML files.
As an example of a typical config:
Copy code
ops:
  fetch_pubchem_compound_by_name:
    config:
      compounds:
        - FeCl3
        - Vitamin B7
        ...
      compound_return_parameters:
        - label: Molecular Weight
          alias: molecular_weight
        ...
In this case it’s possible I’ll need to get different compounds or return different parts of the pubchem response, and I’d prefer not touching the underlying asset code when that’s necessary. In case this really is an anti-pattern, I’d love to see what other solutions people have been using with SDAs
d
Not clear to me if its a good idea or not, but I have been using somewhat trivial resources for those that return a string or dict or whatever
and then in different asset groups/repos, using different resource implementations
s
@Vinnie this makes sense - do you have any thoughts on what an ideal API would look like to accomplish this?
v
@sandy I remember a few months back when we floated the idea of a 
.configured
method attached to AssetsDefinitions. This would definitely solve my use case, although for how I’ve developed thus far, I haven’t needed to keep different configurations of the same asset at the same time, so even just having a 
config
flag on the decorators or 
.from_graph
method (analog to 
@job(config={}
) would already be enough. For more context, all the assets I’ve written thus far take some form of mandatory configuration. From fairly basic like the example with the compounds, to more complex in case they rely on the same underlying API and need to hit different endpoints and process the data in different ways. My pattern has always been to put them in `job`s and configure the `job`s, which has been good enough for everything thus far, but wouldn’t allow me to adopt 
FreshnessPolicies
or the 
asset_reconciliation_sensor
. So this is definitely not a major issue, but a “nice to have”
s
Got it. We currently have a PR out for adding 
configured
to 
AssetsDefinition
, though it's a bit stalled. If you need it in the meantime, you could just copy the core of the implementation into your own function: https://github.com/dagster-io/dagster/pull/10497/files#diff-8361871844cf00752df2558e86b2673b185530c5b56c3e152267b78ef0ce5369R878. E.g. make a 
configured_asset
function that accepts an 
AssetsDefinition
and a config dictionary and creates a new 
AssetsDefinition
using the code I linked.
v
Thanks Sandy! I’ll see if I can adapt it a little bit and pass some feedback in that case, but I’m on PTO for the last two weeks of the year so there’s a chance you’ll have it merged before I get around to it 🙂
Just wanted to followup to say, I played around with the suggestion for a bit but couldn’t seem to get it working (assets would still require me to input the config when launching from the launchpad), but I did catch something interesting: when trying to pre-configure a graph before turning it into an asset, dagster raises the error 
dagster._core.errors.DagsterInvalidDefinitionError: Only graphs utilizing config mapping can be pre-configured. The graph "my_graph" does not have a config mapping, and thus has nothing to be configured.
even though the underlying `op`s that compose the 
GraphDefinition
do require a config. This would mean I’d need to recreate the config mapping within the 
@graph
definition, effectively something like (working example):
Copy code
@op(config_schema={"my_str": str})
def my_op(context):
    return context.op_config["my_str"]


@graph(config={"my_op": {"config": {"my_str": "str"}}})
def my_graph():
    return my_op()


some_asset = AssetsDefinition.from_graph(
    my_graph.configured(
        {"my_op": {"config": {"my_str": "hello"}}},
        name="some_asset",
    ),
)
I also saw that the 
config
parameter within 
@graph
is undocumented, only showing up in the docs for 
GraphDefinition
Ultimately this is probably a super niche case, but wanted to raise it just to be sure. I’ll stick to my current methods by now which already work just fine for our purposes 🙂
🌈 1
a
It would be so cool to have this feature. Any changes that it would be merged soon?☺️: https://github.com/dagster-io/dagster/pull/10497/files#diff-8361871844cf00752df2558e86b2673b185530c5b56c3e152267b78ef0ce5369R878
s
Hey @Ajita Khanal - are you able to tell us a little bit about what you'd use it for? Most of the times when someone uses asset config, it's to pass values at run time, but I'm assuming that's not the case in your case?
a
Yes we have a set of assets that depend on one another. we were hoping that there could be default values appearing in the launchpad from the config defined in the assets using the configSchema so we could just change the parameters in the definition time while materialising the asset. We were copy pasting the configuration everytime but I just noticed that there is the scaffold configuration button which is useful too.
s
I don't think you would need 
configured
for this instead, you can set default values: https://docs.dagster.io/concepts/configuration/advanced-config-types#defaults-and-optional-config-fields
a
I did try setting up the default values but then i cannot change that through the launchpad anymore incase I would like to have the paramaters something else than the default values. And sadly in this case i cant even press the scaffold config button to help me change the config
s
default values should definitely be changeable through the launchpad. cc @ben
3 Views
Open in Slack
PreviousNext
Powered by