I really like the `asset_reconciliation_sensor` wi...
# dagster-feedback
I really like the
with SLAs, but this makes the lack of configuration options in assets and multi-assets even more relevant than it was before. While we can set the configs on the job level, it’s impossible to configure the assets themselves. Maybe I’m still stuck in dagster anti-patterns, but I’d like to keep my code as static as possible and configure assets with YAML files.
As an example of a typical config:
Copy code
        - FeCl3
        - Vitamin B7
        - label: Molecular Weight
          alias: molecular_weight
In this case it’s possible I’ll need to get different compounds or return different parts of the pubchem response, and I’d prefer not touching the underlying asset code when that’s necessary. In case this really is an anti-pattern, I’d love to see what other solutions people have been using with SDAs
Not clear to me if its a good idea or not, but I have been using somewhat trivial resources for those that return a string or dict or whatever
and then in different asset groups/repos, using different resource implementations
@Vinnie this makes sense - do you have any thoughts on what an ideal API would look like to accomplish this?
@sandy I remember a few months back when we floated the idea of a
method attached to AssetsDefinitions. This would definitely solve my use case, although for how I’ve developed thus far, I haven’t needed to keep different configurations of the same asset at the same time, so even just having a
flag on the decorators or
method (analog to
) would already be enough. For more context, all the assets I’ve written thus far take some form of mandatory configuration. From fairly basic like the example with the compounds, to more complex in case they rely on the same underlying API and need to hit different endpoints and process the data in different ways. My pattern has always been to put them in `job`s and configure the `job`s, which has been good enough for everything thus far, but wouldn’t allow me to adopt
or the
. So this is definitely not a major issue, but a “nice to have”
Got it. We currently have a PR out for adding
, though it's a bit stalled. If you need it in the meantime, you could just copy the core of the implementation into your own function: https://github.com/dagster-io/dagster/pull/10497/files#diff-8361871844cf00752df2558e86b2673b185530c5b56c3e152267b78ef0ce5369R878. E.g. make a
function that accepts an
and a config dictionary and creates a new
using the code I linked.
Thanks Sandy! I’ll see if I can adapt it a little bit and pass some feedback in that case, but I’m on PTO for the last two weeks of the year so there’s a chance you’ll have it merged before I get around to it 🙂
Just wanted to followup to say, I played around with the suggestion for a bit but couldn’t seem to get it working (assets would still require me to input the config when launching from the launchpad), but I did catch something interesting: when trying to pre-configure a graph before turning it into an asset, dagster raises the error
dagster._core.errors.DagsterInvalidDefinitionError: Only graphs utilizing config mapping can be pre-configured. The graph "my_graph" does not have a config mapping, and thus has nothing to be configured.
even though the underlying `op`s that compose the
do require a config. This would mean I’d need to recreate the config mapping within the
definition, effectively something like (working example):
Copy code
@op(config_schema={"my_str": str})
def my_op(context):
    return context.op_config["my_str"]

@graph(config={"my_op": {"config": {"my_str": "str"}}})
def my_graph():
    return my_op()

some_asset = AssetsDefinition.from_graph(
        {"my_op": {"config": {"my_str": "hello"}}},
I also saw that the
parameter within
is undocumented, only showing up in the docs for
Ultimately this is probably a super niche case, but wanted to raise it just to be sure. I’ll stick to my current methods by now which already work just fine for our purposes 🙂
🌈 1
It would be so cool to have this feature. Any changes that it would be merged soon?☺️: https://github.com/dagster-io/dagster/pull/10497/files#diff-8361871844cf00752df2558e86b2673b185530c5b56c3e152267b78ef0ce5369R878
Hey @Ajita Khanal - are you able to tell us a little bit about what you'd use it for? Most of the times when someone uses asset config, it's to pass values at run time, but I'm assuming that's not the case in your case?
Yes we have a set of assets that depend on one another. we were hoping that there could be default values appearing in the launchpad from the config defined in the assets using the configSchema so we could just change the parameters in the definition time while materialising the asset. We were copy pasting the configuration everytime but I just noticed that there is the scaffold configuration button which is useful too.
I don't think you would need
for this instead, you can set default values: https://docs.dagster.io/concepts/configuration/advanced-config-types#defaults-and-optional-config-fields
I did try setting up the default values but then i cannot change that through the launchpad anymore incase I would like to have the paramaters something else than the default values. And sadly in this case i cant even press the scaffold config button to help me change the config
default values should definitely be changeable through the launchpad. cc @ben