I really like the `asset reconciliation sensor` with SLAs bu

Question

I really like the `asset reconciliation sensor` with SLAs but this makes the lack of configuration options in assets and multi assets even more relevant than it was before While we can set the configs on the job level it s impossible to configure the assets themselves Maybe I m still stuck in dagster anti patterns but I d like to keep my code as static as possible and configure assets with YAML files

Vinnie · Answer

As an example of a typical config ```ops fetch pubchem compound by name config compounds FeCl3 Vitamin B7 compound return parameters label Molecular Weight alias molecular weight ``` In this case it s possible I ll need to get different compounds or return different parts of the pubchem response and I d prefer not touching the underlying asset code when that s necessary In case this really is an anti pattern I d love to see what other solutions people have been using with SDAs

Daniel Mosesson · Answer

Not clear to me if its a good idea or not but I have been using somewhat trivial resources for those that return a string or dict or whatever

Daniel Mosesson · Answer

and then in different asset groups repos using different resource implementations

sandy · Answer

< Vinnie> this makes sense do you have any thoughts on what an ideal API would look like to accomplish this

Vinnie · Answer

< sandy> I remember a few months back when we floated the idea of a ` configured` method attached to AssetsDefinitions This would definitely solve my use case although for how I ve developed thus far I haven t needed to keep different configurations of the same asset at the same time so even just having a `config` flag on the decorators or ` from graph` method analog to ` job config= ` would already be enough For more context all the assets I ve written thus far take some form of mandatory configuration From fairly basic like the example with the compounds to more complex in case they rely on the same underlying API and need to hit different endpoints and process the data in different ways My pattern has always been to put them in `job`s and configure the `job`s which has been good enough for everything thus far but wouldn t allow me to adopt `FreshnessPolicies` or the `asset reconciliation sensor` So this is definitely not a major issue but a nice to have

sandy · Answer

Got it We currently have a PR out for adding `configured` to `AssetsDefinition` though it s a bit stalled If you need it in the meantime you could just copy the core of the implementation into your own function <https github com dagster io dagster pull 10497 files diff 8361871844cf00752df2558e86b2673b185530c5b56c3e152267b78ef0ce5369R878> E g make a `configured asset` function that accepts an `AssetsDefinition` and a config dictionary and creates a new `AssetsDefinition` using the code I linked

Vinnie · Answer

Thanks Sandy I ll see if I can adapt it a little bit and pass some feedback in that case but I m on PTO for the last two weeks of the year so there s a chance you ll have it merged before I get around to it slightly smiling face

Vinnie · Answer

Just wanted to followup to say I played around with the suggestion for a bit but couldn t seem to get it working assets would still require me to input the config when launching from the launchpad but I did catch something interesting when trying to pre configure a graph before turning it into an asset dagster raises the error `dagster core errors DagsterInvalidDefinitionError Only graphs utilizing config mapping can be pre configured The graph my graph does not have a config mapping and thus has nothing to be configured ` even though the underlying `op`s that compose the `GraphDefinition` do require a config This would mean I d need to recreate the config mapping within the ` graph` definition effectively something like working example ``` op config schema= my str str def my op context return context op config my str graph config= my op config my str str def my graph return my op some asset = AssetsDefinition from graph my graph configured my op config my str hello name= some asset ``` I also saw that the `config` parameter within ` graph` is undocumented only showing up in the docs for `GraphDefinition` Ultimately this is probably a super niche case but wanted to raise it just to be sure I ll stick to my current methods by now which already work just fine for our purposes slightly smiling face

Ajita Khanal · Answer

It would be so cool to have this feature Any changes that it would be merged soon relaxed <https github com dagster io dagster pull 10497 files diff 8361871844cf00752df2558e86b2673b185530c5b56c3e152267b78ef0ce5369R878>

sandy · Answer

Hey < Ajita Khanal> are you able to tell us a little bit about what you d use it for Most of the times when someone uses asset config it s to pass values at run time but I m assuming that s not the case in your case

Ajita Khanal · Answer

Yes we have a set of assets that depend on one another we were hoping that there could be default values appearing in the launchpad from the config defined in the assets using the configSchema so we could just change the parameters in the definition time while materialising the asset We were copy pasting the configuration everytime but I just noticed that there is the scaffold configuration button which is useful too

sandy · Answer

I don t think you would need `configured` for this instead you can set default values <https docs dagster io concepts configuration advanced config types defaults and optional config fields>

Ajita Khanal · Answer

I did try setting up the default values but then i cannot change that through the launchpad anymore incase I would like to have the paramaters something else than the default values And sadly in this case i cant even press the scaffold config button to help me change the config

sandy · Answer

default values should definitely be changeable through the launchpad cc < ben>