Hi all, maybe a silly question but trying to work ...
# ask-community
Hi all, maybe a silly question but trying to work out the best way to do this. I'm currently using software-defined assets and only want to use the local
. I don't want to set up a dagster instance yet. I have an asset that takes a long amount of time to generate. I want to be able to configurably use the memoized results of this asset, preferably only when the user says so (i.e. by default, run the long computation every time to generate the asset). This asset later gets used by some other downstream assets. • I've tried to use the memoization API but had various problems with it (among which was having to set up a temporary dagster instance with
, but also some other issues that I don't fully remember). • An easy option would be to provide a configuration option to the asset, but it seems that
does not support configuration/config schemas? • Another option would be to have a configurable resource that feeds into the asset, but this seems like a strange workaround? • Previously I had two separate jobs built manually, and had 2 assets with the same asset name (one to do the long computation, one to simply read the latest stored results), so downstream assets didn't care which of the 2 assets I had picked. However, now I'm using
, but it can't accept two assets with the same name, so I have to still set up separate AssetGroups manually listing all their upstream and downstream assets, which is a bit of a pain. Just wanted to see if there were any recommendations on doing something like this, or which of the above options feels cleanest?
🤖 1
Hi Dan, Asset config is landing in the next few weeks-- that will definitely be your cleanest option. It's even possible a thin version will be included in today's release, I'll let you know.
Fantastic, thanks a lot for your comments!
@sean Just wanted to confirm, it seems that a thin version didn't make it to the .17 release today right? Could you let me know if there is a branch and/or issue in which it's being worked on so I can follow the progress? Thanks again!
Looks like possibly this is the one to follow 🙂 https://github.com/dagster-io/dagster/pull/7590
Hi Dan, yes you’re right it didn’t land in this week’s release but definitely coming next week. A lot of the work is dagit-oriented and not in a PR yet.
👍 1