https://dagster.io/ logo
Title
m

Manish Khatri

11/18/2022, 10:51 AM
Hi :wave_anim: we have written a custom library for Stitch Data which is almost identically based on the template of
dagster-fivetran
. When it comes to building assets via build_fivetran_assets, I can see that the
dagster-fivetran
library will trigger a sync job. Under the hood, dagster assigns this op to the Asset so when any manual materialisation is triggered in the UI, there is an assigned op to do this. For our Stitch Data connector, we implement the same logic but call it
build_stitch_assets
. Ideally we do not want to trigger a sync for all our ingestions just to build the assets upon first loading of the repository. Instead, it would be more optimal to fetch the metadata via the API for the last runs & table schema which will give as all the juicy info to build the Asset without triggering some potentially long syncs. The problem is that if we use the logic to build assets from the metadata approach mentioned above, the op associated with it would be the metadata pull and not a sync. Is there a way we can override the op when using the
@multi_asset
? From the source code, I can see the
@multi_asset
decorator will create an
AssetsDefinition
and then assign an op via
node_def
arg. However, having to replicate all the decorator logic just to override the
node_def
assignment doesn’t seem like a good approach. Any advice or input would be appreciated.
c

claire

11/18/2022, 9:39 PM
Hi Manish, one other thing you could consider is using `AssetsDefinition.from_op` method to create your multi-assets. By doing this, you could avoid duplicating all of the multi-asset decorator logic and just use this method instead
m

Manish Khatri

11/18/2022, 9:44 PM
Thanks Claire, I’ll look into this :dagstar:
Is there any example code of its use? It would be good to get an idea of the kwarg inputs for a real world scenario. I searched the code base but only see tests using it