Manish Khatri
11/18/2022, 10:51 AMdagster-fivetran
. When it comes to building assets via build_fivetran_assets, I can see that the dagster-fivetran
library will trigger a sync job. Under the hood, dagster assigns this op to the Asset so when any manual materialisation is triggered in the UI, there is an assigned op to do this.
For our Stitch Data connector, we implement the same logic but call it build_stitch_assets
. Ideally we do not want to trigger a sync for all our ingestions just to build the assets upon first loading of the repository. Instead, it would be more optimal to fetch the metadata via the API for the last runs & table schema which will give as all the juicy info to build the Asset without triggering some potentially long syncs.
The problem is that if we use the logic to build assets from the metadata approach mentioned above, the op associated with it would be the metadata pull and not a sync. Is there a way we can override the op when using the @multi_asset
? From the source code, I can see the @multi_asset
decorator will create an AssetsDefinition
and then assign an op via node_def
arg. However, having to replicate all the decorator logic just to override the node_def
assignment doesn’t seem like a good approach.
Any advice or input would be appreciated.claire
11/18/2022, 9:39 PMManish Khatri
11/18/2022, 9:44 PM