https://dagster.io/ logo
#ask-community
Title
# ask-community
c

Christoph Leinemann

03/24/2023, 4:17 PM
Hi there, when I'm importing all my fivetran assets and I'm importing all my dbt assets - is dagster expected to create the lineage between my connector asset and the downstream dbt models or do I need to create that lineage in the code?
j

jamie

03/24/2023, 4:24 PM
if the assetkeys of your fivetran assets match the sources definied in your dbt project, dagster should be able to automatically determine the lineage this tutorial isn’t an exact mirror of your situation, but part one where the dbt sources are defined and part three where upstream assets with matching names are created demonstrates the concept
c

Christoph Leinemann

03/24/2023, 5:07 PM
ok, that's helpful, thank you. So it is really about using the same key prefix for loading dbt assets and loading fivetran assets? As on the asset_key I don't have any influence as it's managed by the import. And yes, there will be a match as dbt picks up from tables that fivetran writes to... let me try that
These are the asset key's that I observe when providing the same key-prefix for both, when loading fivetran and dbt assets: fivetran asset <key-prefix>/<schema-name>/<tablename> dbt asset <dbt-source-name>/<tablename>
they both represent the same table. The dbt asset has all the lineage. The fivetran asset does not have any lineage
so my understanding is that the
snowflake_io_manager
could be the binding element between my assets imported form dbt and fivetran. But then when I initialize a storage manage it doesn't seem to get used. I'm testing that by providing it with wrong credentials and nothing fails and I also don't see any snowflake connections when providing correct credentials. My question remains - can I expect dagster to automatically detect the dbt downstream dependencies of my fivetran assets or do I need to manually create that relationship?
I found a (rather hacky) way of making it work. Not providing key prefixes and aligning the name of dbt source with the schema I guess produces a key naming match in dagster and my lineage between fivetran and dbt is automatically created.
two more observations: • when passing the same key_prefix to both the fivetran load and the dbt load the lineage is not created. It is only created when not passing a key prefix. • The storage_manager doesn't seem to play a role. Even when not passing a storage manager the lineage is created based on the key (when no prefix is used)
Any further insights would be appreciated - should the lineage be created leveraging the storage manger and actual schema and table names rather then matching on asset keys?
j

jamie

03/27/2023, 2:10 PM
how are you providing the storage manager? it should be being used, but maybe there’s some specification missing cc-ing @ben and @owen since they are more familiar with the fivetran and dbt integrations and will have more insight as so how the lineage is created when using both
b

ben

03/27/2023, 3:18 PM
Hi Christoph, your observation about lineage here is correct - it’s based solely on asset key. The reason you’re seeing the dbt source not take into account a key prefix is because in this case we have a separate parameter,
source_key_prefix
You should be able to provide a
key_prefix
to fivetran and the same
source_key_prefix
to dbt to chain them together.
4 Views