Slackbot
02/21/2024, 7:34 PMScout
02/21/2024, 7:34 PMCustomDagsterDbtTranslator
, you can use these mechanisms to align the asset keys. Here's how you can approach this: 1. Define Asset Keys in dbt: You can customize the asset keys generated by Dagster for dbt assets by defining a meta
key on your dbt node's .yml
file. For example, if you want to override the asset key for a source and table as snowflake/jaffle_shop/orders
, you would define it like this in your dbt source file: yaml sources: - name: jaffle_shop tables: - name: orders meta: dagster: asset_key: ["snowflake", "jaffle_shop", "orders"]
2. Custom DagsterDbtTranslator: If you want to override the asset key generation for all dbt nodes in your dbt project, you can create a custom DagsterDbtTranslator
and implement the get_asset_key
method. This method can be overridden to provide a custom asset key for a dbt resource. For example, to add a prefix to the default generated asset key, you would do something like this: python from typing import Any, Mapping from dagster import AssetKey from dagster_dbt import DagsterDbtTranslator class CustomDagsterDbtTranslator(DagsterDbtTranslator): @classmethod def get_asset_key(cls, dbt_resource_props: Mapping[str, Any]) -> AssetKey: # Add custom logic to generate the asset key # For example, adding a prefix: return super().get_asset_key(dbt_resource_props).with_prefix("prefix")
3. Link Fivetran Assets: Ensure that the Fivetran assets have asset keys that match the ones you've defined or generated for your dbt assets. If the Fivetran connector is storing data in a table and the corresponding asset key is public/survey_responses
, you would need to make sure that the dbt asset key matches this format. 4. Use the Custom Translator: When loading assets from your dbt project, make sure to use your custom translator so that the asset keys are generated according to your custom logic. By aligning the asset keys between Fivetran and dbt, you will be able to link the assets in Dagster, allowing you to visualize and manage dependencies between them effectively. Remember to adjust the prefixes and asset key definitions so that they match across both Fivetran and dbt.