Am I right that I can use auto materialisation and then only need to define a job to refresh my source tables/assets? All other assets will then be build if the source asset changes?

Yes, you are correct. Auto-materialization policies (AMPs) in Dagster allow you to automate the process of keeping assets up-to-date based on certain conditions. When you define an auto-materialization policy for your assets, Dagster can automatically materialize downstream assets when the source assets change. For example, you can set an auto-materialization policy to be

eager

, which means that any downstream assets will be refreshed anytime the source data asset is updated. This allows you to only define a job to refresh your source tables/assets, and the rest of the assets will be automatically built when the source asset changes, adhering to the policies you've set. The documentation also mentions that AMPs can be declared on an asset-by-asset basis and can be applied to multiple assets at once. This means you can have fine-grained control over how and when each asset is materialized in response to changes in their dependencies. Remember that AMPs are an experimental feature, and you should refer to the [Auto-materializing Assets documentation](https://docs.dagster.io/concepts/assets/asset-auto-execution) for more details on how to implement them in your pipelines.

@Scout How and where do I define the auto-materialization policy?

To define an auto-materialization policy in Dagster, you would typically do so in the asset definition. Here's a step-by-step guide on how to define and apply an auto-materialization policy: 1. Enable Auto-Materialization: Before you can use auto-materialization policies, you need to enable this feature in your Dagster instance. This can be done through the Dagster UI by toggling the feature on in the "Deployment" section under "Daemons" for open source deployments or "Agents" for Dagster Cloud. 2. Import the Policy: Import the

AutoMaterializePolicy

class from the

dagster

module. 3. Define the Policy: Create an instance of

AutoMaterializePolicy

using one of the prebuilt policies like

eager

or

lazy

, or customize it by adding or removing rules. 4. Apply the Policy to Assets: Assign the policy to your assets using the

auto_materialize_policy

argument in the asset definition. Here's an example of how you might define and apply an eager auto-materialization policy to an asset:

python from dagster import asset, AutoMaterializePolicy  # Define an eager auto-materialization policy eager_policy = AutoMaterializePolicy.eager()  @asset(auto_materialize_policy=eager_policy) def my_asset():     # Asset computation logic here     ...

In this example,

my_asset

will be automatically materialized whenever its upstream dependencies are updated, because the

eager

policy includes rules that trigger materialization when parent assets are updated. If you want to customize the policy further, you can add or remove rules using the

with_rules

or

without_rules

methods:

python from dagster import AutoMaterializePolicy, AutoMaterializeRule  # Customize the policy by removing a specific rule custom_policy = AutoMaterializePolicy.eager().without_rules(     AutoMaterializeRule.skip_on_parent_missing(), )  @asset(auto_materialize_policy=custom_policy) def my_custom_asset():     # Asset computation logic here     ...

In this customized policy,

my_custom_asset

will still be materialized eagerly, but it won't skip materialization if a parent asset is missing. Please note that the API for defining auto-materialization policies is subject to change, as it is an experimental feature. Always refer to the latest [Dagster documentation](https://docs.dagster.io/_apidocs/assets) for the most up-to-date guidance on using auto-materialization policies.

@Scout how would the configuration look like for dbt assets, so when I run a job that refreshes the source data, the downstream dbt models will also be built automatically, using auto materialisation?

For dbt assets, you can use the

@dbt_assets

decorator provided by the

dagster-dbt

integration to load your dbt models as Dagster assets. To ensure that downstream dbt models are automatically built when you refresh the source data, you can apply an auto-materialization policy to these dbt assets. Here's a general outline of how you might set this up: 1. Load dbt Models as Assets: Use the

@dbt_assets

decorator to load your dbt models into Dagster as assets. 2. Define Auto-Materialization Policy: Define an auto-materialization policy that suits your needs (e.g.,

eager

to automatically materialize downstream assets when the source changes). 3. Apply the Policy to dbt Assets: Apply the auto-materialization policy to your dbt assets. Here's an example of how this might look in code:

python from dagster_dbt import dbt_assets from dagster import AutoMaterializePolicy  # Define an eager auto-materialization policy eager_policy = AutoMaterializePolicy.eager()  # Load dbt models as assets and apply the auto-materialization policy @dbt_assets(auto_materialize_policy=eager_policy) def my_dbt_assets(context, dbt):     # This function should invoke dbt commands to run your dbt project     # For example, you might run 'dbt run' to materialize your dbt models     result = dbt.run()     yield from result

In this example,

my_dbt_assets

represents your dbt models loaded as Dagster assets, and they are configured to be materialized eagerly based on the defined policy. When you run a job that refreshes the source data, Dagster will automatically trigger the materialization of downstream dbt models that depend on those sources, according to the rules defined in the

eager_policy

. Please note that the actual implementation may vary based on your specific dbt project setup and the version of Dagster and dagster-dbt you are using. Always refer to the latest [dagster-dbt documentation](https://docs.dagster.io/_apidocs/libraries/dagster-dbt) for detailed instructions and examples on how to integrate dbt with Dagster, including the use of auto-materialization policies with dbt assets.

@Scout what do you recommend: how should I split the building of my dbt assets? I will first stage them, then transform them and eventually create marts that I transfer to another service.

When structuring your dbt project within Dagster, it's common to follow dbt's recommended practice of splitting your models into stages that reflect their role in the data transformation pipeline. Typically, these stages are: 1. Staging (raw to staged): Models that clean and prepare raw data for transformation. They often correspond to source tables and perform light transformations to standardize and validate data. 2. Transforming (staged to transformed): Models that perform more complex transformations on staged data. These models might aggregate, join, or otherwise transform data into a shape that is useful for analysis. 3. Marts (transformed to marts): Models that create the final analytical outputs, such as data marts, which are often tailored to specific business use cases and are used for reporting and analytics. 4. Transfer to another service: The final step where data marts or other transformed data are sent to another service for consumption. In Dagster, you can represent each of these stages as a set of assets or a job. Here's how you might approach it: - Separate dbt invocations by stage: You can create separate jobs or asset groups for each stage of your dbt project. This allows you to manage dependencies and materializations at each stage independently. - Use asset dependencies to manage flow: Define dependencies between assets to ensure that data flows through your pipeline in the correct order. For example, your transforming models should depend on your staging models, and your marts should depend on your transforming models. - Apply auto-materialization policies: Use auto-materialization policies to automatically rebuild downstream assets when upstream assets change. This ensures that your data marts are always up-to-date with the latest transformations. - Transfer to another service: Once your data marts are materialized, you can create a separate job or asset that handles the transfer of data to another service. This could involve exporting data to a file, sending it to an API, or using a tool like dbt's

run-operation

to trigger external processes. Here's a simplified example of how you might define these stages in Dagster: ```python from dagster import job, op from dagster_dbt import dbt_cli_resource @op(required_resource_keys={"dbt"}) def stage_models(context): context.resources.dbt.run(models=["+tag:staging"]) @op(required_resource_keys={"dbt"}) def transform_models(context): context.resources.dbt.run(models=["+tag:transforming"]) @op(required_resource_keys={"dbt"}) def mart_models(context): context.resources.dbt.run(models=["+tag:marts"]) @op def transfer_to_service(): # Logic to transfer data to another service ... @job(resource_defs={"dbt": dbt_cli_resource