@Scout What's the best pattern to pass filenames from one asset job to another?

The best pattern for passing filenames or any kind of data from one asset to another in Dagster is to use Software-Defined Assets (SDAs). With SDAs, Dagster can manage dependencies and pass data between assets across separate job runs. Here's how you can accomplish this: 1. Define your assets using the

@asset

decorator. Each asset function should return the data that will be consumed by downstream assets. If you're dealing with filenames, the asset can return the filename as its output. 2. Ensure that each asset is configured with an appropriate IOManager that knows how to handle the storage and retrieval of the data (in this case, filenames). The IOManager will take care of materializing the asset's output to a storage system and retrieving it for downstream assets. 3. Set up dependencies between assets by using the outputs of one asset as inputs to another. When you define an asset that depends on another asset, Dagster will automatically ensure that the upstream asset is computed before the downstream asset. Here's an example based on the documentation:

python from dagster import asset  @asset def upstream_asset():     filename = "data_file.txt"     # ... code to generate the file ...     return filename  @asset def downstream_asset(upstream_asset):     filename = upstream_asset     # ... code that uses the filename ...

In this example,

downstream_asset

depends on

upstream_asset

, and Dagster will pass the filename from

upstream_asset

to

downstream_asset

when both are materialized. If you need to pass filenames between separate jobs rather than within a single job, you can still use assets, and Dagster will manage the dependencies as long as the assets are defined in a way that captures their relationship. Remember to configure your IOManager to handle the specific storage and retrieval logic for your filenames. If you're using a filesystem-based IOManager, for example, you might store the filenames in a specific directory, and the IOManager would know to look there to retrieve them for downstream assets. This pattern allows you to decouple the generation and consumption of data, making your pipelines more modular and maintainable.

@Scout Can this not be done by passing metadata in contexts?

Passing metadata between assets or ops within a Dagster job can indeed be done using the context object to emit and retrieve metadata. However, it's important to note that metadata is typically used for logging, annotating, and displaying information in the Dagster UI, rather than for passing data that is used as input to downstream computations. For example, you can use

AssetMaterialization

to attach metadata to an asset when it is materialized, which can then be viewed in the Dagster UI. Here's an example from the documentation:

python @op def emit_metadata(context, df):     yield AssetMaterialization(         asset_key="my_dataset",         metadata={             "my_text_label": MetadataValue.text("hello")         },     )

However, if you want to pass actual data or filenames that downstream assets or ops will use as inputs, it's better to use the return values of assets or ops and define dependencies between them, as described in the previous message. The metadata system is not designed for passing data between computations. Instead, it's intended for providing additional context and information about the computations themselves, which can be useful for monitoring, debugging, and analysis purposes. If you need to pass filenames or other data between jobs or across separate runs, you should use the return values of assets or ops and set up dependencies accordingly. This ensures that the data is managed correctly by Dagster's execution engine and that the dependencies between computations are explicitly defined.