@Scout How can I generate lineage from 2 assets if I can’t use the asset def name as a parameter?

Based on the documentation, when defining assets in Dagster, you can specify dependencies between assets by using the

deps

parameter in the

@asset

decorator. This allows you to define the lineage between assets. If you're encountering issues with using the asset definition name as a parameter, you might be facing a situation similar to the one described in a GitHub issue where the variable name of a source asset failed as an input argument to a downstream asset. In that case, the solution was to use the asset key of the source asset as the input argument to the downstream asset, rather than the variable name of the source asset. Here's an example from the documentation that illustrates how to define an asset with a dependency on another asset using the

AssetIn

class:

python from dagster import AssetIn, asset  @asset(ins={"upstream": AssetIn(key="upstream_asset")}) def downstream_asset(upstream):     return upstream + [4]

In this example,

downstream_asset

depends on an upstream asset with the key

"upstream_asset"

. The key is used to establish the lineage between the two assets. If you have a source asset that is not defined within the same codebase or repository, you can use the

SourceAsset

class to include these external assets and allow your other assets to depend on them:

python from dagster import AssetKey, SourceAsset, asset  my_source_asset = SourceAsset(key=AssetKey("a_source_asset"))  @asset(deps=[my_source_asset]) def my_derived_asset():     # Logic that uses the source asset

In this case,

my_derived_asset

depends on

my_source_asset

, and the lineage is established by including

my_source_asset

in the

deps

list. Remember that the asset key must be provided as the argument to downstream assets, and the asset key is used to establish the lineage, not the variable name of the asset definition. If you need further assistance with a specific scenario or error message, please provide additional details so that I can offer more targeted help.

How can use 1 asset as the input of another one if the first asset is in another code location?

To use one asset as the input of another asset when the first asset is in another code location, you can utilize

SourceAsset

to represent the asset from the external code location. This allows you to define dependencies across different code locations in Dagster. Here's a step-by-step guide based on the documentation: 1. Define the asset in the first code location (code_location_1.py):

python # code_location_1.py  from dagster import asset  @asset def code_location_1_asset():     # Logic for the asset     return some_data

2. In the second code location (code_location_2.py), import

SourceAsset

and

AssetKey

from

dagster

and create a

SourceAsset

that represents the asset from the first code location:

python # code_location_2.py  from dagster import AssetKey, SourceAsset, asset  # Create a SourceAsset to represent the asset from the first code location code_location_1_source_asset = SourceAsset(key=AssetKey("code_location_1_asset"))  @asset def code_location_2_asset(code_location_1_asset):     # Logic that uses the data from the first asset     return code_location_1_asset + " processed"

3. Ensure that both code locations are included in your workspace.yaml file so that Dagster knows to load both of them:

yaml # workspace.yaml  load_from:   - python_file: code_location_1.py   - python_file: code_location_2.py

By following these steps, you establish a dependency between

code_location_2_asset

and

code_location_1_asset

across different code locations. When you load both code locations into the Dagster instance, the system will understand the lineage between these assets, and the Dagster UI will be able to represent the asset lineage across the code locations. If you encounter any issues or have specific requirements that are not covered by this explanation, please provide more details so that I can assist you further.

@Scout When importing an asset as a source asset from 1 code location to the other, Dagster return me the error dagster._core.errors.DagsterInvalidDefinitionError: Input asset ‘[“my_new_source_asset_”]’ for asset ‘[“my_key_prefix”, “my_graph_asset”]’ is not produced by any of the provided asset ops and is not one of the provided sources. How can I fix this

The error message you're encountering suggests that Dagster is unable to resolve the dependency of an asset on a

SourceAsset

. This can happen if the

SourceAsset

is not correctly defined or not included in the

Definitions

object that is loaded by Dagster. To fix this issue, you should ensure the following: 1. The

SourceAsset

is correctly defined with the appropriate

AssetKey

. 2. The

SourceAsset

is included in the

Definitions

object that is being loaded by Dagster. Here's an example of how you might define a

SourceAsset

and include it in the `Definitions`:

python # code_location_1.py from dagster import asset  @asset def my_new_source_asset():     # Logic for the asset     return some_data  # code_location_2.py from dagster import AssetKey, SourceAsset, asset, Definitions  # Define the SourceAsset with the key corresponding to the asset in code_location_1 my_new_source_asset = SourceAsset(key=AssetKey("my_new_source_asset"))  @asset def my_graph_asset(my_new_source_asset):     # Logic that uses the data from the source asset     return my_new_source_asset + " processed"  # Include the SourceAsset in the Definitions object defs = Definitions(assets=[my_new_source_asset, my_graph_asset])

Make sure that the

AssetKey

used to define the

SourceAsset

matches the key of the asset in the first code location. Also, ensure that the

SourceAsset

is passed to the

Definitions

object, which is then loaded by Dagster. If you continue to experience issues, please check the following: - The

AssetKey

is unique and correctly references the asset you intend to use as a source. - There are no typos or mismatches in the asset keys between the two code locations. - The

SourceAsset

is correctly imported and referenced in the dependent asset's definition. - The workspace configuration includes both code locations so that Dagster can load all necessary assets and their dependencies. If the problem persists after checking these points, please provide more details about your code and configuration so that we can further diagnose the issue.