https://dagster.io/ logo
#ask-ai
Title
# ask-ai
j

Jacob Marcil

02/06/2024, 2:24 PM
@Scout How can I generate lineage from 2 assets if I can’t use the asset def name as a parameter?
s

Scout

02/06/2024, 2:24 PM
Based on the documentation, when defining assets in Dagster, you can specify dependencies between assets by using the
deps
parameter in the
@asset
decorator. This allows you to define the lineage between assets. If you're encountering issues with using the asset definition name as a parameter, you might be facing a situation similar to the one described in a GitHub issue where the variable name of a source asset failed as an input argument to a downstream asset. In that case, the solution was to use the asset key of the source asset as the input argument to the downstream asset, rather than the variable name of the source asset. Here's an example from the documentation that illustrates how to define an asset with a dependency on another asset using the
AssetIn
class:
Copy code
python from dagster import AssetIn, asset  @asset(ins={"upstream": AssetIn(key="upstream_asset")}) def downstream_asset(upstream):     return upstream + [4]
In this example,
downstream_asset
depends on an upstream asset with the key
"upstream_asset"
. The key is used to establish the lineage between the two assets. If you have a source asset that is not defined within the same codebase or repository, you can use the
SourceAsset
class to include these external assets and allow your other assets to depend on them:
Copy code
python from dagster import AssetKey, SourceAsset, asset  my_source_asset = SourceAsset(key=AssetKey("a_source_asset"))  @asset(deps=[my_source_asset]) def my_derived_asset():     # Logic that uses the source asset
In this case,
my_derived_asset
depends on
my_source_asset
, and the lineage is established by including
my_source_asset
in the
deps
list. Remember that the asset key must be provided as the argument to downstream assets, and the asset key is used to establish the lineage, not the variable name of the asset definition. If you need further assistance with a specific scenario or error message, please provide additional details so that I can offer more targeted help.
j

Jacob Marcil

02/06/2024, 2:40 PM
How can use 1 asset as the input of another one if the first asset is in another code location?
@Scout How can use 1 asset as the input of another one if the first asset is in another code location?
s

Scout

02/06/2024, 2:40 PM
To use one asset as the input of another asset when the first asset is in another code location, you can utilize
SourceAsset
to represent the asset from the external code location. This allows you to define dependencies across different code locations in Dagster. Here's a step-by-step guide based on the documentation: 1. Define the asset in the first code location (code_location_1.py):
Copy code
python # code_location_1.py  from dagster import asset  @asset def code_location_1_asset():     # Logic for the asset     return some_data
2. In the second code location (code_location_2.py), import
SourceAsset
and
AssetKey
from
dagster
and create a
SourceAsset
that represents the asset from the first code location:
Copy code
python # code_location_2.py  from dagster import AssetKey, SourceAsset, asset  # Create a SourceAsset to represent the asset from the first code location code_location_1_source_asset = SourceAsset(key=AssetKey("code_location_1_asset"))  @asset def code_location_2_asset(code_location_1_asset):     # Logic that uses the data from the first asset     return code_location_1_asset + " processed"
3. Ensure that both code locations are included in your workspace.yaml file so that Dagster knows to load both of them:
Copy code
yaml # workspace.yaml  load_from:   - python_file: code_location_1.py   - python_file: code_location_2.py
By following these steps, you establish a dependency between
code_location_2_asset
and
code_location_1_asset
across different code locations. When you load both code locations into the Dagster instance, the system will understand the lineage between these assets, and the Dagster UI will be able to represent the asset lineage across the code locations. If you encounter any issues or have specific requirements that are not covered by this explanation, please provide more details so that I can assist you further.
j

Jacob Marcil

02/06/2024, 2:45 PM
@Scout When importing an asset as a source asset from 1 code location to the other, Dagster return me the error dagster._core.errors.DagsterInvalidDefinitionError: Input asset ‘[“my_new_source_asset_”]’ for asset ‘[“my_key_prefix”, “my_graph_asset”]’ is not produced by any of the provided asset ops and is not one of the provided sources. How can I fix this
s

Scout

02/06/2024, 2:45 PM
The error message you're encountering suggests that Dagster is unable to resolve the dependency of an asset on a
SourceAsset
. This can happen if the
SourceAsset
is not correctly defined or not included in the
Definitions
object that is loaded by Dagster. To fix this issue, you should ensure the following: 1. The
SourceAsset
is correctly defined with the appropriate
AssetKey
. 2. The
SourceAsset
is included in the
Definitions
object that is being loaded by Dagster. Here's an example of how you might define a
SourceAsset
and include it in the `Definitions`:
Copy code
python # code_location_1.py from dagster import asset  @asset def my_new_source_asset():     # Logic for the asset     return some_data  # code_location_2.py from dagster import AssetKey, SourceAsset, asset, Definitions  # Define the SourceAsset with the key corresponding to the asset in code_location_1 my_new_source_asset = SourceAsset(key=AssetKey("my_new_source_asset"))  @asset def my_graph_asset(my_new_source_asset):     # Logic that uses the data from the source asset     return my_new_source_asset + " processed"  # Include the SourceAsset in the Definitions object defs = Definitions(assets=[my_new_source_asset, my_graph_asset])
Make sure that the
AssetKey
used to define the
SourceAsset
matches the key of the asset in the first code location. Also, ensure that the
SourceAsset
is passed to the
Definitions
object, which is then loaded by Dagster. If you continue to experience issues, please check the following: - The
AssetKey
is unique and correctly references the asset you intend to use as a source. - There are no typos or mismatches in the asset keys between the two code locations. - The
SourceAsset
is correctly imported and referenced in the dependent asset's definition. - The workspace configuration includes both code locations so that Dagster can load all necessary assets and their dependencies. If the problem persists after checking these points, please provide more details about your code and configuration so that we can further diagnose the issue.
4 Views