https://dagster.io/ logo
#ask-ai
Title
# ask-ai
a

Alec Koumjian

02/06/2024, 5:41 PM
@Scout I am getting an error when I try to run a partitioned asset that depends on a single partitioned
SourceAsset
. This is the error I get:
Copy code
"dagster._core.errors.DagsterExecutionStepNotFoundError: Can not build subset plan from unknown step: mpc_precovery_results\n
s

Scout

02/06/2024, 5:41 PM
Based on the error message you've provided, it appears that Dagster is unable to build a subset execution plan because it cannot find a step named
mpc_precovery_results
. This kind of error typically occurs when Dagster expects to find a particular step within a job or graph, but that step is not present or is incorrectly referenced. The
DagsterExecutionStepNotFoundError
is a specific type of error that indicates Dagster cannot find a step required for execution. This can happen for a few reasons: 1. The step name is misspelled or does not match the name defined in the job or graph. 2. The step is not included in the subset of steps that are being executed. 3. There is a mismatch between the steps defined in the job or graph and the steps that are being requested for execution. To resolve this issue, you should: - Verify that the step
mpc_precovery_results
is correctly defined in your job or graph. - Check that the step is included in the subset of steps you are trying to execute, if you are running a subset. - Ensure that there are no typos or mismatches in the step names between your job or graph definition and the execution request. If you are using dynamic mapping or have complex dependencies between assets, make sure that the dynamic outputs and dependencies are correctly configured. The documentation does not provide a direct solution to this specific error message, but these general debugging steps are a good place to start. If after checking these points the issue persists, you might want to provide more context or code snippets related to how you've defined the
mpc_precovery_results
step and how you are attempting to run the partitioned asset. This additional information could help in diagnosing the problem more accurately.
a

Alec Koumjian

02/06/2024, 5:54 PM
@Scout It is a regular partitioned asset that has an AssetIn of a source asset that is partitioned identically.
s

Scout

02/06/2024, 5:54 PM
Given that your partitioned asset depends on a
SourceAsset
that is partitioned identically, and you're encountering a
DagsterExecutionStepNotFoundError
, there are a few things you should check to ensure that your assets are set up correctly: 1. Asset Definitions: Ensure that both the partitioned asset and the
SourceAsset
it depends on are defined correctly with the same
PartitionsDefinition
. The partitions should align exactly in terms of their partition keys. 2. Asset Dependencies: Verify that the dependency between the assets is defined correctly using
AssetIn
. The
AssetIn
should correctly reference the
SourceAsset
and its partitions. 3. Partition Mappings: If you are using partition mappings to define dependencies between assets, ensure that the mappings are set up correctly. For example, if you're using a
TimeWindowPartitionMapping
, the offsets should be correctly configured to match the partitions of the dependent asset with those of the
SourceAsset
. 4. Execution Plan: When you attempt to run the partitioned asset, make sure that the execution plan is correctly scoped to include the necessary steps for both the partitioned asset and the
SourceAsset
. If you're running a subset of the execution plan, include all relevant steps. 5. Job Configuration: If you're running a job that includes these assets, ensure that the job is configured to materialize the correct partitions. The job should be aware of the partitioned assets and their dependencies. 6. Code and Configuration Sync: Make sure that the code defining your assets and the configuration you're using to run them are in sync. Any discrepancies could lead to steps not being found. If you've verified all of the above and the issue persists, it could be helpful to look at the specific code where the assets are defined and how you're attempting to run them. This might reveal if there's a mismatch or an oversight in the setup. Remember that the error message indicates that Dagster is looking for a step that it expects to exist based on the provided configuration but cannot find it. This often points to a configuration issue or a mismatch between the asset definitions and the execution request. If you can share more details about your asset definitions and how you're running them, it would be easier to pinpoint the exact cause of the issue.
a

Alec Koumjian

02/06/2024, 6:04 PM
@Scout please generate what a correct AssetIn would look like using a sourceasset
s

Scout

02/06/2024, 6:04 PM
Certainly! Below is an example of how you might define a
SourceAsset
and a downstream asset that depends on it using
AssetIn
. This example assumes that both the source asset and the downstream asset are partitioned identically:
Copy code
python from dagster import AssetIn, asset, SourceAsset, AssetKey  # Define a partitioned SourceAsset my_source_asset = SourceAsset(     key=AssetKey("my_partitioned_source_asset"),     # ... include additional parameters like metadata, io_manager_key, etc. )  # Define a downstream asset that depends on the partitioned SourceAsset @asset(     ins={"upstream_asset": AssetIn(key="my_partitioned_source_asset")} ) def my_downstream_asset(upstream_asset):     # The logic for the downstream asset goes here     # 'upstream_asset' will be the data provided by 'my_partitioned_source_asset'     return do_something_with(upstream_asset)  # Replace 'do_something_with' with the actual logic of your downstream asset.
In this example: -
my_source_asset
is a
SourceAsset
that represents an external asset. It is identified by an
AssetKey
which is a unique identifier for the asset within Dagster. -
my_downstream_asset
is a software-defined asset that depends on
my_source_asset
. The dependency is specified in the
ins
parameter of the
@asset
decorator. The key used in
AssetIn
must match the key of the
SourceAsset
. - When you run a job that includes
my_downstream_asset
, Dagster will know to pull in the data from
my_source_asset
for the corresponding partition. Make sure that the partitioning logic for both the source asset and the downstream asset is aligned, and that you have defined a
PartitionsDefinition
that specifies how the partitions are structured if they are partitioned. Please adapt the example to fit the specifics of your assets, such as the partitioning scheme and any additional parameters that may be necessary for your use case.
4 Views