https://dagster.io/ logo
Join Slack
Powered by
running into a weird sensor error where it's faili...
# ask-community
c
running into a weird sensor error where it's failing due to a partitions def in another code location
Copy code
dagster._core.errors.DagsterDefinitionChangedDeserializationError: Asset kippcamden/dbt/powerschool/src_powerschool__prefs had a PartitionsDefinition at storage-time, but no longer does
full traceback
Copy code
dagster._core.errors.DagsterDefinitionChangedDeserializationError: Asset kippcamden/dbt/powerschool/src_powerschool__prefs had a PartitionsDefinition at storage-time, but no longer does

  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/impl.py", line 375, in get_external_sensor_execution
    return sensor_def.evaluate_tick(sensor_context)
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/sensor_definition.py", line 645, in evaluate_tick
    result = list(self._evaluation_fn(context))
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/sensor_definition.py", line 921, in _wrapped_fn
    result = fn(**context_param, **resource_args_populated)
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/asset_reconciliation_sensor.py", line 1017, in _sensor
    run_requests, updated_cursor = reconcile(
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/asset_reconciliation_sensor.py", line 827, in reconcile
    ) = determine_asset_partitions_to_reconcile(
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/asset_reconciliation_sensor.py", line 543, in determine_asset_partitions_to_reconcile
    backfill_target_asset_graph_subset = get_active_backfill_target_asset_graph_subset(
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/asset_reconciliation_sensor.py", line 309, in get_active_backfill_target_asset_graph_subset
    asset_backfill_data = AssetBackfillData.from_serialized(
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/asset_backfill.py", line 199, in from_serialized
    target_subset=AssetGraphSubset.from_storage_dict(
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/asset_graph_subset.py", line 211, in from_storage_dict
    raise DagsterDefinitionChangedDeserializationError(
they're named differently, they're monitoring completely different asset, they're using different definitions, different images
c
Hm... I wonder if the reconciliation sensor is checking all backfills in the instance (not just your code location) and then trying to deserialize all of them as part of its own status? Perhaps we need to tag the backfills with the reconciliation sensor name and filter for just the backfills triggered by the reconciliation sensor. cc @sandy
s
@Charlie Bini - 1.3 includes `AutoMaterializePolicy`s, which replace asset reconciliation sensors and shouldn't have this problem. Would that be feasible for you to migrate to?
c
Yeah definitely
Haven’t had a chance to try them out but will do next week
@sandy so I implemented the AutoMaterializePolicy, but it doesn't appear to be creating runs. The daemon is running, and this is how I'm assigning it to assets. Am I doing something wrong? https://github.com/TEAMSchools/teamster/blob/f87fd7b5a146e11bcd4d402b5881ec96e89f2472/src/teamster/kipptaf/definitions.py#L29
s
c
yup
s
eager auto-materialization will launch runs when either: • upstream data has changed • there's a freshness policy are either of those the case?
c
yeah upstream partitions have materialized while this was running
s
are they time partitions? currently, only the last time partition is eligible for auto-materializing (although we're looking to address that soon here: https://github.com/dagster-io/dagster/issues/13848)
c
for example
they're dynamic actually
s
@johann or @owen - are you able to help out Charlie here?
j
Would you be able to check for any any errors in the output from the 
dagster dev
process? (or 
dagster-daemon
if running separately)
c
@johann I'm running a hybrid deployment on cloud and kubernetes, do I need to run that on the agent pod?
j
ah sorry, that would only apply to oss. I’ll take a look
c
ok cool, lmk if you need more info
j
you haven’t updated this code location since your report right
c
nope
👍 1
last thing I did was change a sensor tick length around 2 hours ago
j
Following up- we’re seeing performance problems on our end that are preventing the reconciliation from running. We’re going to bump some resources for you while we investigate
c
lol I think that worked
j
we just paused the materializing- due to the high volume we wanted to check that they’re wanted runs. About to post in the kipptaf channel
4 Views
Open in Slack
PreviousNext
Powered by