https://dagster.io/ logo
#ask-community
Title
# ask-community
h

Harrison Conlin

05/01/2023, 12:47 PM
I'm seeing this as an error under the asset materialization daemon. not quite sure what more I can tell you given the lack of debug output from the daemon (I looked for switches to increase the verbosity)
Copy code
dagster._check.CheckError: Failure condition: 
  File "C:\Users\XXX\AppData\Local\pypoetry\Cache\virtualenvs\dagster-warehouse-fEr6haF0-py3.10\lib\site-packages\dagster\_daemon\daemon.py", line 222, in core_loop
    yield from self.run_iteration(workspace_process_context)
  File "C:\Users\XXX\AppData\Local\pypoetry\Cache\virtualenvs\dagster-warehouse-fEr6haF0-py3.10\lib\site-packages\dagster\_daemon\asset_daemon.py", line 73, in run_iteration
    run_requests, new_cursor = reconcile(
  File "C:\Users\XXX\AppData\Local\pypoetry\Cache\virtualenvs\dagster-warehouse-fEr6haF0-py3.10\lib\site-packages\dagster\_core\definitions\asset_reconciliation_sensor.py", line 881, in reconcile
    ) = determine_asset_partitions_to_reconcile(
  File "C:\Users\XXX\AppData\Local\pypoetry\Cache\virtualenvs\dagster-warehouse-fEr6haF0-py3.10\lib\site-packages\dagster\_core\definitions\asset_reconciliation_sensor.py", line 561, in determine_asset_partitions_to_reconcile
    stale_candidates, latest_storage_id = find_parent_materialized_asset_partitions(
  File "C:\Users\XXX\AppData\Local\pypoetry\Cache\virtualenvs\dagster-warehouse-fEr6haF0-py3.10\lib\site-packages\dagster\_core\definitions\asset_reconciliation_sensor.py", line 382, in find_parent_materialized_asset_partitions
    and not instance_queryer.is_asset_planned_for_run(latest_record.run_id, child)
  File "C:\Users\XXX\AppData\Local\pypoetry\Cache\virtualenvs\dagster-warehouse-fEr6haF0-py3.10\lib\site-packages\dagster\_utils\caching_instance_queryer.py", line 453, in is_asset_planned_for_run
    check.failed("")
  File "C:\Users\XXX\AppData\Local\pypoetry\Cache\virtualenvs\dagster-warehouse-fEr6haF0-py3.10\lib\site-packages\dagster\_check\__init__.py", line 1669, in failed
    raise CheckError(f"Failure condition: {desc}")
o

owen

05/01/2023, 5:15 PM
hi @Harrison Conlin! sorry you're running into this -- the symptom can be fixed fairly easily on our end, so I'm putting a PR up for that now, but I'm still not sure of the root cause here. Just to confirm -- is this happening on every iteration of the daemon, or was it transient? and did you recently delete/wipe any runs?
h

Harrison Conlin

05/01/2023, 11:07 PM
hi @owen I generally see it multiple times. no to deleting/wipining runs
o

owen

05/01/2023, 11:19 PM
interesting -- so this is happening on a lot of ticks, but not all of them? what database are you using for your dagster instance? and just as an update the PR mentioned above has been merged in and will be in this week's release.
h

Harrison Conlin

05/01/2023, 11:32 PM
how often is each tick? truthfully, I haven't been paying too much attention to to be able to comment on if its a lot or all. its more of a side project at my day job, so it gets some neglect postgresql
o

owen

05/01/2023, 11:40 PM
gotcha -- the daemon should tick every 30s-1m or so
h

Harrison Conlin

05/01/2023, 11:42 PM
i'll try pay better attention
I can also blow away the database if that helps, i've been wanting to find an excuse to do so
actually now I think about it, I've also seen a 'cant connect to the grpc server' occasionally intermingled with the errors and then the daemon restarts itself. not sure if that makes a difference (sorry, my actual day job has been consuming most of my headspace)
o

owen

05/02/2023, 12:15 AM
no worries! once we get into grpc server issues, that's a bit outside my area of expertise but if you notice that interfering with things definitely feel free to spin up another thread. if you're in the market for an excuse to blow away the database, it is likely that starting your instance database from a blank slate would resolve the immediate issue