Something strange has occurred with the auto materialization dagster #ask-community

Something strange has occurred with the auto-mater...

Philippe Laflamme

07/19/2023, 3:38 AM

Something strange has occurred with the auto-materialization daemon in my dagster instance. After materializing a group of assets from a lazy + freshness policy applied on them, the CPU spiked in the daemon and has more or less been pinned there since then. If I turn off the Auto-materializing daemon from the web UI, the CPU goes back down to the "normal levels". There's nothing in the logs that indicate anything wrong occurring. Nothing in the instance changed, I'm running 1.3.14. Is there anything I can do to figure out what it might be doing to pin the CPU like this?

Philippe Laflamme

07/19/2023, 3:40 AM

FWIW: I have been running this setup for several weeks without any problem.

Philippe Laflamme

07/19/2023, 3:41 AM

Turning the Auto-materializing daemon off and on does not help. The dagster instance has restarted several times already now as well. The CPU gets pinned pretty much immediately after startup.

owen

07/19/2023, 5:25 PM

hi @Philippe Laflamme -- do you happen to have any time-partitioned assets downstream of unpartitioned assets? we're pushing out a fix for an issue related to that in this week's release. If not, do you mind describing some more aspects of your asset graph?

Philippe Laflamme

07/19/2023, 5:26 PM

Yes, this is exactly my case. I debugged locally and it looks like the asset deamon was spending all its time looping through that time-partitioned asset's partitions.

Philippe Laflamme

07/19/2023, 5:28 PM

It caused the heartbeat to timeout, the deamon would exit, k8s would start it up again... rince and repeat.

Philippe Laflamme

07/19/2023, 5:28 PM

Is it safe to rollback to 1.3.13 until the new release is out?

owen

07/19/2023, 5:29 PM

ah yep -- that would be the best solution there, sorry about that!

Philippe Laflamme

07/19/2023, 5:30 PM

ok, just to make totally sure: • I bumped to 1.3.14 • ran

dagster instance migrate

Rolling back to 1.3.13 will "just work" ?

owen

07/19/2023, 5:37 PM

Looks like there were no new schema changes between 1.3.13 and 1.3.14, so the database state should be fine. Beyond that, I don't believe there were any storage changes that would be backwards incompatible with 1.3.13

Philippe Laflamme

07/19/2023, 5:39 PM

Alright, thanks, I'll give that a try. Alternatively, when is the next release planned?

owen

07/19/2023, 5:40 PM

that will roll out tomorrow

Philippe Laflamme

07/19/2023, 5:40 PM

Alright, thanks!

Philippe Laflamme

07/19/2023, 5:41 PM

Is there any risk that the unpartitioned asset materialization will kick off an automaterialization of ALL downstream partitions?

Philippe Laflamme

07/19/2023, 5:41 PM

That's the only thing I really don't want to happen here...

owen

07/19/2023, 5:42 PM

If you have

max_materializations_per_minute

set to the default value (1) then then only the most recent partition of the downstream time partitioned asset will be materialized

Philippe Laflamme

07/19/2023, 5:42 PM

I see, and in tomorrow's release this is all "fixed"?

owen

07/19/2023, 5:43 PM

yep exactly -- the older time partitions will not even be evaluated

Philippe Laflamme

07/19/2023, 5:43 PM

alright, thanks for all the details, I've got enough information to decide what to do

Philippe Laflamme

07/19/2023, 5:48 PM

Copy code

2023-07-19 13:47:40 -0400 - dagster.daemon.AssetDaemon - ERROR - Caught error:
dagster._check.CheckError: Invariant failed. Description: Invalid serialized cursor

Stack Trace:
  File "/.venv/lib/python3.11/site-packages/dagster/_daemon/daemon.py", line 225, in core_loop
    yield from self.run_iteration(workspace_process_context)
  File "/.venv/lib/python3.11/site-packages/dagster/_daemon/asset_daemon.py", line 95, in run_iteration
    AssetReconciliationCursor.from_serialized(raw_cursor, asset_graph)
  File "/.venv/lib/python3.11/site-packages/dagster/_core/definitions/asset_reconciliation_sensor.py", line 362, in from_serialized
    check.invariant(len(data) in [3, 4], "Invalid serialized cursor")
  File "/.venv/lib/python3.11/site-packages/dagster/_check/__init__.py", line 1654, in invariant
    raise CheckError(f"Invariant failed. Description: {desc}")

Philippe Laflamme

07/19/2023, 5:48 PM

I get this when reverting to

1.3.13

owen

07/19/2023, 5:50 PM

ah sorry about that -- forgot about that particular change. That is theoretically fixable with some manual database changes, but it might be best to wait for tomorrow's release

👍 1

Philippe Laflamme

07/20/2023, 11:29 PM

So there's no 1.3.15? I need to bump to 1.4.0 to get the fix?

owen

07/20/2023, 11:55 PM

ah yes that's accurate, sorry for the confusion

Philippe Laflamme

07/20/2023, 11:55 PM

ok, thanks

Philippe Laflamme

07/20/2023, 11:58 PM

I don't typically use

.0

releases. I'm going to revert to 1.3.13 instead, can I simply set the cursor to the "empty" cursor? Will that risk kicking off any new materializations or will it instead "reset" the cursor to the latest after the first run?

owen

07/21/2023, 12:03 AM

got it, for resetting the cursor you can do something like:

Copy code

instance.daemon_cursor_storage.set_cursor_values(
        {"ASSET_DAEMON_CURSOR": None}
    )

if all of your assets are completely up-to-date, then it won't kick off any new materializations, but it could materialize at most one partition per partitioned asset if the most recent partition is out of date

Philippe Laflamme

07/21/2023, 12:04 AM

alright, sounds good, thanks for the details 🙏

Philippe Laflamme

07/21/2023, 2:22 AM

For posterity, I ended up having to do this:

Copy code

from dagster._daemon.asset_daemon import CURSOR_KEY,AssetReconciliationCursor
instance.daemon_cursor_storage.set_cursor_values({CURSOR_KEY: AssetReconciliationCursor.empty().serialize()})

thanks again for the pointers

2 Views

Open in Slack

Previous Next