Hi team, we are currently in the process of migrating our Dagster prod k8s environment from 0.15.8 to 1.3.1. Our migration pod is struck for a long time and not sure if there is any way to monitor what's going on. Here are logs that I am seeing on the migrate pod. Its struck at

Querying run storage

for more than 30 mins

time="2023-05-04T00:32:16Z" level=info msg="spawning process: [dagster instance migrate]" app=vault-env
$DAGSTER_HOME: /opt/dagster/dagster_home

Updating run storage...
Skipping already applied data migration: run_partitions
Starting data migration: run_repo_label_tags
Querying run storage.

I could definitely imagine that migration taking a while for that many runs, yeah

Hmm, we waited for more than 2.5hrs, it was not even able to finish 10% of the runs. However, the migration failed with the following error

Updating run storage...
Skipping already applied data migration: run_partitions
Starting data migration: run_repo_label_tags
Querying run storage.
Traceback (most recent call last):
  File "/usr/local/bin/dagster", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/dagster/_cli/__init__.py", line 46, in main
    cli(auto_envvar_prefix=ENV_PREFIX)  # pylint:disable=E1123
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/dagster/_cli/instance.py", line 49, in migrate_command
    instance.upgrade(click.echo)
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/instance/__init__.py", line 830, in upgrade
    self._run_storage.migrate(print_fn)
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/storage/runs/sql_run_storage.py", line 986, in migrate
    self._execute_data_migrations(REQUIRED_DATA_MIGRATIONS, print_fn, force_rebuild_all)
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/storage/runs/sql_run_storage.py", line 980, in _execute_data_migrations
    migration_fn()(self, print_fn)
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/storage/runs/migration.py", line 197, in migrate_run_repo_tags
    run = deserialize_value(row[0], DagsterRun)
  File "/usr/local/lib/python3.7/site-packages/dagster/_serdes/serdes.py", line 658, in deserialize_value
    packed_value = seven.json.loads(val)
  File "/usr/local/lib/python3.7/json/__init__.py", line 361, in loads
    return cls(**kw).decode(s)
  File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 3631 (char 3630)

How important is it to keep all 1.3 million runs?

Now the db is in a corrupted state and we had a downtime. So, we tried to revert to the old state, by reverting to the db snapshot. However, the daemon starts hanging continuously every 5 mins

This is on the old version but before the migration?

Yes, this is the old version which we snapshotted before starting the migration

I think you're going to need to complete the migration for things to run smoothly on the new version, especially with 1.3 million runs

We actually want to get back to the old state to avoid more downtime, hence reverted the DB to the old snapshot which we took before starting the migration. However, that does not seem to work

The same code and database that were working before are no longer working?

Clearing some old runs might be fine. But at this point we are just trying to get back to the old state

I don't have an immediate explanation for why if nothing changed things would stop working

I don't see anything off in the daemon logs. It just stopped abruptly at a particular time.

It's hanging? Or crashed?

The daemon pod looks health though

Right

The old version was running at the same scale even before, but it was working fine. I am trying to see if there is anything off anywhere

The same py-spy tool that I recommended on a different thread for perf investigation can also help identify why a python process is hanging, if that's what's happening here: https://github.com/benfred/py-spy

The cpu and memory seems to be fine fine for the daemon pod. When I restart the daemon pod, it works for sometime, but eventually hangs

got it - if it's hanging, running my-spy is what i'd recommend to get to the bottom of why

Ok, I am am able to find this

Traceback (most recent call last):
  File "/usr/local/bin/dagster-daemon", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/cli/__init__.py", line 127, in main
    cli(obj={})  # pylint:disable=E1123
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/cli/__init__.py", line 43, in run_command
    _daemon_run_command(instance, kwargs)
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/telemetry.py", line 110, in wrap
    result = f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/cli/__init__.py", line 55, in _daemon_run_command
    controller.check_daemon_loop()
  File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/controller.py", line 273, in check_daemon_loop
    self.check_daemon_heartbeats()
  File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/controller.py", line 248, in check_daemon_heartbeats
    failed_daemons=failed_daemons
Exception: Stopping dagster-daemon process since the following threads are no longer sending heartbeats: ['SCHEDULER']

what version of grpcio do you have installed?

Its 1.47.0

OK, likely not that then

How can I run py-spy in a running pod? Do I have to attach it before starting the deployment?

in k8s you have to add this securityContext: https://github.com/benfred/py-spy#how-do-i-run-py-spy-in-kubernetes But then you can just pip install py-spy and run it

I see, Not sure how I can tweak the security context. Most of the k8s tooling are controlled by our internal team. At this point, I am not sure if that's the fastest way to get back to a healthy state

that's the main tool in my toolbox for debugging hanging pods - if that's not an option you could try selectively disabling schedules until things get back in a good state, take a closer look through the logs and try to find the last thing that happened and see if it's suspicious..

Is there any way we can disable scheduler daemon

you could set dagsterDaemon.heartbeatTolerance to a very high number in your helm chart and that would prevent it from crashing if the scheduler daemon is hanging

We had to truncate al the past runs and complete the migration

Got it - are things going more smoothly now?

It looks to be running fine now. I assume that the sensor cursor and ticks should still be persistent though? So that we are not re-running all the jobs

Yeah, those should stay the same

Thanks Daniel for the support. However, this is surely not ideal as we have a long downtime and very tough migration (trying different things to make it work for a long time). Some warnings in the migration guide about scale considerations would have been helpful. Also, the current migration approach seems to be non scalable, having to update runs tags for 100 runs at a time might not be ideal. Not sure how it worked for other deployment migrations

The data migration should be idempotent - so if you added them back and re-ran it, that could work. I'm not sure that would be a zero-downtime operation though

Thanks, I am not sure if we want to take a risk doing any manual DB operation and mess up things now. We might just fine losing the runs information

We have this for ticks but not yet for runs: https://docs.dagster.io/deployment/dagster-instance#data-retention I'll check about the status of run retention

Yeah, we are already using it for sensor ticks. Would also be great to have a similar setting for runs

I think the feature request tracking that is here: https://github.com/dagster-io/dagster/issues/4100

Thanks for sharing. Curious if we are expected to do this at a regular cadence? Are there any numbers on what scale can Dagster support today? I want to understand this to determine the how often I need to run this clean up jobs

I don't think there's a single number of jobs, because there are so many variables at play (the size of your DB, the size of the runs, the # of events, etc.)

In this case, this was solely dependent on the size of the runs table as the migrate finished quickly once we truncated the runs table. I am just wondering how it worked well for other users and want to take appropriate actions to avoid this in our next upgrade

The issue you ran into there was fairly specific to that migration I think - I don't expect similar problems going forward

Also, previously I was told that I can assume the entire migration runs in a single transaction. It does not look like that's true anymore?

I think that applies to schema migrations - this was a data migration

I see. Sorry for the bunch of questions and thanks for answering 🙂 Just want to avoid the pain during our next upgrade and prepare better. From next time, we probably need to have a better idea on what kind of migration is going to happen or even test it before hand on a prod DB replica.

Testing it out first on a replica sounds like a great idea to me yeah

In case, if you have any thoughts on this question, I would really appreciate your response. Thanks https://dagster.slack.com/archives/C01U954MEER/p1683239393374199