https://dagster.io/ logo
Join Slack
Powered by
hi y'all - question about Dagster Cloud Hybrid and...
# dagster-plus
s
hi y'all - question about Dagster Cloud Hybrid and queue management 🧵
âž• 1
we're running on Cloud Hybrid with Fargate. we ran into a situation where we queued an exorbitant number (26K) of runs (sensor gone awry!), and we'd like to clear the queue (in fact, we'd be just as happy to simply drop the entire dagster DB). I've found this (https://github.com/dagster-io/dagster/discussions/14203) from @daniel recently, but I'm not sure if it applies to dagster cloud (concretely, I'm not sure how to get access to the dagster instance in hybrid mode). anyone have any thoughts/pointers?
a
hi, I'm running into this! we've tried recreating the deployment, which didn't work, and tried ssh'ing into the Dagster agent ECS task via 
aws ecs execute-command
but both approaches didn't work. any suggestions? we're unable to run any jobs on dagster until this is fixed.
d
hi all - I amended the discussion with a way to do this in Dagster Cloud. We're actively working on bulk termination in the UI as well, so hopefully this is a less annoying experience soon. We can also clear these out for you in this particular case if you give us the go-ahead to do so
a
thanks @daniel! you can go ahead and clear these out, but thanks for the reference for future. is the agent token the same one we see in Deployments > Tokens > Manage agent tokens
I tried the script and got the following error:
Copy code
Traceback (most recent call last):
  File "script.py", line 3, in <module>
    instance = DagsterInstance.get() # needs your DAGSTER_HOME to be set
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_core/instance/__init__.py", line 479, in get
    return DagsterInstance.from_config(dagster_home_path)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_core/instance/__init__.py", line 495, in from_config
    return DagsterInstance.from_ref(instance_ref)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_core/instance/__init__.py", line 507, in from_ref
    unified_storage = instance_ref.storage
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_core/instance/ref.py", line 463, in storage
    return self.storage_data.rehydrate() if self.storage_data else None
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_serdes/config_class.py", line 99, in rehydrate
    return klass.from_config_value(self, check.not_none(result.value))
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_core/storage/legacy_storage.py", line 131, in from_config_value
    ConfigurableClassData(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster/_serdes/config_class.py", line 70, in rehydrate
    module = importlib.import_module(self.module_name)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster_cloud/storage/schedules/__init__.py", line 1, in <module>
    from .storage import GraphQLScheduleStorage as GraphQLScheduleStorage
  File "/home/ubuntu/.local/lib/python3.8/site-packages/dagster_cloud/storage/schedules/storage.py", line 34, in <module>
    class GraphQLScheduleStorage(ScheduleStorage["DagsterCloudAgentInstance"], ConfigurableClass):
  File "/usr/lib/python3.8/typing.py", line 261, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3.8/typing.py", line 897, in __class_getitem__
    _check_generic(cls, params)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/typing_extensions.py", line 100, in _check_generic
    raise TypeError(f"{cls} is not a generic class")
TypeError: <class 'dagster._core.storage.schedules.base.ScheduleStorage'> is not a generic class
nevermind, got it to work! just needed a dagster version upgrade. thanks for the help
2 Views
Open in Slack
PreviousNext
Powered by