Does anyone know if there's a way to monitor when ...
# ask-community
i
Does anyone know if there's a way to monitor when a scheduled run fails to start (e.g. due to a bad config or missing env var that causes a resource initialisation failure)? The
@run_failure_sensor
doesn't seem to pick up on that
c
it's not programmatic like what you're looking for I think, but jobs that fail to start will now be logged in the timeline view in dagit as of 0.15.3
Given that resource init failure results in a DagsterRunStatus.Failure, a bit surprised that's not getting picked up. Will check in with the team.
y
Do you mind sharing the failure message/log in the scheduled run?
i
It's usually something like this:
Copy code
Exception: Missing environment variable ENV_VAR
  File "/usr/local/lib/python3.8/site-packages/dagster/grpc/impl.py", line 82, in core_execute_run
    recon_pipeline.get_definition()
  File "/usr/local/lib/python3.8/site-packages/dagster/core/definitions/reconstruct.py", line 180, in get_definition
    defn = self.repository.get_definition().get_pipeline(self.pipeline_name)
  File "/usr/local/lib/python3.8/site-packages/dagster/core/definitions/reconstruct.py", line 83, in get_definition
    return repository_def_from_pointer(self.pointer)
  File "/usr/local/lib/python3.8/site-packages/dagster/core/definitions/reconstruct.py", line 686, in repository_def_from_pointer
    target = def_from_pointer(pointer)
  File "/usr/local/lib/python3.8/site-packages/dagster/core/definitions/reconstruct.py", line 607, in def_from_pointer
    target = pointer.load_target()
  File "/usr/local/lib/python3.8/site-packages/dagster/core/code_pointer.py", line 176, in load_target
    module = load_python_file(self.python_file, self.working_directory)
  File "/usr/local/lib/python3.8/site-packages/dagster/core/code_pointer.py", line 86, in load_python_file
    return import_module_from_path(module_name, python_file)
  File "/usr/local/lib/python3.8/site-packages/dagster/seven/__init__.py", line 50, in import_module_from_path
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "repo.py", line 3, in <module>
    from <http://data.jobs|data.jobs> import (
  File "/opt/dagster/app/data/jobs.py", line 62, in <module>
    some_resource
  File "/opt/dagster/app/data/some_graphs.py", line 14, in some_resource
    raise Exception(