Hi, I run multiprocess jobs daily and once failed ...
# ask-community
Hi, I run multiprocess jobs daily and once failed to create a multiprocess. Should I assume that this can happen at the right time? The environment is a dagster cloud ECS environment, version v1.3.5.
The error message is as follows.
Copy code
dagster._core.errors.DagsterSubprocessError: During multiprocess execution errors occurred in child processes:
In process 48: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Stack Trace:
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/executor/child_process_executor.py", line 79, in _execute_command_in_child_process
    for step_event in command.execute():
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/executor/multiprocess.py", line 76, in execute
    execution_plan = create_execution_plan(
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/api.py", line 704, in create_execution_plan
    job_def = job.get_definition()
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/reconstruct.py", line 257, in get_definition
    return self.repository.get_definition().get_maybe_subset_job_def(
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/reconstruct.py", line 120, in get_definition
    return repository_def_from_pointer(self.pointer, self.repository_load_data)
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/reconstruct.py", line 741, in repository_def_from_pointer
    target = def_from_pointer(pointer)
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/reconstruct.py", line 633, in def_from_pointer
    target = pointer.load_target()
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/code_pointer.py", line 224, in load_target
    module = load_python_module(self.module, self.working_directory)
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/code_pointer.py", line 135, in load_python_module
    return importlib.import_module(module_name)
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/root/etl/__init__.py", line 1, in <module>
    from .definition import defs
  File "/root/etl/definition.py", line 4, in <module>
    from etl.jobs.job import elt_job
  File "/root/etl/jobs/job.py", line 3, in <module>
    from etl.assets.asset import (
  File "/root/etl/assets/asset.py", line 7, in <module>
    dbt_assets = load_assets_from_dbt_project(
  File "/usr/local/lib/python3.10/site-packages/dagster_dbt/asset_defs.py", line 519, in load_assets_from_dbt_project
    manifest_json, cli_output = _load_manifest_for_project(
  File "/usr/local/lib/python3.10/site-packages/dagster_dbt/asset_defs.py", line 95, in _load_manifest_for_project
    return json.load(f), cli_output
  File "/usr/local/lib/python3.10/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/usr/local/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None

The above exception occurred during handling of the following exception:
StopIteration: 0

Stack Trace:
  File "/usr/local/lib/python3.10/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)

  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/api.py", line 754, in job_execution_iterator
    for event in job_context.executor.execute(job_context, execution_plan):
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/executor/multiprocess.py", line 309, in execute
    raise DagsterSubprocessError(
it looks like the error here is happening inside
, i.e. it's a problem with decoding the JSON manifest from your dbt project
Yes. However, we have not made any special modifications to the system and it works fine on other days. The manifest file and other files are obtained from github each time, so I don’t think the contents have changed. Is it safe to assume that this is a rare occurrence?
It's hard to say because I don't know what's causing it for sure, but sounds very plausible, if there's some anomalous race condition causing that manifest file to be empty sometimes.
Maybe there is something wrong with the dbt side of the process. Thank youthankyou bow
I'm seeing this same error - wondering if there is any progress.
@Aki Iwa try using
if you haven't https://dagster.slack.com/archives/C01U954MEER/p1686658815279699
ty thankyou 1
thankyou bow 1