Hi I run multiprocess jobs daily and once failed to create a dagster #ask-community

Hi, I run multiprocess jobs daily and once failed ...

Aki Iwa

06/07/2023, 4:47 AM

Hi, I run multiprocess jobs daily and once failed to create a multiprocess. Should I assume that this can happen at the right time? The environment is a dagster cloud ECS environment, version v1.3.5.

Aki Iwa

06/07/2023, 4:48 AM

The error message is as follows.

Copy code

dagster._core.errors.DagsterSubprocessError: During multiprocess execution errors occurred in child processes:
In process 48: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Stack Trace:
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/executor/child_process_executor.py", line 79, in _execute_command_in_child_process
    for step_event in command.execute():
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/executor/multiprocess.py", line 76, in execute
    execution_plan = create_execution_plan(
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/api.py", line 704, in create_execution_plan
    job_def = job.get_definition()
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/reconstruct.py", line 257, in get_definition
    return self.repository.get_definition().get_maybe_subset_job_def(
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/reconstruct.py", line 120, in get_definition
    return repository_def_from_pointer(self.pointer, self.repository_load_data)
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/reconstruct.py", line 741, in repository_def_from_pointer
    target = def_from_pointer(pointer)
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/reconstruct.py", line 633, in def_from_pointer
    target = pointer.load_target()
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/code_pointer.py", line 224, in load_target
    module = load_python_module(self.module, self.working_directory)
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/code_pointer.py", line 135, in load_python_module
    return importlib.import_module(module_name)
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/root/etl/__init__.py", line 1, in <module>
    from .definition import defs
  File "/root/etl/definition.py", line 4, in <module>
    from etl.jobs.job import elt_job
  File "/root/etl/jobs/job.py", line 3, in <module>
    from etl.assets.asset import (
  File "/root/etl/assets/asset.py", line 7, in <module>
    dbt_assets = load_assets_from_dbt_project(
  File "/usr/local/lib/python3.10/site-packages/dagster_dbt/asset_defs.py", line 519, in load_assets_from_dbt_project
    manifest_json, cli_output = _load_manifest_for_project(
  File "/usr/local/lib/python3.10/site-packages/dagster_dbt/asset_defs.py", line 95, in _load_manifest_for_project
    return json.load(f), cli_output
  File "/usr/local/lib/python3.10/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/usr/local/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None

The above exception occurred during handling of the following exception:
StopIteration: 0

Stack Trace:
  File "/usr/local/lib/python3.10/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)


  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/api.py", line 754, in job_execution_iterator
    for event in job_context.executor.execute(job_context, execution_plan):
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/executor/multiprocess.py", line 309, in execute
    raise DagsterSubprocessError(

➕ 1

sandy

06/07/2023, 8:39 PM

it looks like the error here is happening inside

load_assets_from_dbt_project

, i.e. it's a problem with decoding the JSON manifest from your dbt project

Aki Iwa

06/07/2023, 9:23 PM

Yes. However, we have not made any special modifications to the system and it works fine on other days. The manifest file and other files are obtained from github each time, so I don’t think the contents have changed. Is it safe to assume that this is a rare occurrence?

sandy

06/07/2023, 10:22 PM

It's hard to say because I don't know what's causing it for sure, but sounds very plausible, if there's some anomalous race condition causing that manifest file to be empty sometimes.

Aki Iwa

06/07/2023, 10:35 PM

Maybe there is something wrong with the dbt side of the process. Thank youthankyou bow

Simon Frid

07/11/2023, 9:23 PM

I'm seeing this same error - wondering if there is any progress.

Simon Frid

07/12/2023, 4:59 AM

@Aki Iwa try using

load_assets_from_dbt_manifest

if you haven't https://dagster.slack.com/archives/C01U954MEER/p1686658815279699

ty thankyou 1

thankyou bow 1

2 Views

Open in Slack

Previous Next