04/20/2023, 11:00 AM
I am trying to run the dagster hoolie example but observe strange parsing errors when DBT is used (for 1.3.0 dagster)
dagster._core.errors.DagsterSubprocessError: During multiprocess execution errors occurred in child processes:
In process 86457: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Stack Trace:
  File "/demeetup/lib/python3.10/site-packages/dagster/_core/executor/", line 79, in _execute_command_in_child_process
    for step_event in command.execute():
  File "/demeetup/lib/python3.10/site-packages/dagster/_core/executor/", line 72, in execute
    execution_plan = create_execution_plan(
  File "/demeetup/lib/python3.10/site-packages/dagster/_core/execution/", line 759, in create_execution_plan
    pipeline_def = pipeline.get_definition()
  File "/demeetup/lib/python3.10/site-packages/dagster/_core/definitions/", line 238, in get_definition
    return self.repository.get_definition().get_maybe_subset_job_def(
  File "/demeetup/lib/python3.10/site-packages/dagster/_core/definitions/", line 117, in get_definition
    return repository_def_from_pointer(self.pointer, self.repository_load_data)
  File "/demeetup/lib/python3.10/site-packages/dagster/_core/definitions/", line 789, in repository_def_from_pointer
    target = def_from_pointer(pointer)
  File "/demeetup/lib/python3.10/site-packages/dagster/_core/definitions/", line 679, in def_from_pointer
    target = pointer.load_target()
  File "/demeetup/lib/python3.10/site-packages/dagster/_core/", line 221, in load_target
    module = load_python_module(self.module, self.working_directory)
  File "/demeetup/lib/python3.10/site-packages/dagster/_core/", line 135, in load_python_module
    return importlib.import_module(module_name)
  File "/demeetup/lib/python3.10/importlib/", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/de_meetup_pipelines/hooli-data-eng-pipelines/hooli_data_eng/", line 1, in <module>
    from .definitions import defs
  File "/de_meetup_pipelines/hooli-data-eng-pipelines/hooli_data_eng/", line 5, in <module>
    from hooli_data_eng.assets import forecasting, raw_data, marketing, dbt_assets
  File "/de_meetup_pipelines/hooli-data-eng-pipelines/hooli_data_eng/assets/", line 76, in <module>
    daily_dbt_assets = load_assets_from_dbt_project(
  File "/demeetup/lib/python3.10/site-packages/dagster_dbt/", line 505, in load_assets_from_dbt_project
    manifest_json, cli_output = _load_manifest_for_project(
  File "/demeetup/lib/python3.10/site-packages/dagster_dbt/", line 94, in _load_manifest_for_project
    return json.load(f), cli_output
  File "/demeetup/lib/python3.10/json/", line 293, in load
    return loads(,
  File "/demeetup/lib/python3.10/json/", line 346, in loads
    return _default_decoder.decode(s)
  File "/demeetup/lib/python3.10/json/", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/demeetup/lib/python3.10/json/", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None

The above exception occurred during handling of the following exception:
StopIteration: 0

Stack Trace:
  File "/demeetup/lib/python3.10/json/", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
@Sean Lopp is this a version mismatch of the latest dagster in dagster-hooli?
Or is this intentional to showcase some flakiness? At least it is unclear to me
I am not sure if the dynamic partitions are discovered correctly - at least I had to hand set them and they were not automatically derived from the sku_stats
The outputted key_product_deepdive assets are some binary file (at least when simply choosing foo) as the name. I thought that a jupyter notebook would be outputted there as well?
it is pickled - but for the hex resource I was expecting to see the notebook somewhere? Is it intentional that the result is only a pickled pandas data frame?

Sean Lopp

04/20/2023, 2:01 PM
Thanks for trying this out Georg! The JSON decode error is unexpected, I will take a look to see if I can reproduce with the latest 1.3.0 release. The dynamic partitions inside the project are currently a bit contrived - the idea is to show how you can create dynamic partitions through the UI, if for example you had an executive request to look into a specific SKU. The expected output currently is a pickled file (and the metadata is shown for the materialized partition which is quite nice). The hex label is there because we use that pickle file as the input to a simple hex notebook, but the hex notebook is not part of the hooli project. There is a separate asset that shows an integrated Jupyter notebook. If we wanted to automatically create dynamic partitions based on the upstream data we'd need to add in a sensor that yields
when it finds new partitions in the data
:rainbow-daggy: 1