Hello when using `dagster materialize` I am getting an error dagster #ask-community

Hello, when using `dagster.materialize` I am getti...

Jeff

12/12/2022, 5:17 PM

Hello, when using

dagster.materialize

I am getting an error

E           dagster._check.ParameterCheckError: Param "recon_pipeline" is not a ReconstructablePipeline. Got <dagster._core.definitions.pipeline_base.InMemoryPipeline object at 0x7eff75ff40d0> which is type <class 'dagster._core.definitions.pipeline_base.InMemoryPipeline'>.

I believe this is because I am using pyspark. More details in 🧵

Jeff

12/12/2022, 5:19 PM

Stack trace:

Copy code

/home/jeff/miniconda3/envs/asg-jobs/lib/python3.8/site-packages/dagster/_core/definitions/materialize.py:68: in materialize
    return build_assets_job(
/home/jeff/miniconda3/envs/asg-jobs/lib/python3.8/site-packages/dagster/_core/definitions/job_definition.py:387: in execute_in_process
    return core_execute_in_process(
/home/jeff/miniconda3/envs/asg-jobs/lib/python3.8/site-packages/dagster/_core/execution/execute_in_process.py:77: in core_execute_in_process
    event_list = list(execute_run_iterable)
/home/jeff/miniconda3/envs/asg-jobs/lib/python3.8/site-packages/dagster/_core/execution/api.py:1187: in __iter__
    yield from self.iterator(
/home/jeff/miniconda3/envs/asg-jobs/lib/python3.8/site-packages/dagster/_core/execution/api.py:1091: in pipeline_execution_iterator
    for event in pipeline_context.executor.execute(pipeline_context, execution_plan):
/home/jeff/miniconda3/envs/asg-jobs/lib/python3.8/site-packages/dagster/_core/executor/in_process.py:38: in execute
    yield from iter(
/home/jeff/miniconda3/envs/asg-jobs/lib/python3.8/site-packages/dagster/_core/execution/api.py:1187: in __iter__
    yield from self.iterator(
/home/jeff/miniconda3/envs/asg-jobs/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_plan.py:120: in inner_plan_execution_iterator
    for step_event in check.generator(
/home/jeff/miniconda3/envs/asg-jobs/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_plan.py:349: in dagster_event_sequence_for_step
    raise error
/home/jeff/miniconda3/envs/asg-jobs/lib/python3.8/site-packages/dagster/_core/execution/plan/execute_plan.py:251: in dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
/home/jeff/miniconda3/envs/asg-jobs/lib/python3.8/site-packages/dagster_databricks/databricks_pyspark_step_launcher.py:210: in launch_step
    step_run_ref = step_context_to_step_run_ref(
/home/jeff/miniconda3/envs/asg-jobs/lib/python3.8/site-packages/dagster/_core/execution/plan/external_step.py:152: in step_context_to_step_run_ref
    return StepRunRef(
/home/jeff/miniconda3/envs/asg-jobs/lib/python3.8/site-packages/dagster/_core/definitions/step_launcher.py:53: in __new__
    check.inst_param(recon_pipeline, "recon_pipeline", ReconstructablePipeline),

Jeff

12/12/2022, 5:19 PM

Let me know if any more details are needed.

Open in Slack

Previous Next