https://dagster.io/ logo
Join Slack
Powered by
question about dagstermill with passing results-- ...
# announcements
k
question about dagstermill with passing results-- i've been following the examples provided but I seem to have run into an issue when passing my results from one dagstermill solid to another. This is the typical error received:
Copy code
dagster.core.errors.DagsterInvalidDefinitionError: In @pipeline LDA_pipeline, received invalid type <class 'str'> for input "base_path" (passed by keyword) in solid invocation "ETL". Must pass the output from previous solid invocations or inputs to the composition function as inputs when invoking solids during composition.
just trying to create some pipelines with dagstermill attached is my .py which defines my solids -- each of the notebooks does execute: 
dagstermill.yield_result(...)
dagstermill_nb_pipeline.py
m
why are you calling 
str
on line 196?
k
prior to declaring it as type str() it was returning "... invalid type <class 'NoneType'>..." the output is supposed to be a string from the dagstermill solid
m
can i see the contents of the cell in the 
dagstermill_02.04_1_minio_solid_ML_demo.ipynb
notebook that calls 
yield_result
k
dm.yield_result(base_path, output_name='base_path')
i didn't see any examples of dm.yield_result, so I will admit to some ad-libbing here
am i missing something on how inputs are defined for dagstermill solids as the other error that i receive is:
Copy code
dagster.core.errors.DagsterInvalidDefinitionError: In @pipeline LDA_pipeline, received too many inputs for solid invocation ETL_solid. Only 0 defined, received 1
m
can i see the logs produced by the invocation of minioSetup
k
like the errors produced from trying to run the pipeline?
m
yeah, the logs that are produced by that solid when you run the pipeline
k
Okay here we are, I removed the 
str()
and some minor changes and this is the output:
Copy code
dagster pipeline execute -f dagstermill_nb_pipeline.py -a LDA_pipeline
/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/definitions/mode.py:64: UserWarning: system_storage_defs are deprecated and will be removed in 0.10.0 and should be replaced with intermediate_storage_defs for intermediates and resource_defs for files
  warnings.warn(
Traceback (most recent call last):
  File "/home/kevin/anaconda3/envs/dagstermill/bin/dagster", line 8, in <module>
    sys.exit(main())
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/cli/__init__.py", line 38, in main
    cli(obj={})  # pylint:disable=E1123
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/cli/pipeline.py", line 280, in pipeline_execute_command
    execute_execute_command(instance, kwargs)
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/telemetry.py", line 89, in wrap
    result = f(*args, **kwargs)
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/cli/pipeline.py", line 296, in execute_execute_command
    pipeline_origin = get_pipeline_python_origin_from_kwargs(kwargs)
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/cli/workspace/cli_target.py", line 369, in get_pipeline_python_origin_from_kwargs
    repo_definition = recon_repo.get_definition()
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/definitions/reconstructable.py", line 39, in get_definition
    return repository_def_from_pointer(self.pointer)
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/definitions/reconstructable.py", line 477, in repository_def_from_pointer
    target = def_from_pointer(pointer)
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/definitions/reconstructable.py", line 426, in def_from_pointer
    target = pointer.load_target()
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/code_pointer.py", line 255, in load_target
    module = load_python_file(self.python_file, self.working_directory)
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/code_pointer.py", line 134, in load_python_file
    module = import_module_from_path(module_name, python_file)
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/seven/__init__.py", line 116, in import_module_from_path
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "dagstermill_nb_pipeline.py", line 193, in <module>
    def LDA_pipeline():
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/definitions/decorators/pipeline.py", line 112, in pipeline
    return _Pipeline()(name)
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/definitions/decorators/pipeline.py", line 37, in __call__
    fn()
  File "dagstermill_nb_pipeline.py", line 195, in LDA_pipeline
    LDA(ETL(base_path=bp))
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/definitions/solid.py", line 149, in __call__
    return CallableSolidNode(self)(*args, **kwargs)
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/definitions/composition.py", line 156, in __call__
    raise DagsterInvalidDefinitionError(
dagster.core.errors.DagsterInvalidDefinitionError: In @pipeline LDA_pipeline, received too many inputs for solid invocation ETL_solid. Only 0 defined, received 1
m
can you run for me, 
from dagster import execute_solid; execute_solid(minioSetup)
k
Copy code
>>> import dagstermill_nb_pipeline.py
/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/definitions/mode.py:64: UserWarning: system_storage_defs are deprecated and will be removed in 0.10.0 and should be replaced with intermediate_storage_defs for intermediates and resource_defs for files
  warnings.warn(
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/kevin/workspace/dm/dagstermill_nb_pipeline.py", line 193, in <module>
    def LDA_pipeline():
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/definitions/decorators/pipeline.py", line 112, in pipeline
    return _Pipeline()(name)
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/definitions/decorators/pipeline.py", line 37, in __call__
    fn()
  File "/home/kevin/workspace/dm/dagstermill_nb_pipeline.py", line 195, in LDA_pipeline
    LDA(ETL(base_path=bp))
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/definitions/solid.py", line 149, in __call__
    return CallableSolidNode(self)(*args, **kwargs)
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/definitions/composition.py", line 156, in __call__
    raise DagsterInvalidDefinitionError(
dagster.core.errors.DagsterInvalidDefinitionError: In @pipeline LDA_pipeline, received too many inputs for solid invocation ETL_solid. Only 0 defined, received 1
>>> from dagster import execute_solid; execute_solid(minioSetup)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'minioSetup' is not defined
and adding that line to the py file didn't change anything from what i can see
m
yeah sorry, can you comment out the pipeline definition and just try running the minioSetup solid in isolation
👍 1
using 
execute_solid
i just want to establish that it's actually yielding the result you expect
k
Copy code
python dagstermill_nb_pipeline.py
/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/definitions/mode.py:64: UserWarning: system_storage_defs are deprecated and will be removed in 0.10.0 and should be replaced with intermediate_storage_defs for intermediates and resource_defs for files
  warnings.warn(
2020-09-18 19:35:17 - dagster - DEBUG - ephemeral_minioSetup_solid_pipeline - 2f1a827a-f1b7-49ae-b9c3-00942931e28d - PIPELINE_START - Started execution of pipeline "ephemeral_minioSetup_solid_pipeline".
                 pid = 81642
2020-09-18 19:35:17 - dagster - DEBUG - ephemeral_minioSetup_solid_pipeline - 2f1a827a-f1b7-49ae-b9c3-00942931e28d - ENGINE_EVENT - Executing steps in process (pid: 81642)
 event_specific_data = {"error": null, "marker_end": null, "marker_start": null, "metadata_entries": [["pid", null, ["81642"]], ["step_keys", null, ["['minioSetup.compute']"]]]}
                 pid = 81642
2020-09-18 19:35:17 - dagster - DEBUG - ephemeral_minioSetup_solid_pipeline - 2f1a827a-f1b7-49ae-b9c3-00942931e28d - STEP_START - Started execution of step "minioSetup.compute".
                 pid = 81642
               solid = "minioSetup"
    solid_definition = "minioSetup"
            step_key = "minioSetup.compute"
2020-09-18 19:35:17 - dagster - DEBUG - ephemeral_minioSetup_solid_pipeline - 2f1a827a-f1b7-49ae-b9c3-00942931e28d - STEP_INPUT - Got input "minioPath" of type "String". (Type check passed).
 event_specific_data = {"input_name": "minioPath", "type_check_data": [true, "minioPath", null, []]}
                 pid = 81642
               solid = "minioSetup"
    solid_definition = "minioSetup"
            step_key = "minioSetup.compute"
2020-09-18 19:35:17 - dagster - DEBUG - ephemeral_minioSetup_solid_pipeline - 2f1a827a-f1b7-49ae-b9c3-00942931e28d - STEP_INPUT - Got input "bucketName" of type "String". (Type check passed).
 event_specific_data = {"input_name": "bucketName", "type_check_data": [true, "bucketName", null, []]}
                 pid = 81642
               solid = "minioSetup"
    solid_definition = "minioSetup"
            step_key = "minioSetup.compute"
2020-09-18 19:35:17 - dagster - DEBUG - ephemeral_minioSetup_solid_pipeline - 2f1a827a-f1b7-49ae-b9c3-00942931e28d - STEP_INPUT - Got input "srcFilename" of type "String". (Type check passed).
 event_specific_data = {"input_name": "srcFilename", "type_check_data": [true, "srcFilename", null, []]}
                 pid = 81642
               solid = "minioSetup"
    solid_definition = "minioSetup"
            step_key = "minioSetup.compute"
2020-09-18 19:35:17 - dagster - DEBUG - ephemeral_minioSetup_solid_pipeline - 2f1a827a-f1b7-49ae-b9c3-00942931e28d - STEP_INPUT - Got input "dstFilename" of type "String". (Type check passed).
 event_specific_data = {"input_name": "dstFilename", "type_check_data": [true, "dstFilename", null, []]}
                 pid = 81642
               solid = "minioSetup"
    solid_definition = "minioSetup"
            step_key = "minioSetup.compute"
2020-09-18 19:35:17 - dagster - DEBUG - ephemeral_minioSetup_solid_pipeline - 2f1a827a-f1b7-49ae-b9c3-00942931e28d - STEP_INPUT - Got input "srcStopwordsFilename" of type "String". (Type check passed).
 event_specific_data = {"input_name": "srcStopwordsFilename", "type_check_data": [true, "srcStopwordsFilename", null, []]}
                 pid = 81642
               solid = "minioSetup"
    solid_definition = "minioSetup"
            step_key = "minioSetup.compute"
2020-09-18 19:35:17 - dagster - DEBUG - ephemeral_minioSetup_solid_pipeline - 2f1a827a-f1b7-49ae-b9c3-00942931e28d - STEP_INPUT - Got input "dstStopwordsFilename" of type "String". (Type check passed).
 event_specific_data = {"input_name": "dstStopwordsFilename", "type_check_data": [true, "dstStopwordsFilename", null, []]}
                 pid = 81642
               solid = "minioSetup"
    solid_definition = "minioSetup"
            step_key = "minioSetup.compute"
2020-09-18 19:35:17 - dagster - ERROR - ephemeral_minioSetup_solid_pipeline - 2f1a827a-f1b7-49ae-b9c3-00942931e28d - STEP_FAILURE - Execution of step "minioSetup.compute" failed.
            cls_name = "DagstermillError"
       error_message = "dagstermill.errors.DagstermillError: Can't execute a dagstermill solid from a pipeline that is not reconstructable. Use the reconstructable() function if executing from python\n"
                 pid = 81642
               solid = "minioSetup"
    solid_definition = "minioSetup"
            step_key = "minioSetup.compute"

dagstermill.errors.DagstermillError: Can't execute a dagstermill solid from a pipeline that is not reconstructable. Use the reconstructable() function if executing from python

  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/execution/plan/execute_plan.py", line 213, in _dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/execution/plan/execute_step.py", line 269, in core_dagster_event_sequence_for_step
    for user_event in check.generator(
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/execution/plan/execute_step.py", line 54, in _step_output_error_checked_user_event_sequence
    for user_event in user_event_sequence:
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/execution/plan/execute_step.py", line 404, in _user_event_sequence_for_step_compute_fn
    for event in gen:
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/execution/plan/compute.py", line 100, in _execute_core_compute
    for step_output in _yield_compute_results(compute_context, inputs, compute_fn):
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/execution/plan/compute.py", line 71, in _yield_compute_results
    for event in user_event_sequence:
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagstermill/solids.py", line 172, in _t_fn
    get_papermill_parameters(system_compute_context, inputs, output_log_path),
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagstermill/solids.py", line 104, in get_papermill_parameters
    raise DagstermillError(

2020-09-18 19:35:17 - dagster - ERROR - ephemeral_minioSetup_solid_pipeline - 2f1a827a-f1b7-49ae-b9c3-00942931e28d - PIPELINE_FAILURE - Execution of pipeline "ephemeral_minioSetup_solid_pipeline" failed.
                 pid = 81642
Traceback (most recent call last):
  File "dagstermill_nb_pipeline.py", line 201, in <module>
    from dagster import execute_solid; execute_solid(minioSetup)
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/utils/test/__init__.py", line 315, in execute_solid
    result = execute_pipeline(
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/execution/api.py", line 311, in execute_pipeline
    return _logged_execute_pipeline(
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/telemetry.py", line 89, in wrap
    result = f(*args, **kwargs)
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/execution/api.py", line 362, in _logged_execute_pipeline
    return execute_run(pipeline, pipeline_run, instance, raise_on_error=raise_on_error)
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/execution/api.py", line 161, in execute_run
    event_list = list(_execute_run_iterable)
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/execution/api.py", line 722, in __iter__
    for event in self.iterator(
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/execution/api.py", line 675, in _pipeline_execution_iterator
    for event in _core_execution_iterator(
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/execution/api.py", line 642, in _core_execution_iterator
    for event in pipeline_context.executor.execute(pipeline_context, execution_plan):
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/executor/in_process.py", line 36, in execute
    for event in inner_plan_execution_iterator(pipeline_context, execution_plan):
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/execution/plan/execute_plan.py", line 76, in inner_plan_execution_iterator
    for step_event in check.generator(
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/execution/plan/execute_plan.py", line 282, in _dagster_event_sequence_for_step
    raise dagster_error
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/execution/plan/execute_plan.py", line 213, in _dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/execution/plan/execute_step.py", line 269, in core_dagster_event_sequence_for_step
    for user_event in check.generator(
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/execution/plan/execute_step.py", line 54, in _step_output_error_checked_user_event_sequence
    for user_event in user_event_sequence:
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/execution/plan/execute_step.py", line 404, in _user_event_sequence_for_step_compute_fn
    for event in gen:
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/execution/plan/compute.py", line 100, in _execute_core_compute
    for step_output in _yield_compute_results(compute_context, inputs, compute_fn):
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/execution/plan/compute.py", line 71, in _yield_compute_results
    for event in user_event_sequence:
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagstermill/solids.py", line 172, in _t_fn
    get_papermill_parameters(system_compute_context, inputs, output_log_path),
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagstermill/solids.py", line 104, in get_papermill_parameters
    raise DagstermillError(
dagstermill.errors.DagstermillError: Can't execute a dagstermill solid from a pipeline that is not reconstructable. Use the reconstructable() function if executing from python
m
sigh ok
that's an annoying hole we need to fill
k
well at least we found that there is a hole, and that i'm not going crazy, or being an idiot 😄
m
sorry, i mean in execute_solid
that's not the issue that you're facing
k
womp womp, spoke too soon
enlighten me max
m
would you be ok sharing the source of your project? i can't repro with toy examples which makes me think this is something that we're not even considering
k
for sure, it's nothing fancy just trying out a quick LDA example
m
cool if you can zip it up and send it to me that would be amazing
DM is fine or here if its not sensitive
k
sent
m
ok i can repro, give me a sec
ok i don't think this is actually anything very esoteric
i think it's a real error
i was led a little astray by the naming but
you've named the 
LDA
solid 
ETL_solid
the error message correctly says: 
In @pipeline LDA_pipeline, received too many inputs for solid invocation ETL_solid. Only 0 defined, received 1
because you're calling it with the output from 
ETL
, but 
LDA
has no inputs defined
Copy code
LDA = dm.define_dagstermill_solid(
    "ETL_solid",
    "dagstermill_02.04_3_LDA_solid_ML_demo.ipynb",
    config_schema={
        "minioPath": Field(
            str,
            default_value="<http://minio2.studio.azure.art>",
            is_required=False,
            description="URL path to the minio server",
        ),
        "bucketName": Field(
            str,
            default_value="spark-demo-data",
            is_required=False,
            description="S3 bucket name",
        ),
        "pklPath": Field(
            str,
            default_value="./spark-demo-data.pkl",
            is_required=False,
            description="Path to pkl containing dataframe",
        ),
    },
)
k
oh boy -- i knew it would be a stupid error
i was reading it expecting it to come from the ETL solid, forgot to change the name :s
well thanks max, I appreciate teh second set of eyes
so that cleared up the first error but has instead led me to another:
Copy code
2020-09-18 20:31:09 - dagster - ERROR - LDA_pipeline - 573f0ffb-129c-470e-9827-bd0e756fa372 - STEP_FAILURE - Execution of step "minioSetup.compute" failed.
            cls_name = "CheckError"
       error_message = "dagster.check.CheckError: Failure condition: Can not produce an instance reference for <dagster.core.instance.DagsterInstance object at 0x7fa36b6fc970>\n"
                 pid = 88931
               solid = "minioSetup"
    solid_definition = "minioSetup"
            step_key = "minioSetup.compute"

dagster.check.CheckError: Failure condition: Can not produce an instance reference for <dagster.core.instance.DagsterInstance object at 0x7fa36b6fc970>

  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/errors.py", line 181, in user_code_error_boundary
    yield
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/execution/plan/execute_step.py", line 404, in _user_event_sequence_for_step_compute_fn
    for event in gen:
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/execution/plan/compute.py", line 100, in _execute_core_compute
    for step_output in _yield_compute_results(compute_context, inputs, compute_fn):
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/execution/plan/compute.py", line 71, in _yield_compute_results
    for event in user_event_sequence:
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagstermill/solids.py", line 172, in _t_fn
    get_papermill_parameters(system_compute_context, inputs, output_log_path),
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagstermill/solids.py", line 136, in get_papermill_parameters
    parameters["__dm_instance_ref_dict"] = pack_value(compute_context.instance.get_ref())
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/instance/__init__.py", line 305, in get_ref
    check.failed("Can not produce an instance reference for {t}".format(t=self))
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/check/__init__.py", line 109, in failed
    raise_with_traceback(CheckError("Failure condition: {desc}".format(desc=desc)))
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/future/utils/__init__.py", line 446, in raise_with_traceback
    raise exc.with_traceback(traceback)
any idea what i could be missing?
m
yep yep, i assume you're using 
execute_pipeline
?
you will need to explicitly pass the instance object
instance=DagsterInstance.get()
k
is that placed within the solids pipeline or ipynb files? i'm assuming pipeline? and the command i run is 
dagster pipeline execute
i added it in the pipeline i got more or less the same error...
m
hm, i see
sorry about all this
is the environment variable 
DAGSTER_HOME
set?
k
oh no worries, i'm happy to go threw the growing pains
m
if not, maybe set that
k
yeah that fixed that - didn't have it set in this conda env but now further into the rabbit hole gives me this:
Copy code
2020-09-18 21:06:50 - dagster - ERROR - LDA_pipeline - b3dc7196-f8e3-40f4-b448-e1f3f3b6f466 - STEP_FAILURE - Execution of step "minioSetup.compute" failed.
            cls_name = "PapermillExecutionError"
       error_message = "papermill.exceptions.PapermillExecutionError: \n---------------------------------------------------------------------------\nException encountered at \"In [13]\":\n---------------------------------------------------------------------------\nNameError                                 Traceback (most recent call last)\n<ipython-input-13-9ae53e464281> in <module>\n----> 1 dm.yield_result(base_path, output_name='base_path')\n\nNameError: name 'dm' is not defined\n\n"
                 pid = 93494
               solid = "minioSetup"
    solid_definition = "minioSetup"
            step_key = "minioSetup.compute"

papermill.exceptions.PapermillExecutionError:
---------------------------------------------------------------------------
Exception encountered at "In [13]":
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-13-9ae53e464281> in <module>
----> 1 dm.yield_result(base_path, output_name='base_path')

NameError: name 'dm' is not defined


  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagster/core/errors.py", line 181, in user_code_error_boundary
    yield
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagstermill/solids.py", line 218, in _t_fn
    raise exc
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/dagstermill/solids.py", line 187, in _t_fn
    papermill.execute_notebook(
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/papermill/execute.py", line 108, in execute_notebook
    raise_for_execution_errors(nb, output_path)
  File "/home/kevin/anaconda3/envs/dagstermill/lib/python3.8/site-packages/papermill/execute.py", line 192, in raise_for_execution_errors
    raise error
which is really confusing as earlier in the ipynb file it very clearly has:
Copy code
import dagstermill as dm
curious, apparently it has to be defined within that particular cell... is that intended?
m
hm, no, it shouldn't be necessary
can you look at the output notebook file?
just to check that everything executed in order
4 Views
Open in Slack
PreviousNext
Powered by