question about dagstermill with passing results i ve been fo

Question

question about dagstermill with passing results i ve been following the examples provided but I seem to have run into an issue when passing my results from one dagstermill solid to another This is the typical error received ```dagster core errors DagsterInvalidDefinitionError In pipeline LDA pipeline received invalid type <class str > for input base path passed by keyword in solid invocation ETL Must pass the output from previous solid invocations or inputs to the composition function as inputs when invoking solids during composition ``` just trying to create some pipelines with dagstermill attached is my py which defines my solids each of the notebooks does execute `dagstermill yield result `

max · Answer

why are you calling `str` on line 196

Kevin · Answer

prior to declaring it as type str it was returning invalid type lt class NoneType gt the output is supposed to be a string from the dagstermill solid

max · Answer

can i see the contents of the cell in the `dagstermill 02 04 1 minio solid ML demo ipynb` notebook that calls `yield result`

Kevin · Answer

dm yield result base path output name= base path

Kevin · Answer

i didn t see any examples of dm yield result so I will admit to some ad libbing here

Kevin · Answer

am i missing something on how inputs are defined for dagstermill solids as the other error that i receive is ```dagster core errors DagsterInvalidDefinitionError In pipeline LDA pipeline received too many inputs for solid invocation ETL solid Only 0 defined received 1```

max · Answer

can i see the logs produced by the invocation of minioSetup

Kevin · Answer

like the errors produced from trying to run the pipeline

max · Answer

yeah the logs that are produced by that solid when you run the pipeline

Kevin · Answer

Okay here we are I removed the `str ` and some minor changes and this is the output

Kevin · Answer

```dagster pipeline execute f dagstermill nb pipeline py a LDA pipeline home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core definitions mode py 64 UserWarning system storage defs are deprecated and will be removed in 0 10 0 and should be replaced with intermediate storage defs for intermediates and resource defs for files warnings warn Traceback most recent call last File home kevin anaconda3 envs dagstermill bin dagster line 8 in lt module gt sys exit main File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster cli init py line 38 in main cli obj= pylint disable=E1123 File home kevin anaconda3 envs dagstermill lib python3 8 site packages click core py line 829 in call return self main args kwargs File home kevin anaconda3 envs dagstermill lib python3 8 site packages click core py line 782 in main rv = self invoke ctx File home kevin anaconda3 envs dagstermill lib python3 8 site packages click core py line 1259 in invoke return process result sub ctx command invoke sub ctx File home kevin anaconda3 envs dagstermill lib python3 8 site packages click core py line 1259 in invoke return process result sub ctx command invoke sub ctx File home kevin anaconda3 envs dagstermill lib python3 8 site packages click core py line 1066 in invoke return ctx invoke self callback ctx params File home kevin anaconda3 envs dagstermill lib python3 8 site packages click core py line 610 in invoke return callback args kwargs File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster cli pipeline py line 280 in pipeline execute command execute execute command instance kwargs File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core telemetry py line 89 in wrap result = f args kwargs File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster cli pipeline py line 296 in execute execute command pipeline origin = get pipeline python origin from kwargs kwargs File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster cli workspace cli target py line 369 in get pipeline python origin from kwargs repo definition = recon repo get definition File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core definitions reconstructable py line 39 in get definition return repository def from pointer self pointer File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core definitions reconstructable py line 477 in repository def from pointer target = def from pointer pointer File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core definitions reconstructable py line 426 in def from pointer target = pointer load target File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core code pointer py line 255 in load target module = load python file self python file self working directory File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core code pointer py line 134 in load python file module = import module from path module name python file File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster seven init py line 116 in import module from path spec loader exec module module File lt frozen importlib bootstrap external gt line 783 in exec module File lt frozen importlib bootstrap gt line 219 in call with frames removed File dagstermill nb pipeline py line 193 in lt module gt def LDA pipeline File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core definitions decorators pipeline py line 112 in pipeline return Pipeline name File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core definitions decorators pipeline py line 37 in call fn File dagstermill nb pipeline py line 195 in LDA pipeline LDA ETL base path=bp File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core definitions solid py line 149 in call return CallableSolidNode self args kwargs File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core definitions composition py line 156 in call raise DagsterInvalidDefinitionError dagster core errors DagsterInvalidDefinitionError In pipeline LDA pipeline received too many inputs for solid invocation ETL solid Only 0 defined received 1```

max · Answer

can you run for me `from dagster import execute solid execute solid minioSetup `

Kevin · Answer

``` gt gt gt import dagstermill nb pipeline py home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core definitions mode py 64 UserWarning system storage defs are deprecated and will be removed in 0 10 0 and should be replaced with intermediate storage defs for intermediates and resource defs for files warnings warn Traceback most recent call last File lt stdin gt line 1 in lt module gt File home kevin workspace dm dagstermill nb pipeline py line 193 in lt module gt def LDA pipeline File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core definitions decorators pipeline py line 112 in pipeline return Pipeline name File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core definitions decorators pipeline py line 37 in call fn File home kevin workspace dm dagstermill nb pipeline py line 195 in LDA pipeline LDA ETL base path=bp File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core definitions solid py line 149 in call return CallableSolidNode self args kwargs File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core definitions composition py line 156 in call raise DagsterInvalidDefinitionError dagster core errors DagsterInvalidDefinitionError In pipeline LDA pipeline received too many inputs for solid invocation ETL solid Only 0 defined received 1 gt gt gt from dagster import execute solid execute solid minioSetup Traceback most recent call last File lt stdin gt line 1 in lt module gt NameError name minioSetup is not defined```

Kevin · Answer

and adding that line to the py file didn t change anything from what i can see

max · Answer

yeah sorry can you comment out the pipeline definition and just try running the minioSetup solid in isolation

max · Answer

using `execute solid`

max · Answer

i just want to establish that it s actually yielding the result you expect

Kevin · Answer

```python dagstermill nb pipeline py home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core definitions mode py 64 UserWarning system storage defs are deprecated and will be removed in 0 10 0 and should be replaced with intermediate storage defs for intermediates and resource defs for files warnings warn 2020 09 18 19 35 17 dagster DEBUG ephemeral minioSetup solid pipeline 2f1a827a f1b7 49ae b9c3 00942931e28d PIPELINE START Started execution of pipeline ephemeral minioSetup solid pipeline pid = 81642 2020 09 18 19 35 17 dagster DEBUG ephemeral minioSetup solid pipeline 2f1a827a f1b7 49ae b9c3 00942931e28d ENGINE EVENT Executing steps in process pid 81642 event specific data = error null marker end null marker start null metadata entries pid null 81642 step keys null minioSetup compute pid = 81642 2020 09 18 19 35 17 dagster DEBUG ephemeral minioSetup solid pipeline 2f1a827a f1b7 49ae b9c3 00942931e28d STEP START Started execution of step minioSetup compute pid = 81642 solid = minioSetup solid definition = minioSetup step key = minioSetup compute 2020 09 18 19 35 17 dagster DEBUG ephemeral minioSetup solid pipeline 2f1a827a f1b7 49ae b9c3 00942931e28d STEP INPUT Got input minioPath of type String Type check passed event specific data = input name minioPath type check data true minioPath null pid = 81642 solid = minioSetup solid definition = minioSetup step key = minioSetup compute 2020 09 18 19 35 17 dagster DEBUG ephemeral minioSetup solid pipeline 2f1a827a f1b7 49ae b9c3 00942931e28d STEP INPUT Got input bucketName of type String Type check passed event specific data = input name bucketName type check data true bucketName null pid = 81642 solid = minioSetup solid definition = minioSetup step key = minioSetup compute 2020 09 18 19 35 17 dagster DEBUG ephemeral minioSetup solid pipeline 2f1a827a f1b7 49ae b9c3 00942931e28d STEP INPUT Got input srcFilename of type String Type check passed event specific data = input name srcFilename type check data true srcFilename null pid = 81642 solid = minioSetup solid definition = minioSetup step key = minioSetup compute 2020 09 18 19 35 17 dagster DEBUG ephemeral minioSetup solid pipeline 2f1a827a f1b7 49ae b9c3 00942931e28d STEP INPUT Got input dstFilename of type String Type check passed event specific data = input name dstFilename type check data true dstFilename null pid = 81642 solid = minioSetup solid definition = minioSetup step key = minioSetup compute 2020 09 18 19 35 17 dagster DEBUG ephemeral minioSetup solid pipeline 2f1a827a f1b7 49ae b9c3 00942931e28d STEP INPUT Got input srcStopwordsFilename of type String Type check passed event specific data = input name srcStopwordsFilename type check data true srcStopwordsFilename null pid = 81642 solid = minioSetup solid definition = minioSetup step key = minioSetup compute 2020 09 18 19 35 17 dagster DEBUG ephemeral minioSetup solid pipeline 2f1a827a f1b7 49ae b9c3 00942931e28d STEP INPUT Got input dstStopwordsFilename of type String Type check passed event specific data = input name dstStopwordsFilename type check data true dstStopwordsFilename null pid = 81642 solid = minioSetup solid definition = minioSetup step key = minioSetup compute 2020 09 18 19 35 17 dagster ERROR ephemeral minioSetup solid pipeline 2f1a827a f1b7 49ae b9c3 00942931e28d STEP FAILURE Execution of step minioSetup compute failed cls name = DagstermillError error message = dagstermill errors DagstermillError Can t execute a dagstermill solid from a pipeline that is not reconstructable Use the reconstructable function if executing from python n pid = 81642 solid = minioSetup solid definition = minioSetup step key = minioSetup compute dagstermill errors DagstermillError Can t execute a dagstermill solid from a pipeline that is not reconstructable Use the reconstructable function if executing from python File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core execution plan execute plan py line 213 in dagster event sequence for step for step event in check generator step events File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core execution plan execute step py line 269 in core dagster event sequence for step for user event in check generator File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core execution plan execute step py line 54 in step output error checked user event sequence for user event in user event sequence File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core execution plan execute step py line 404 in user event sequence for step compute fn for event in gen File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core execution plan compute py line 100 in execute core compute for step output in yield compute results compute context inputs compute fn File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core execution plan compute py line 71 in yield compute results for event in user event sequence File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagstermill solids py line 172 in t fn get papermill parameters system compute context inputs output log path File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagstermill solids py line 104 in get papermill parameters raise DagstermillError 2020 09 18 19 35 17 dagster ERROR ephemeral minioSetup solid pipeline 2f1a827a f1b7 49ae b9c3 00942931e28d PIPELINE FAILURE Execution of pipeline ephemeral minioSetup solid pipeline failed pid = 81642 Traceback most recent call last File dagstermill nb pipeline py line 201 in lt module gt from dagster import execute solid execute solid minioSetup File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster utils test init py line 315 in execute solid result = execute pipeline File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core execution api py line 311 in execute pipeline return logged execute pipeline File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core telemetry py line 89 in wrap result = f args kwargs File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core execution api py line 362 in logged execute pipeline return execute run pipeline pipeline run instance raise on error=raise on error File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core execution api py line 161 in execute run event list = list execute run iterable File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core execution api py line 722 in iter for event in self iterator File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core execution api py line 675 in pipeline execution iterator for event in core execution iterator File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core execution api py line 642 in core execution iterator for event in pipeline context executor execute pipeline context execution plan File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core executor in process py line 36 in execute for event in inner plan execution iterator pipeline context execution plan File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core execution plan execute plan py line 76 in inner plan execution iterator for step event in check generator File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core execution plan execute plan py line 282 in dagster event sequence for step raise dagster error File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core execution plan execute plan py line 213 in dagster event sequence for step for step event in check generator step events File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core execution plan execute step py line 269 in core dagster event sequence for step for user event in check generator File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core execution plan execute step py line 54 in step output error checked user event sequence for user event in user event sequence File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core execution plan execute step py line 404 in user event sequence for step compute fn for event in gen File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core execution plan compute py line 100 in execute core compute for step output in yield compute results compute context inputs compute fn File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core execution plan compute py line 71 in yield compute results for event in user event sequence File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagstermill solids py line 172 in t fn get papermill parameters system compute context inputs output log path File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagstermill solids py line 104 in get papermill parameters raise DagstermillError dagstermill errors DagstermillError Can t execute a dagstermill solid from a pipeline that is not reconstructable Use the reconstructable function if executing from python```

max · Answer

sigh ok

max · Answer

that s an annoying hole we need to fill

Kevin · Answer

well at least we found that there is a hole and that i m not going crazy or being an idiot smile

max · Answer

sorry i mean in execute solid

max · Answer

that s not the issue that you re facing

Kevin · Answer

womp womp spoke too soon

Kevin · Answer

enlighten me max

max · Answer

would you be ok sharing the source of your project i can t repro with toy examples which makes me think this is something that we re not even considering

Kevin · Answer

for sure it s nothing fancy just trying out a quick LDA example

max · Answer

cool if you can zip it up and send it to me that would be amazing

max · Answer

DM is fine or here if its not sensitive

Kevin · Answer

sent

max · Answer

ok i can repro give me a sec

max · Answer

ok i don t think this is actually anything very esoteric

max · Answer

i think it s a real error

max · Answer

i was led a little astray by the naming but

max · Answer

you ve named the `LDA` solid `ETL solid`

max · Answer

the error message correctly says `In pipeline LDA pipeline received too many inputs for solid invocation ETL solid Only 0 defined received 1`

max · Answer

because you re calling it with the output from `ETL` but `LDA` has no inputs defined

max · Answer

```LDA = dm define dagstermill solid ETL solid dagstermill 02 04 3 LDA solid ML demo ipynb config schema= minioPath Field str default value= <http minio2 studio azure art> is required=False description= URL path to the minio server bucketName Field str default value= spark demo data is required=False description= S3 bucket name pklPath Field str default value= spark demo data pkl is required=False description= Path to pkl containing dataframe ```

Kevin · Answer

oh boy i knew it would be a stupid error

Kevin · Answer

i was reading it expecting it to come from the ETL solid forgot to change the name s

Kevin · Answer

well thanks max I appreciate teh second set of eyes

Kevin · Answer

so that cleared up the first error but has instead led me to another ```2020 09 18 20 31 09 dagster ERROR LDA pipeline 573f0ffb 129c 470e 9827 bd0e756fa372 STEP FAILURE Execution of step minioSetup compute failed cls name = CheckError error message = dagster check CheckError Failure condition Can not produce an instance reference for lt dagster core instance DagsterInstance object at 0x7fa36b6fc970 gt n pid = 88931 solid = minioSetup solid definition = minioSetup step key = minioSetup compute dagster check CheckError Failure condition Can not produce an instance reference for lt dagster core instance DagsterInstance object at 0x7fa36b6fc970 gt File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core errors py line 181 in user code error boundary yield File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core execution plan execute step py line 404 in user event sequence for step compute fn for event in gen File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core execution plan compute py line 100 in execute core compute for step output in yield compute results compute context inputs compute fn File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core execution plan compute py line 71 in yield compute results for event in user event sequence File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagstermill solids py line 172 in t fn get papermill parameters system compute context inputs output log path File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagstermill solids py line 136 in get papermill parameters parameters dm instance ref dict = pack value compute context instance get ref File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core instance init py line 305 in get ref check failed Can not produce an instance reference for t format t=self File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster check init py line 109 in failed raise with traceback CheckError Failure condition desc format desc=desc File home kevin anaconda3 envs dagstermill lib python3 8 site packages future utils init py line 446 in raise with traceback raise exc with traceback traceback ```

Kevin · Answer

any idea what i could be missing

max · Answer

yep yep i assume you re using `execute pipeline`

max · Answer

you will need to explicitly pass the instance object

max · Answer

`instance=DagsterInstance get `

Kevin · Answer

is that placed within the solids pipeline or ipynb files i m assuming pipeline and the command i run is `dagster pipeline execute`

Kevin · Answer

i added it in the pipeline i got more or less the same error

max · Answer

hm i see

max · Answer

sorry about all this

max · Answer

is the environment variable `DAGSTER HOME` set

Kevin · Answer

oh no worries i m happy to go threw the growing pains

max · Answer

if not maybe set that

Kevin · Answer

yeah that fixed that didn t have it set in this conda env but now further into the rabbit hole gives me this ```2020 09 18 21 06 50 dagster ERROR LDA pipeline b3dc7196 f8e3 40f4 b448 e1f3f3b6f466 STEP FAILURE Execution of step minioSetup compute failed cls name = PapermillExecutionError error message = papermill exceptions PapermillExecutionError n nException encountered at In 13 n nNameError Traceback most recent call last n lt ipython input 13 9ae53e464281 gt in lt module gt n gt 1 dm yield result base path output name= base path n nNameError name dm is not defined n n pid = 93494 solid = minioSetup solid definition = minioSetup step key = minioSetup compute papermill exceptions PapermillExecutionError Exception encountered at In 13 NameError Traceback most recent call last lt ipython input 13 9ae53e464281 gt in lt module gt gt 1 dm yield result base path output name= base path NameError name dm is not defined File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagster core errors py line 181 in user code error boundary yield File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagstermill solids py line 218 in t fn raise exc File home kevin anaconda3 envs dagstermill lib python3 8 site packages dagstermill solids py line 187 in t fn papermill execute notebook File home kevin anaconda3 envs dagstermill lib python3 8 site packages papermill execute py line 108 in execute notebook raise for execution errors nb output path File home kevin anaconda3 envs dagstermill lib python3 8 site packages papermill execute py line 192 in raise for execution errors raise error``` which is really confusing as earlier in the ipynb file it very clearly has ```import dagstermill as dm```

Kevin · Answer

curious apparently it has to be defined within that particular cell is that intended

max · Answer

hm no it shouldn t be necessary

max · Answer

can you look at the output notebook file

max · Answer

just to check that everything executed in order