How can I get the output of a notebook in dagster through da

Question

How can I get the output of a notebook in dagster through dagstermill I ve followed the guide and API docs on the site but I keep running into this errror ```dagster core errors DagsterExecutionStepExecutionError Error occurred while executing solid dbt job The above exception was caused by the following exception KeyError 55fafb57 29ab 4d0e a7b2 c6a30bcdf1db dbt job dbt job status File C Users aafendi Anaconda3 envs dagster environment lib site packages dagster core execution plan utils py line 44 in solid execution error boundary yield File C Users aafendi Anaconda3 envs dagster environment lib site packages dagster utils init py line 383 in iterate with context next output = next iterator File C Users aafendi Anaconda3 envs dagster environment lib site packages dagstermill solids py line 268 in t fn build input context upstream output=output context File C Users aafendi Anaconda3 envs dagster environment lib site packages dagster core storage mem io manager py line 14 in load input return self values keys ```

prha · Answer

Hi Anaqi Can you share your job definition You may need to specify an IO manager that is not the in memory IO manager we have to pass inputs across a process boundary to a dagstermill op We have the `fs io manager` in the `dagster` module that could probably work here ```from dagster import fs io manager job resource defs= io manager fs io manager def my job ```

Anaqi Afendi · Answer

```dbt job = dm define dagstermill solid name= dbt job notebook path=script relative path modules dbt job ipynb input defs= InputDefinition network status complete bool description= Toy variable output defs= OutputDefinition str name= dbt job status description= The status of the dbt job run required resource keys = fs description= A notebook that triggers a dbt cloud job then polls the run to check for its completion ```

prha · Answer

do you have an io manager specified in your pipeline s resources do you mind sharing your pipeline definition

Anaqi Afendi · Answer

``` pipeline mode defs= ModeDefinition resource defs= fs fs io manager file manager local file manager io mem io manager def test pipeline update network status = update network files dbt job status = dbt job update network status take dagstermill output dbt job status ```

Anaqi Afendi · Answer

I tried running the dagstermill solid with the file manager resource key instead and got the same error

prha · Answer

I see can you add ` io manager fs io manager` to your resource defs

Anaqi Afendi · Answer

And should I use io manager as the required resource

Anaqi Afendi · Answer

for the dagstermill solid

Anaqi Afendi · Answer

Okay this worked thats kind of wierd it had to be called io manager in the key since that name shouldn t matter unless dagstermill doesn t have a way to configure what the resource key it uses as the fs io manager is

prha · Answer

yeah io manager is a special cased resource there s some doc on it here <https docs dagster io concepts io management io managers io managers>

Anaqi Afendi · Answer

Would it be better to migrate my code if possible from ipynb and into a function call if that means it can be more flexible I just really like being able to see the actual code that runs in each block within the DAG in dagster when its in a notebook format

prha · Answer

It could be beneficial yes to split them out as individual functions you could then make use of solid level retries and other solid based features

Anaqi Afendi · Answer

Awesome thanks

prha · Answer

I think one case where you would want to keep things in notebooks is if the notebook itself contains artifacts that are important to keep around e g contains some plots visualizations

prha · Answer

in that case you probably want to use the file manager to persist the evaluated notebook to some stored location

Anaqi Afendi · Answer

unrelated or kind of Is there a way to specify dependancies for solids without using I O I have a bunch of solids dependant on one another to run in sequence but they are all just triggers for dbt databricks jobs and rn I m passing dummy variables between them

prha · Answer

no the main way to specify dependencies is using I O But we do have this concept of a `Nothing` output which allows you to specify the dependency without actually passing any data between the solids <https docs dagster io concepts solids pipelines pipelines order based dependencies nothing dependencies>

prha · Answer

< Dagster Bot> issue detect when an in memory IO manager is used for jobs using dagstermill ops and raise a better error message

Dagster Bot · Answer

Created issue at <https github com dagster io dagster issues 5438>