Has someone seen this error I think i am struggling to disam dagster #ask-community

Has someone seen this error? I think i am struggli...

Rahul Dave

01/30/2023, 3:39 PM

Has someone seen this error? I think i am struggling to disambiguate the op from the graph from the asset backed by the graph...

jamie

01/30/2023, 4:19 PM

Hi @Rahul Dave what’s the output you’re trying to return from the notebook? you aren’t required to include the line

Copy code

dagstermill.yield_result(out, output_name="encoders")

in order to turn a notebook into an op

Rahul Dave

01/30/2023, 4:23 PM

@jamie the dagstermill docs sat to do this here: https://docs.dagster.io/integrations/dagstermill/reference#results-and-custom-materializations

Rahul Dave

01/30/2023, 4:24 PM

the notebook has a dictionary in

out

which i would like to save as a software designed asset, to consume in downstream ops

Rahul Dave

01/30/2023, 4:25 PM

this is also why i am not using the output notebook as a SDA (i dont care for the output notebook, only the data in the

out

variable

jamie

01/30/2023, 4:27 PM

ok that makes sense. so right now, dagstermill assets don’t support yielding results for reasons outlined in this github issue when you create a dagstermill op and wrap it in a graph and then turn that graph into an assets, it’s still basically turning the dagstermill op into a dagstermill asset, so then yield_result become unsupported. Adding yield_result support to the dagstermill asset is in my backlog, but i dont have a firm time when i’ll be able to work on it.

Rahul Dave

01/30/2023, 4:30 PM

Interestingly even adding a downstream no-op did not affect this

Rahul Dave

01/30/2023, 4:30 PM

Copy code

```
encoder_op = define_dagstermill_op(
    name="encoder_op",
    notebook_path=file_relative_path(__file__, "../notebooks/encoder.ipynb"),
    output_notebook_name="output_encoder",
    outs={"encoders": Out(dict)},
    ins={"df_train": In(pd.DataFrame), "df_test": In(pd.DataFrame)}
)

@op
def pass_thru(encoders):
    return encoders

@graph(out = {'result': GraphOut()})
def encoder_graph(df_train, df_test):
    encoders, _ = encoder_op(df_train, df_test)
    result =  pass_thru(encoders)
    return result

encoder_asset = AssetsDefinition.from_graph(encoder_graph,
    keys_by_input_name={"df_train": AssetKey("train_dataset"), "df_test": AssetKey("test_dataset")},
    keys_by_output_name={"result": AssetKey("encoders_asset")}
)
```

Rahul Dave

01/30/2023, 4:31 PM

So the "assetization" is passed back through the graph!!

Rahul Dave

01/30/2023, 4:32 PM

Is there a work-around? How do you suggest i approach this? Wrap papermill in a python executor using their python api? or use events..is there an example of this?

Rahul Dave

01/30/2023, 4:33 PM

Since events will create AssetMaterializations I would lose the graph structure...

Rahul Dave

01/30/2023, 4:34 PM

(BTW incase you are wondering, i am trying to wrap an entire machine learning pipeline into dagster)

jamie

01/30/2023, 4:34 PM

AssetMaterialization events would work, but yes you’d loose the graph structure. For full support we need to make a version of

dagstermill_asset

that uses multi asset instead of a single asset

Rahul Dave

01/30/2023, 4:40 PM

I guess the other option is to forgo assets and run as a graph of ops with the file-interchange done by convention

jamie

01/30/2023, 4:42 PM

yeah that would also be an option

Rahul Dave

01/30/2023, 4:46 PM

That does seem to lose the core SDA advantage of dagster though (although that is lost in event driven materialization too!)

jamie

01/30/2023, 4:49 PM

yeah for sure. for what it’s worth, being able to yield results from asset notebooks is a feature i’d really like to work on, but i have higher priority features at the moment. l may be able to find some time to squeeze it in, but i can’t make any promises. If you want to join the #dagster-noteable channel, i’ll announce new dagstermill features there

❤️ 1

Rahul Dave

01/30/2023, 4:50 PM

will join! Please let me know how i can help. I think i will go with ops and graphs for now and upgrade to the assets when you give me the green light

Chris Nogradi

02/07/2023, 7:01 PM

I just ran into this issue as well. And would love to see it resolved. Maybe a quick solution would be to not yield the resulting completed notebook?

jamie

02/08/2023, 2:22 PM

The code change to do that might be as intensive as converting to a multi-asset. i can look into it, but i can’t make any promises

Chris Nogradi

02/08/2023, 2:24 PM

Thanks @jamie

jamie

02/08/2023, 2:25 PM

i do want to reiterate that i want to make this feature! i just have some high priority things i’m working on right now that i can’t put on hold

Rahul Dave

02/08/2023, 5:23 PM

Thanks @jamie!

Chris Nogradi

04/11/2023, 9:54 PM

@jamie have you been able to work this feature?

jamie

04/12/2023, 1:33 PM

not yet, unfortunately. I recommend following the github issue i linked earlier in the thread for updates!

Rahul Dave

04/12/2023, 1:42 PM

Lemme add my voice to the chorus for the issue! There is a whole set of apps this will unleash!

Chris Nogradi

04/12/2023, 9:04 PM

@jamie oops sorry, I will watch the issue!

jamie

04/12/2023, 9:05 PM

no worries! easy to get lost so far up the thread

Open in Slack

Previous Next