to be more explicit about where i m starting from i have a b dagster #ask-community

to be more explicit about where i’m starting from:...

William Grisaitis

02/23/2023, 10:05 PM

to be more explicit about where i’m starting from: i have a bunch of functions with signatures like

def make_thing(df1: pd.DataFrame, df2: pd.DataFrame, param1, param2) -> pd.DataFrame: ...

, and i want to use dagster to call these and memoize results. EDIT: made this question shorter 🙂

claire

02/25/2023, 12:05 AM

Hi William. Something to note is that in Dagster assets expect to deterministically materialize based on the upstream assets. What this means is that memoization for assets relies on checking if the input data is the same as it was for the previous materialization, and then skipping redundant materializations if the input data and computation are both the same. We do have a guide that describes this in more detail. Since in Dagster we only compare against the last materialization for memoization, assets might not be the best use case for you since the inputs to your

make_thing

functions can change each time.

👍 1

claire

02/25/2023, 12:07 AM

I think one way to use your library with dagster is to create `@op`s, you can create your own functions e.g.

Copy code

@op 
def my_make_thing_fn(input_1, input_2):
   return make_thing(input_1, input_2):

if you want to use your functions within jobs without decorating them.

👍 1

William Grisaitis

02/28/2023, 11:44 PM

thank you SO much for taking the time to help me out on this! i was thinking wrapping things in ops might make more sense than assets… if my vague memory is right. thanks for sharing the guide, as well. i’ll do some more reading to educate myself on this, and understad what you’re saying about assets.

claire

02/28/2023, 11:45 PM

no problem! feel free to drop any more questions in the support channel and we'll help you out 🌈

🙂 1

Open in Slack

Previous Next