👋 I'm trying to use

MemoizableIOManager

but I having hard time, and right now hitting 2 issues/questions, I would highly appreciate your help. I'm on dagster==0.10.5. 1.

has_output

appears to be getting called with context set to

None

, thus I can't validate if the external data exists, this is a simplified version of my current POC: https://gist.github.com/ravwojdyla/6a546e3fa65459b17413aac10c643f25 2. when I hard-code

had_output

to return

True

(just as a POC), I'm getting this error:

for step in self.get_steps_to_execute_by_level()[0]:
IndexError: list index out of range

I was expecting to get an error in that case, but that specific error made me think how the memoization appears to work, is it fair to say that for memoization to work the "central" scheduler needs to have access the the previously run "steps" (to fetch config etc)?

hey! thanks for giving memoization a shot. Can I get more information regarding how you're executing? As in, are you using the python API, CLI, etc?

👋 @chris executing via python API and CLI (tried both)

hey! so I was able to reproduce the second error

@chris do you mind if I create an issue for the 2nd problem? that way I can report that back?

yup that's fine

@chris re 1st issue q: • correct - no version - is it required?

ah, so yea it's required that you provide a version attribute to your solids, which essentially represents the version of the code enclosed by the solid. You can set it to some fixed value, a la

@solid(version="hello")

to get up and running

@chris I see, added version. just FYI as you probably expect that doesn't mitigate the issue of

context.log

being

None

, maybe some kind of init is not done before

has_output

?

Right. regarding the logging issue, that is definitely a bug.

@chris ok, will create an issue for that as well

Appreciated! Will try to get fixes out for these soon

@chris thanks for your help thus far btw, going back to the q:

I was expecting to get an error in that case, but that specific error made me think how the memoization appears to work, is it fair to say that for memoization to work the "central" scheduler needs to have access the the previously run "steps" (to fetch config etc)?

Right, so this error is actually happening downstream of the memoization process

@chris right, for context I am currently evaluating whether we should start using Dagster as an orchestrator, so I'm trying grasp the assumptions dagster makes about memoization. So for memoization to work, Dagster needs information about previous runs of the pipeline or not?

In theory, it should not. Memoization is designed not to be tied to runs, but instead to computed versions for a given output.

@chris in theory? is the practice different than theory?

Sorry, in theory was a poor choice of words. "It should not". Is a more accurate way of saying it.

@chris cool, thanks! FYI the 2nd issue: https://github.com/dagster-io/dagster/issues/3690. Speaking of the memoization, I'm sorry I have all these questions, but I don't think there is documentation for this (???), but my 2nd question is about memoization depending on the value of parameters (given from CLI), is there a documentation or example you could point me at?

To give a brief primer on the way memoization works 1. Given the full execution plan, we assign a "version" to each output in the pipeline. This version is dependent upon all upstream versions in the pipeline. 2. For each output, we use the io manager to determine whether or not an object has already been stored with the version tied to that output. If a given output/version has not been stored, then that output + all downstream outputs to that output must be recomputed. 3. We execute the set of steps that have not been memoized. For steps that expect inputs from steps that have cached outputs, we just use the io manager to retrieve those inputs.

@chris yea, so full discloser, I been reading the doc and playing with dagster for only a couple of hours, and I think the doc is a bit all over the place, it's missing a kind of "golden/blessed path" example with a "standard" data pipeline with "best data engineering practices". And by standard data pipeline, I mean a pipeline that works with files (which are "memoized"), multiple idempotent steps. In the example there should be at least 2 such pipelines and some data dependencies between them.

so are you referring to the dagster docs in general, or the memoization doc specifically?

@chris dagster doc in general, there's airline example, that's too complex as an example to illustrate best practices?

ah gotcha. So memoization is still an experimental feature, and only works from the CLI. Thus, we don't really expose it in examples intended to reflect more idiomatic usage yet

@chris so would you agree with the statement that right now dagster doesn't have idiomatic support for memoized file outputs?

Memoization is definitely designed around using something like file outputs, I think it's more accurate to say that memoization in general is still quite rough around the edges

@chris I see, could you please elaborate why? there is obvious cost to it (and I mean readability). And maybe it's worth to scope the problem: I can understand why you would like that in a super general case of an orchestrator, BUT if we focus on an idiomatic data pipeline (idempotent, ETL with date partitioned output), in most cases you don't need to recompute on code changes (apart from limited special cases), since you just produce new data in a new date partition? Am I missing sth that dagster assumes?

Right, if you don't care about recomputing on code changes, then you can hard code the version argument and then recomputation will only happen as inputs and config change