19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
dagster-de
dagster-cube
gigs-freelance
dagster-wandb
dagster-machine-learning
dagster-bigquery
tools
dagster-dask
dagster-snowflake
dagster-releases
data-platform-design
events
dagster-serverless
dagster-noteable
introductions
dagster-airflow
dagster-cloud
random
dagster-dbt
faq-read-me-before-posting
dagster-airbyte
dagster-spatial
dagster-support
announcements
github-discussions
豆瓣酱帮
dagster-feedback
dagstereo
dagster-ecs
dagster-kubernetes
jobs
dagster-showcase
Title
m
Michel Rouly
07/12/2021, 12:23 PMHey folks. Trying out the Memoizable IO Manager and running into what seems like potentially a bug in output creation. I've got this sample code (pasting in thread) using
versioned_filesystem_io_managerdagster.check.ParameterCheckError: Param "context.version" is not a str. Got None which is type <class 'NoneType'>.from dagster import repository, solid, pipeline, ModeDefinition
from dagster.core.storage.memoizable_io_manager import versioned_filesystem_io_manager
@solid(version="v0")
def test_solid() -> str:
return "hello versioned world"
test_mode = ModeDefinition("test_mode", resource_defs={"io_manager": versioned_filesystem_io_manager})
@pipeline(mode_defs=[test_mode])
def test_pipeline():
test_solid()
@repository()
def test_repo():
return [test_pipeline]I was looking around in this slack channel history and I found this issue, but AFAICT I've only got one object that requires versioning (the solid), no other resources.
c
chris
07/12/2021, 2:34 PMyou need to set the
MEMOIZED_RUN_TAGfrom dagster.core.storage.tags import MEMOIZED_RUN_TAG
@pipeline(
mode_defs=[
ModeDefinition("memoized", resource_defs={"io_manager": versioned_filesystem_io_manager}),
],
tags={MEMOIZED_RUN_TAG: "true"},
)
def my_pipeline():
return emit_sentence(emit_dog(), emit_tree())m
Michel Rouly
07/12/2021, 2:34 PMGotcha!
Is that documented somewhere and I just missed it?
c
chris
07/12/2021, 2:34 PMm
Michel Rouly
07/12/2021, 2:35 PMGotcha! I didn't actually know about that page, was just looking at the API docs. Good to know.
c
chris
07/12/2021, 2:35 PMYup! No worries. Definitely a bit buried
m
Michel Rouly
07/12/2021, 2:50 PMHm, that didn't seem to do the trick either.
@pipeline(mode_defs=[test_mode], tags={"MEMOIZED_RUN_TAG": "true"})
def test_pipeline():
test_solid()Oh I'm silly, I used a string literal instead of the import. One moment 🙂
c
chris
07/12/2021, 2:52 PMno worries: let me know if that fixes
m
Michel Rouly
07/12/2021, 2:53 PMYep that worked. Although it unfortunately highlights this sentence in that doc page you linked:
Note that memoized execution is not yet supported from Dagit.
Tag "dagster/is_memoized_run" was found when initializing pipeline run, however, memoized execution is only supported from the dagster CLI. This pipeline will run, but outputs from previous executions will be ignored. In order to execute this pipeline using memoization, provide the "dagster/is_memoized_run" tag to the `dagster pipeline execute` CLI. The CLI is documented at the provided link.Thanks for the assist. Guess I'll wait until support for memoization in Dagit is added 🙂
c
chris
07/12/2021, 3:48 PMyup - definitely very limited in its support right now but working on changing that.
🙌 1