19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
announcements
dagster-airbyte
dagster-airflow
dagster-bigquery
dagster-cloud
dagster-cube
dagster-dask
dagster-dbt
dagster-de
dagster-ecs
dagster-feedback
dagster-kubernetes
dagster-machine-learning
dagster-noteable
dagster-releases
dagster-serverless
dagster-showcase
dagster-snowflake
dagster-spatial
dagster-support
dagster-wandb
dagstereo
data-platform-design
events
faq-read-me-before-posting
gigs-freelance
github-discussions
introductions
jobs
random
tools
豆瓣酱帮
Title
c
Caleb Parnell Lampen
04/28/2023, 4:27 PMI'm finding when I increase the `code_version`of an asset, I'm not seeing any indication of the asset being stale in dagit. Is there something I need to do to make that appear other than reloading the definitions?
I'm using dagster 1.3.1. This is the first time I've experimented with staleness.
s
sean
04/28/2023, 5:28 PMHi Caleb, That’s odd. If you increment the code version and reload definitions, the asset should get a “New Code Version” tag
Are you by any chance using the automaterialize daemon?
I’m wondering if it’s being quickly detected as outdated and materialized in the background without you realizing
c
Caleb Parnell Lampen
04/28/2023, 6:05 PMSo, in the code, the version was
v5I realize I'm sparse on details here, I'm not sure what to mention...
A few odd things I might be doing different than typical. I have a partition mapping between static partitions. An upstream asset has a single partition that is mapped to multiple partitions for a downstream asset. Additionally, these partitions are defined by a factory function, regenerated whenever the code location is reloaded (they are dynamically generated based upon the files present on disk).
I don't know why any of that would impact code_version behavior, but I get the impression that partitions are relatively new, so I figure its worth mentioning.
s
sean
04/28/2023, 6:54 PMAh, yeah partitions do not play nicely with data versions right now. This is an actively area of focus.
Previously, the “staleness” system was totally disabled for partitions. When we broke up “stale” into separate indicators for “New code version” and “New data version”, looks like we kept both tags disabled for partitioned assets, but we should be able to enable easily for code versions for next week’s release (tracking upstream data versions with partitions is the hard case because there are potentially thousands of dependencies).
So thanks for the report, this is a bug and I will fix it.
c
Caleb Parnell Lampen
04/28/2023, 8:05 PMAh, interesting.
Perhaps related: I'm getting an erroneous "Upstream data missing" warning message when I go to the materialize dialog for the downstream asset.
There are actually three assets involved here, which I'll named Upstream1, Upstream2, and Downstream:
Upstream1: Uses CoursePartition
Upstream2: Uses FinePartition
Downstream: Uses FinePartition.
CoursePartition maps to FinePartition in a one to many mapping. If I have Upstream1 already materialized, and try to start a job to materialize Upstream2 and Downstream, I get an "Upstream data missing" warning message in the materlialize dialog box. However, the job executes without problem if I go ahead anyway.
An additional piece of feedback here: The "Upstream data missing" warning doesn't say what asset it thinks is missing, so I don't know if it is complaining about Upstream1 or Upstream2.
Also, you can't create a job or run that materializes all three together, which would be convenient, but I think you already know about that "problem".