I m finding when I increase the `code version`of an asset I dagster #ask-community

I'm finding when I increase the `code_version`of a...

Caleb Parnell Lampen

04/28/2023, 4:27 PM

I'm finding when I increase the `code_version`of an asset, I'm not seeing any indication of the asset being stale in dagit. Is there something I need to do to make that appear other than reloading the definitions? I'm using dagster 1.3.1. This is the first time I've experimented with staleness.

sean

04/28/2023, 5:28 PM

Hi Caleb, That’s odd. If you increment the code version and reload definitions, the asset should get a “New Code Version” tag

sean

04/28/2023, 5:29 PM

Are you by any chance using the automaterialize daemon?

sean

04/28/2023, 5:29 PM

I’m wondering if it’s being quickly detected as outdated and materialized in the background without you realizing

Caleb Parnell Lampen

04/28/2023, 6:05 PM

So, in the code, the version was

v5

. In the asset view in dagit the code_version was listed as`v4`. Just to check that the code was being observed properly by dagit, I manually re-materialized the asset. The version number incremented to v5 in the dagit metadata. So, it does pull the correct code_version at materialization Snapshot of where I'm looking for the code version metadata in dagit:

Caleb Parnell Lampen

04/28/2023, 6:13 PM

I realize I'm sparse on details here, I'm not sure what to mention... A few odd things I might be doing different than typical. I have a partition mapping between static partitions. An upstream asset has a single partition that is mapped to multiple partitions for a downstream asset. Additionally, these partitions are defined by a factory function, regenerated whenever the code location is reloaded (they are dynamically generated based upon the files present on disk). I don't know why any of that would impact code_version behavior, but I get the impression that partitions are relatively new, so I figure its worth mentioning.

sean

04/28/2023, 6:54 PM

Ah, yeah partitions do not play nicely with data versions right now. This is an actively area of focus. Previously, the “staleness” system was totally disabled for partitions. When we broke up “stale” into separate indicators for “New code version” and “New data version”, looks like we kept both tags disabled for partitioned assets, but we should be able to enable easily for code versions for next week’s release (tracking upstream data versions with partitions is the hard case because there are potentially thousands of dependencies). So thanks for the report, this is a bug and I will fix it.

Caleb Parnell Lampen

04/28/2023, 8:05 PM

Ah, interesting. Perhaps related: I'm getting an erroneous "Upstream data missing" warning message when I go to the materialize dialog for the downstream asset. There are actually three assets involved here, which I'll named Upstream1, Upstream2, and Downstream: Upstream1: Uses CoursePartition Upstream2: Uses FinePartition Downstream: Uses FinePartition. CoursePartition maps to FinePartition in a one to many mapping. If I have Upstream1 already materialized, and try to start a job to materialize Upstream2 and Downstream, I get an "Upstream data missing" warning message in the materlialize dialog box. However, the job executes without problem if I go ahead anyway. An additional piece of feedback here: The "Upstream data missing" warning doesn't say what asset it thinks is missing, so I don't know if it is complaining about Upstream1 or Upstream2. Also, you can't create a job or run that materializes all three together, which would be convenient, but I think you already know about that "problem".

12 Views

Open in Slack

Previous Next