The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Hey guys! Looking at setting up dagster and it looks great. Are there any best practices around version controlling the logic in your ops and in particular, guaranteeing any assets are in sync with the latest logic?

For example, let’s say my final output was a csv, and there’s a bunch of ops that are all reading and writing csvs to disk. Then, I edit the logic for an op upstream, but don’t run the job. I would like to know which downstream assets are now stale because of my upstream logic change. Is there anything like that? 

Hey Andy! Have you seen <https://docs.dagster.io/guides/dagster/software-defined-assets>?

The benefit of putting assets first solves your need directly, as quoted from the link above:

&gt; Know when you need to take action on an asset - In a unified view, Dagster compares the assets you've defined in code to the assets you've materialized in storage. You can catch that you've deployed code for generating a new table, but that you haven't yet materialized it. Or that you've deployed code that adds a column to a table, but that your stored table is still missing that column. Or that you've removed an asset definition, but the table still exists in storage.

<@U01AV4K3W14> does this only work if your logic has schema changes? Let’s say you deploy code that doesn’t change the schema but just changed the implementation of a column. Would dagster still understand that it’s out of date?

You can catch that you've deployed code for generating a new table, but that you haven't yet materialized it. Or that you've deployed code that adds a column to a table, but that your stored table is still missing that column. Or that you've removed an asset definition, but the table still exists in storage.

I ask because this seems like all schema changes

<https://docs.dagster.io/guides/dagster/memoization|https://docs.dagster.io/guides/dagster/memoization>

This looks like it might be closer? Not sure if it’s been ported to ops yet tho. I’ll take a look

Ah looks like it’s in ops. Ok I’ll give it a try. Thanks!

Hey Andy! Right now, software-defined assets will not track changes to logic. This is something we're looking into supporting, however.

awesome thanks! lmk if there's a github issue or something i canf ollow

Circling back almost a year later! We're close to shipping asset versioning and memoization. <@U02KDHN92SK> - if this is still something you're interested in, give us a shout