sean
05/13/2022, 5:41 PMdagster
) so that they are formally private. Users should target all dagster
imports to the top-level dagster
module:
# CORRECT
from dagster import op
# INCORRECT
from dagster.core.definitions.decorators import op
Renaming starts in our most recent release (0.14.15
) with the rename of dagster.check
> dagster._check
.
We hope this change will not affect the vast majority of users, since our public API has long been informally defined as the top-level dagster
module. However, for users who are using private APIs or otherwise importing from dagster
submodules, we have made some temporary accommodations. There’s a shim in place that will prevent our module renames from breaking runtime-- imports of old module names are dynamically resolved to their new locations (so import dagster.check
is equivalent to import dagster._check
as far as runtime is concerned).
However, all submodule imports will now trigger linter warnings-- either because you are importing a formally private module (e.g. dagster._check
) or because you are importing from a module name invisible to static analyzers (e.g. dagster.check
).
Please reach out in #dagster-support if this transition causes you any other problem!Mollie
05/17/2022, 7:33 PMMollie
06/07/2022, 6:36 PMMollie
06/10/2022, 1:48 AMsandy
06/16/2022, 2:39 PMgroup_name
s. These assets will be grouped together in Dagit.
• Graph-backed assets - software-defined assets can now be interleaved with ops.
• *Graph & job input values -*`execute_in_process` and <http://GraphDefinition.to|GraphDefinition.to>_job
now both accept an input_values
argument, so you can pass arbitrary Python objects to the root inputs of your graphs and jobs.
• Run retries - You can now automatically re-execute runs from failure, like op-level retries, but at the job level.
• Redesigned partition & backfill pages - Now they're faster and show the status of all partitions.
• Redesigned navigation pane in Dagit - It now organizes jobs and asset groups by repository.
• Generic Outputs - Ops that return `Output`s and `DynamicOutput`s now work well with Python type annotations.
• Job metadata - You can now attach metadata to jobs and see that metadata displayed in Dagit.
And more
• Full release notes
• Migration guide
A special thanks to everyone in the community who contributed to this release: @Charles Lariviere, @fahad, @Abdó Roig, @Donny Winston, @3cham, @Frank Dekervel, @peay, @bollwyvl, @trevenrawr, @Javier162380, @jrouly, @dhume, @ascrookes, @kstennettlull, @swotai, @HAMZA310, @kbd, @LeoHuckvale, @joe-hdai, @antquinonez5, @Alex Kerney, @frcode, @dwall, @proteusiq, @kahnwong, and @iswariyam!
And thanks to everyone who helped us name this release: https://github.com/dagster-io/dagster/discussions/8234Mollie
06/16/2022, 9:03 PMMollie
06/20/2022, 7:31 PMMollie
06/21/2022, 1:11 AMMollie
06/21/2022, 6:24 PMMollie
06/21/2022, 6:58 PMschrockn
07/12/2022, 5:12 PMMollie
08/02/2022, 9:25 PMsandy
08/05/2022, 3:27 PMexecutor_def
or logger_defs
.
• Run status sensors can now yield RunRequests
- allowing you to kick off a job in response to the status of another job.
• Input managers - when loading an upstream asset or op output as an input, you can now set custom loading behavior using the input_manager_key
argument to AssetIn and In.
• Global asset lineage graph - back and better than before. The graph clusters together assets that are in the same group, and the query bar allows you to visualize a custom slice of your asset graph.
• dagster project CLI - contains commands for bootstrapping Dagster projects from scratch or from examples, with our recommended project structure.
• Legacy and deprecated API removal - many deprecated APIs have been removed. Take a look at the migration guide for details.
• Local-to-prod guide - a new guide on how config, resources, and environment variables fit together to make jobs and assets portable across environments.
• Library versioning - Dagster’s integration libraries are now at version 0.16 and are compatible with Dagster 1.0. We’ll mark them 1.x after we audit their APIs in the same way we’ve audited the core APIs.
Resources
• Full release notes
• Migration guide
And thank you to everyone who helped us name this release!Mollie
08/09/2022, 3:11 PMFraser Marlow
09/21/2022, 5:27 PMFraser Marlow
09/22/2022, 7:41 PMFraser Marlow
09/26/2022, 10:03 PMDagster Jarred
10/04/2022, 4:00 PMFraser Marlow
10/05/2022, 7:16 PMFraser Marlow
10/07/2022, 3:42 PMFraser Marlow
10/10/2022, 5:43 PMFraser Marlow
10/11/2022, 3:30 PMyuhan
10/12/2022, 4:29 PMjosh
10/17/2022, 3:54 PMNew workspace pages, overview pages, and top navigation
feature flag in your user settings.
We’d love for you to try these out and let us know what you think! Feel free to share any feedback in this thread or in the #dagster-feedback channel.Fraser Marlow
10/22/2022, 8:55 PMrex
10/25/2022, 1:38 AMrex
10/25/2022, 2:14 AMrex
10/26/2022, 3:09 PMDagsterCloudOperator
or DagsterOperator
operators in your existing Airflow DAGs.
2. 🤯 You can automatically create a Dagster job from an existing Airflow DAG - we’ll infer all the dependencies for you.
3. 👀 You can take advantage of Dagster’s programming model to create robust, testable, and reusable jobs to materialize your data assets - we’ll infer the Airflow DAG for you.
Join the channel for updates and to discuss your Airflow migration questions dagster spinFraser Marlow
10/31/2022, 4:10 AM