<!channel> 🎉 👌 D 0.14.0 is out!
Dagster 0.14.0 “Never Felt Like This Before”
We’re proud to release version 0.14.0 of Dagster. This version introduces much more mature version of software-defined assets_,_ new integrations, a new homepage for Dagit, and a wide set of other features and improvements.
Software Defined Assets
Software-defined assets, which offer a declarative approach to data orchestration on top of Dagster’s core job/op/graph APIs, have matured significantly. Improvements include partitioned assets, a revamped asset details page in Dagit, a cross-repository asset graph view in Dagit, Dagster types on assets, structured metadata on assets, and the ability to materialize ad-hoc selections of assets without defining jobs. Users can expect the APIs to only undergo minor changes before being declared fully stable in Dagster’s next major release. For more information, view the software-defined assets concepts page
here.
Factory Floor View
Dagit has a new homepage, dubbed the “factory floor” view, that provides an overview of recent activity for all the jobs. From it, you can monitor the status of each job’s latest run or quickly re-execute a job. The new timeline view reports the status of all recent runs in a convenient gantt chart.
Dagster-Airbyte
A new Airbyte integration (dagster-airbyte) allows you to kick off and monitor
Airbyte syncs from within Dagster. This contribution includes a
resource implementation as well as a
pre-built op for this purpose, and we’ve extended this library to support
software-defined asset use cases as well.
Dagster-Pandera
A new
Pandera integration (dagster-pandera) allows you to use Pandera’s validation library to wrap dataframe schemas in Dagster types, enabling runtime data validation of Pandas dataframes in Dagster ops/assets. Additionally, Dagit displays Pandera schema information using a new
TableSchema
API.
Metadata on Dagster Types
Dagster Types can now have attached metadata. This allows
TableSchema
objects to be attached to Dagster Types via
TableSchemaMetadata.
A Dagster Type with a
TableSchema
will have the schema rendered in Dagit.
Op Selection in Subgraphs
Op selection now supports selecting ops inside subgraphs. For example, to select an op
my_op
inside a subgraph
my_graph
, you can now specify the query as
<http://my_graph.my|my_graph.my>_op
. This is supported in both Dagit and Python APIs.
ECSRunLauncher
The
ECSRunLauncher is no longer considered experimental. You can bootstrap your own Dagster deployment on ECS using our
docker compose example or you can use it in conjunction with a
managed Dagster Cloud deployment!
Asset Observations
The new `AssetObservation`
event enables recording metadata about an asset without indicating that the asset has been updated.
Op Events Without Yield
Dagster now supports logging
AssetMaterialization
,
ExpectationResult
and
AssetObservation
events via
OpExecutionContext.log_event
. Previously, Dagster only supported yielding these events within the body of an op.
Auto-start Sensors and Schedules
Sensors and schedules now include a
default_status
parameter. If this parameter is set to
RUNNING
, sensors and schedules will default to running as soon as they are loaded in your workspace.
Some screenshots of the new UI: