Any plans to support multi-line docstrings for assets? (summary line, followed by empty line, followed by detailed description). At the moment Dagit drops all the newlines and displays the entire docstring as one long line in the description. I'd expect it to just use the first line of the docstring so it appears neatly in the UI. And maybe expose the rest somewhere else in the UI. Or at least not drop all the newlines! 🙂

are there any plans to migrate to SQLAlchemy 2.X? per this issue, it seems like a popular request: Add support for sqlalchemy 2.0+ · Issue #12994 · dagster-io/dagster (github.com) this is preventing me from using dagster as there's some bug with writing to parquet files

I've recently noticed that the timeline overview doesn't scale with the width of the window when you expand the window by dragging it, meaning that the hour marks will move but the runs stay in the same place. This is on firefox.

In dagster-dbt when running a job that is made up of multiple dbt models the view shows them all as one large op. Having them broken down to one op per model would make it much easier to see what is actually happening when the dag is running, and understand each models perf. It would also make it easier to re run only the models that failed when something breaks.

With software-defined assets, it was really surprising that there isn't anything that stops overlapping execution of the same asset op - for example you can just click "materialize" a bunch of times on the same asset when dagit shows it's already running. I found this out the hard way when I was trying observable source assets with auto-materializing downstream assets, which started many overlapping staging table refresh actions and my process broke down. This is all with non-argument dependencies, if that makes a difference. I'm scrambling now to figure out celery or dask to avoid this problem with multiple users potentially materializing assets on demand.

Did scrolling to zoom the DAG view go away? In the job overview DAG view, I can click to grab and move the canvas which works great. If I scroll, that also only moves the canvas up/down, and to zoom in/out I don't have a mouse control, I have to click on the +/- zoom control. I think scrolling used to zoom in/out, and I would like that to use it to zoom. (I tried shift-scroll and ctrl-scroll which didn't work either.)

Preview: New Asset Overview Page Hey all! I wanted to share a preview of a new experimental UI feature we’re launching in tomorrow’s release. The new Asset Overview tab gives you a summary of the status of every asset across your deployment and aims to help surface assets that are missing, failed or violating their freshness policy. Check out the demo here. You’ll be able to access this new Overview page tomorrow by enabling the feature from your User Settings. Give it a try and let us know what you think!

Updated to

1.3.4

and noticed a dagit regression - on the run page, long tags used to display a tooltip containing the full tag (as they are truncated in the run table) - that tooltip no longer seems to appear

I think could be useful to support config type that are in python standard lib like Path

For my understanding If i have a sensor running and a certain request fail, it will not run again as the job was successful I imagine this behavior is to avoid having to fail again in x seconds when the sensor re-run but in case of detected code change of the job could make sense to rerun the request of failing jobs?

Could someone check out my response: https://dagster.slack.com/archives/C01U954MEER/p1684376089938359?thread_ts=1684348454.801219&cid=C01U954MEER First of all, is my interpretation of how or where EnvVar() is to be used correct? In other words, the print statement would not print the key's value, just the key? If so, I would actually like to be able to use EnvVar() outside of a configurable Dagster object to obtain the value. I think I may have thought or assumed the same thing as @Sebastien did, expecting to be able to use EnvVar() outside the context of a configurable dagster object and get the key's value.

Hello, As amount of data assets managed by a Dagster instance keeps growing and thanks to the code locations concepts multiple sub-teams can independently add and enrich their set of assets, it can become harder to explore and identify an asset that can be reused for additional needs. Asset description and Dagster type are great way to be able to understand available/implemented data assets (their structure and what information they provide). The current Dagit (

1.3.4

) search seems to be limited to asset/group/job names but does not seem to allow search on description or dagster types. Is this feature expected in your backlog or will we need to use a dedicated data catalog tool (such as DataHub) to manage exploration of data assets / (search in fields / description) ?

In Dagster Cloud, when I click on a step under "Not succeeded", the Waterfall view scrolls for a second or two and then stops, it doesn't scroll all the way to my step any more.

I think Dagster needs better tools for the long-term lifecycle of a project. Let me explain with my scenario. I’m converting a bunch of old assets to a new underlying data model. Kind of a big old refactor. Instead certain assets being dataframes, they’ll be a richer type, called a

Table

. Details aren’t important - what matters is that dataframes and tables are not compatible. So I have these old assets with keys like

ztf_source_dataframe

, which are pandas dataframes. But I want to convert to `Table`s. Here are the options I see: 1. Reuse the

ztf_source_dataframe

asset key, but return a non-dataframe type. Which is confusing, and will break dependents, which can possibly be managed with asset versioning in some way? But the name confusion would be unfortunate. 2. Write new

ztf_source_table

asset, change dependents to use it, and then delete

ztf_source_dataframe.

But this would destroy all history and orphan the materialized assets. Historical runs will be… broken? I don’t know what will happen to the dagit UI for them. 3. Write the new asset, but then keep

ztf_source_dataframe

as relics of a bygone era. But they’ll clutter the UI and the codebase forever. Is there a way to mark assets as “archived” or “deprecated” or “just kept around in the attic?” Gradual migrations like this are really important. I think Dagster could provide tools to manage this, and they could be fantastically better than anything else out there, since Dagster knows so much about my computation graph. I don’t have a concrete suggestion but think this is an important area for new features.

Two pieces of feedback around using per-asset metadata to control the behavior of the io manager configured for that asset: 1. It would be nice to have a special class of metadata that is NOT passed to the IO manager and is only used for display/data dictionary purposes. Letting business-centric metadata live in the same place as operational metadata can be dangerous if a user configures an asset with a metadata key that unknowingly matches with a key that the IO manager expects. 2. It would also be nice to have deploy-time validation of the per-asset metadata that an io manager expects, similar to the validation that can be done for resources. That way, if a user forgets to configure a necessary piece of metadata for the IO manager and no default is set, we can error earlier and avoid a runtime error

Hello, 👋 I would like to suggest improving the mobile support for the web UI, specifically regarding accessing logs. Currently, when I receive Slack notifications for job, sensor, or schedule failures and try to view them in Dagit on my mobile device, it is often impossible to view the logs. Even after rotating the screen, the interface often remains unworkable, leaving me with no choice but to switch to my PC. Better mobile support would empower developers/users to effortlessly monitor failures, access logs, and enhance their overall experience and productivity. Thanks!

hello! is there any timeline to support SQLAlchemy 2.X? this is blocking my scenario from adopting Dagster as there's some bug with datetime types in 1.X that's fixed in 2.X. Prefect (yesterday) added support, and it's currently a decently +1'ed issue in Dagster

This query

SELECT
  event_logs.id,
  event_logs.event
FROM
  event_logs
WHERE
  event_logs.dagster_event_type = $1
ORDER BY
  event_logs.id DESC

Which seems to be invoked by

context: MultiAssetSensorEvaluationContext
context.latest_materialization_records_by_partition_and_asset()

Becomes very expensive as the event_logs table grows (which is indefinite I believe as it's essentially a write-only table). This is expensive even with the appropriate indexes that get leveraged by the query plan. Has there been any throughput testing on this pattern, or any ideas as to how to optimize this. Unless i'm missing something this seems to make the canonical use of multi-asset sensors non viable even at a fairly modest scale as it will only decay in performance as materializations continue until it eventually (or in the case that materializations already exist when turning on the sensor, immediately) can't complete within the hard 60 second timeout

Ran across this error message with an old reference to solids: "The node_handle property is not set on the context when a solid is directly invoked."

Hi! I have been using multi-partitioning for a short time and I find the concept very powerful for my use case. However, it often happens that I don't want to have a partition for each unique combination of dimensional partitions. For example in the case where I need to perform the same operation on several clients that have different time partitioning, I need to consider the oldest date to create all the keys. The consequence of this is that some of the keys are useless to me and return errors if I try to materialize them.

Probably already known since this is still experimental - for observable_source_asset jobs, there isn't an "Observe sources" equivalent button to run the job on demand. Once there's been at least one run, re-executing is one way to do it. It looks like the "observe" equivalent for "materialize" is missing in most places. That "observe sources" button from the assets group works fine in the mean time. The overall direction with observable source assets is good I think. For now, I don't think I'll use it in replication tasks when there's a single source table and single target table, due to adding a layer of indirection (single-click materialize becomes "observe and wait for async auto-materialize, then navigate to see results"). I think I've got a form of global op concurrency control working well enough for my purposes with dask-distributed Semaphores handling locks for multiprocess executor. Added notes in the earlier thread.

For auto-materialize max materializations per minute, I'd probably be looking at longer op "cooldowns" on the order of 15 minutes or an hour. I might implement something like that with RetryRequested exceptions myself in the mean time.

Hi all! In our most recent release of

dagster-dbt

(0.19.6), we added some experimental APIs to help our users create software-defined assets from their dbt projects in a more composable and ergonomic way. I’ve posted an overview of the proposed API changes in a GitHub Discussion — please chime in if you have any questions or feedback about the proposed changes! Here’s a brief run-through of the changes: “”" There are three main new interfaces that we plan to release as part of this release: 1.

@dbt_assets

(link), a decorator-based interface for users to define software-defined assets from their dbt resources. 2.

DbtCli

(link), an improved and lighter interface for invoking dbt commands within the Dagster framework. 3.

DbtManifest

(link), a representation of your dbt project’s resources (models, tests, macros, etc), which will be used as arguments when invoking (1) and (2). “”"

image.png

Hi All, it seems that the dagster-aws docs here (https://docs.dagster.io/_apidocs/libraries/dagster-aws) are a bit out of date. It's missing documentation for Athena, s3 sensors and more. Would be good if these were updated since it looks like you all have built a ton of great functionality I'm only now discovering by looking through the code directly on git!

Run retries documentation doesn't mention that the feature isn't supported with the default event log storage, SQLite .

In the docs I love being able to look at the source code, it would be nice if there was a direct link from the source code to the corresponding source code page on github, which is a bit easier to read and can be navigated.

Love the "scaffold default config" button and the hover-docs. Would be great for that to auto-populate, with

ops

at the top, if the launchpad would otherwise be empty, or for the button to highlighted in some way. I imagine a brand new user is going to be unlikely to just explore and click those buttons.

Love dagster, would really like a minor release made containing https://github.com/dagster-io/dagster/issues/12994 as its blocking some dagster usecases for us. Perhaps in the future, there could even be an automatic dagster canary release for those needing the bleeding edge stuff.