There are currently a few issues with

@multi_asset

in Dagster's UI: 1. At some point, as I remember, it was not possible to select a subset of

is_required=True

assets. It was really convenient as all the assets in the multi asset were selected automatically. Right now I need to manually select all assets. Was this change intentional? 2. At the same time, the assets required for manual selection may appear in very distinct locations on screen. Right now I have a

@multi_asset

which produces 2 assets and one is located at the very top of the screen, while the other one is at the very bottom (using horizontal layout). This makes looking for these assets and selecting them more annoying.

If there’s a feature marked as “experimental” in the docs, how do we tell you that we like it and want to keep it? I found that asset.resource_deps solved an issue I was having in building an asset that creates a snowpark UDTF and would really like it to stick around!

Hi! Is it possible to change run configuration for jobs which are run by scheduler? I can change configuration in the job launchpad and launch the run manually. However, I didn’t find the way to change the run configuration for the scheduler.

Hi! 👋 I find the concept of concurrency limits very interesting and powerful for restricting downloads and calculations. I have a double partitioned job (by client and by date). I download data from an API and then transform this data. I've set up a restriction only on the download asset so as not to overload the API, and no restriction on transformations, and it works very well. This restriction does pose a few problems for me, however. Here's a case in point: I sometimes need to re-download a history for a specific client on the API, which can take several hours or even days. The download of other clients is therefore put on hold, and the upload of data is completely blocked by this backfill. So I'd like to apply this limit, not in the broad asset sense as is currently the case, but to be able to apply this limit on the client dimension. In other words, I'd like to parallelize the downloading of different clients while still having an individual restriction for each client.

As per the docs we are increasingly defining groups as per our data products and data sources. previously we defined them more about the location of data, but that is defined as per asset prefixes now. HOWEVEVER that removes some visibility in thegraph, as it is more difficult to see the different tiers in our medallion architecture when viewing a given data product / group. As an example, given a data source

reddit

we would have

bronze/reddit

silver/reddit

and then maybe

gold/meme_forecast

(the actual sources are obviously not reddit) Previously we would have groups called

raw_reddit

and

silver__reddit

etc, but it became a mess. so now we are considering simply having

reddit

and

meme_forecast

as groups. HOWEVER that means that

reddit

is quite big - that's fine, however the layers are less visible in the graph view. I loved the boxes of groups when viewing the global lineage; could one create nested groupings in the graph view, when viewing groups, that shows the asset key "level", i.e the

reddit

group would have the

raw

assets visibly grouped and

silver

assets visibly grouped. possibly with a

reddit

group inside both

raw

and

silver

to show each level in the asset prefix.

Hello, Thanks again for the great work on the product. We have built multiple code locations on our OS deployment. A per-location tag is used to identified all jobs from a given code location. When runs are triggered by auto-materialize daemon, they are tagged with

automaterialize: true

tag but we can not specify a tag per code location. The main problem I am trying to solve is to display only runs (from Runs view) of a given code location whether they've been auto-materialized or not. This does not seem feasible for now because: • Auto-materialized runs can only be tagged with global tags • Filter does not seem to work with 2 different keys (i.e. filter

auto_materialize: true

OR

app: MYAPP

). Is there any improvement expected for both points ?

Are there any plans in the near future to make Dagster's default IO Manager compatible with single-run backfills? I'm currently using the in_process_executor with mem_io_managers to allow for this right now, but even when my backfill code runs quickly, the run takes quite a while due to the overhead of marking the assets as materialized. I assume this would be faster if I could use the multi_process_executor, but that's not currently compatible with single-run backfills. I know Dagster recently introduced BackfillPolicy logic, so I'm curious if this is something you're looking at too.

Found this page about the dagster-cloud-cli which I thought might help me with something. However, the docs only tell me how to install it, but there’s no mention of what it does or why I’d want to use it. There’s not even a homepage in pypi. The docs really need to be better about telling us why we’d want to use something and what it’s intended to solve; the how comes after that.

Would it be possible to create a read-only version / permission? Does that require Dagster Cloud? I love your Dagster Open Platform idea, and would see it as even stronger if i could actually click around and see "live assets" online. Also from a dagster-demo perspective (Similar to how icepanel has a live demo or home assistant). But from a user-perspective it would also be sweet to have a level of transparency where we can use dagster as a data governance portal, so anyone in our organization can see when data was updated, and use the description field on assets to showcase relevant meta information and visualization links.

I can see some scenarios with auto-materialization where it would be useful to be able to "pause" auto-materialization of specific assets (or groups). Some examples: suppressing errors during a known outage window or for manual intervention on a table. For now can do something similar just by toggling auto-materialize off globally, but that's suspending other unaffected assets.

I noticed that on an asset's page that has prefixes, partitions, and sensors that when you hover over the sensors, it shows sensors not associated with that asset / prefix / partition. Is that what is supposed to happen?

I do wish Dagster would offer a paid on-prem plan that grants access to various paywalled features: SSO, Dagster Insights, etc. The company I work for simply will not allow me to use Dagster Cloud. I've had great success with the OSS version deployed on our internal cloud and would definitely have great use for Insights. There's just no way for me to access it right now.

Hi everyone! We’ve released a discussion about the future of I/O managers as an opt-in layer and would like to hear any feedback you have! We know lots of users really like the I/O manager system, and we aren’t changing any of the APIs for using I/O managers. However, we also know that there are an increasing number of use cases where I/O manager just aren’t the right fit. This discussion goes through some of the new APIs we released in 1.5.0 that make it easier to design your data pipelines if you do not want to use the I/O manager system. If you’ve used these APIs, or are considering using these APIs, let us know what your experience has been like! If there are any feature gaps you’d like to see addressed, let us know about that to!

Hi everyone again! I’m also soliciting feedback on dagster-embedded-elt. Would love to know if you’re using it, for what use cases, and any comments or feedback on future directions! https://github.com/dagster-io/dagster/discussions/17300

Is it possible to sort / filter runs by their duration?

Hi, this type hint example should be corrected, in this blog post.

Hey Dagster Community, Dagster PM here. If you’ve got feedback you want to share about Dagster (and get paid for it) check out our latest user technology survey https://www.surveymonkey.com/r/CPCBT2M. This input influences our thinking about roadmap prioritization for the coming 3-6 months. We’ll send you a $25 dollar gift card of your choice for your time. Thanks for your ongoing support!

Hi Dagster team, Is there a way to stop the UI refresh in Dagit? It's impossible to check history (jobs, sensor ticks, etc) with this on. We're using version

1.3.5

. Thank you!

Not sure if it's a bug, but when utilizing a RetryPolicy, the metadata etc on a Failure isn't shown at the end, so you lose good insight.

Would be great if there was a way to have an auto materialize policy for outdated code version

Shoutout to the team for the experimental asset graph experience celebrate

Short idea; like we now have RetryPolicy, could one have TimeoutPolicy? We can obviously easily write it in python, however our code would be neat if we could do it more declarative. I've thought about just writing a declarative solution for it.

Hi, I tried out Dagster on OpenShift (=Kubernetes) with a local S3 Minio Deployment and the experimental codeServerArgs Mode, so I can reload changes in the user-code container without restarting anything. Works like a charm. When will "codeServerArgs" be production ready/released as an official feature? I think it's rather useful during experimenting phase.

It's probably a bit of a fundamental change, but it'd be absolutely lovely to be able to differentiate jobs that have been terminated from jobs that failed due to an exception. Right now it's very difficult to determine if a job failed and needs to be followed up on, or whether it was terminated by the user. It also makes it so we get a lot of false positive failure alerts generated when folks terminate their jobs

Morning Is there a way to automate deployments? I feel that setting manual would cause a bottleneck

Feedback on the frontend backfkill - when we have form of time partitioning, it seems like it could be pretty natural to allow the view to be by day, month, year etc.

I think re-executing a Run with modified config/tags would be a pretty helpful feature. I often find myself doing this manually: • copy a previous Run config • find the asset/job • open the Launchpad • paste the config • modify and run quite a few steps to do, perhaps shift-clicking on

re-execute

could bring up the Launchpad instead?

https://github.com/dagster-io/dagster/discussions/18312 I would love Dagster to handle asnyc resource dependencies to easier combine pandas and. spark assets.

cross posting https://dagster.slack.com/archives/CCCR6P2UR/p1701282707190949

hey, what happened to tutorial "Part 7: Managing your own I/O"? seems like it got accidentally yeeted here: https://github.com/dagster-io/dagster/pull/15059