Hello 👋 team! I've been using the "_Experimental schedule/sensor logging view_" in Dagit and found it super helpful for debugging sensors. I have 2 suggestions for improvement: 1. It would be great if we could also view logs when using the "_Test Sensor > Evaluate_" functionality. This would provide better visibility and make it easier to identify issues during testing. 2. Additionally, having live logs available while a sensor is running (either during a scheduled tick or a test evaluation) would significantly improve the experience when testing a sensor that takes more than 30 seconds to run. I think these improvements would make sensor logging even more ergonomic and user-friendly. Looking forward to hearing your thoughts on this!

I’m not sure if

AutoMaterializePolicies

are working as intended for multi-partitioned assets. Having a multi-partitioned asset that includes a static and a dynamic partitioned set, creating a new dynamic partition and kicking off a run for all static partitions under the latest dynamic partition will only trigger the materialization on downstream assets (which are set to eager) if all partitions of the upstream asset have been materialized. In case one or more of the dynamic partitions haven’t been materialized (and aren’t queued, so dagster can’t know they will be materialized soon), the downstream run won’t be launched.

It would be very nice if it was possible to select multiple assets in the Asset Catalog by dragging the cursor and drawing a rectangle (lasso) area

An additional QOL nice-to-have in terms of UI/UX would be some level of control over the layout of the Global Asset Lineage view. We have a very wide/flat ingestion layer that condenses down to a few downstream assets. The current UI tends to build width-first, which can be fairly limiting and ends up in a lot of wasted screen real estate. Some things I wish I've wished I could do: • Toggle between top-down and left-right dependency organization • Manually move asset groups, drag and drop style • Ability to have asset groups organize grid style, as opposed to single-row only ◦ We have a fair number of asset groups that are ingestion-only with no downstream dependencies, and they tend to be what's cluttering the ui, stretching it out horizontally • An alternative solution to the above is to have assets with no downstreams land above the rest of the graph, visually. I know none of that is simple to implement, so low expectations here 🙂

In Dagit's "Timed view," I find the display of large numbers to be unintuitive. It might be more user-friendly to increase the units earlier. I've provided some examples in the comments

Just reading up on the AutoMaterializePolicy features, very cool, and natural extension. Are there plans to extend it to other triggers, like schedule- and event-based? I can imagine the combination of

AutomaterializePolicy.cron_schedule("0 0 * * *")

and

AutomaterializePolicy.eager()

could solve a ton of use cases without ever having to learn about sensors and schedules classes.

Hello team! I would love if we could push build args into the Docker build process during a Dagster hybrid deployment. This would look like a new input to this workflow, which would then get passed into this step. Why? Our product and development pipeline uses a lot of environment variables, and I’ve had to make many workarounds to access those variables. Here’s my current process: • I spin up the Kubernetes Helm release using Pulumi (basically, Terraform). In my Pulumi script, I pass environment variables received from our CI/CD pipeline to the Helm release. • Our Dagster jobs/steps must access these variables. Because they aren’t available during the Docker build, I can’t reference any environment variables unless the code is inside an op (anything outside an op, i.e. job definition, would be run at build time, and therefore wouldn’t access the variables). • But ops are run by the image spun up by the Helm release, so i can use

os.environ

as long as the code is inside an op. The series of hoops I’ve jumped through work, but I’ve finally run into a scenario where this doesn’t cut it. I want to use an environment variable to construct a string, which gets set in my job config (it’s the dbt target). If I didn’t need to do any string transformations, I could set

resources: dbt: config: target: env: MY_ENV_VARIABLE

, but because I need to modify it, I am forced to do this in my Python scripts. And my Python scripts can only access environment variables during op runtime… So, if I had build args during the build, I could turn all my ARGs into ENVs in the Dockerfile. I could cut Pulumi out of this whole process - passing variables from CI/CD straight into Dagster - and simplify a lot of my Python that today is restricted by only having access to environment variables at runtime.

I've noticed that if a dagster run is deleted within k8s (while obviously not the preferred termination strategy), runs page no longer renders properly

Operation name: RunsRootQuery

Message: Cannot return null for non-nullable field Run.mode.

Path: ["pipelineRunsOrError","results",5,"mode"]

Locations: [{"line":29,"column":3}]

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 541, in execute_field
    return_type, field_nodes, info, path, result
  File "/usr/local/lib/python3.7/site-packages/graphql/execution/execute.py", line 621, in complete_value
    "Cannot return null for non-nullable field"

Hi folks! Playing with new config system and have two issues hardly described in current docs, so I suppose it would be great to add some examples with it • Usage of

<http://EnvVar.int|EnvVar.int>("VAR_NAME")

- maybe it I'm not so smart but wasting half an hour figuring out how to work correctly with int-based envvar without pydantic errors is not so fun. • Usage of

<http://RunConfig.to|RunConfig.to>_config_dict()

. Probably my use-case is a bit marginal, but I have bunch of jobs which uses

@config_mapping

and is started manually from dagit UI without python invocation of

execute_in_process()

. Previously returning dict from

@config_mapping

was okay-ish option but with all pydantic benefits doing it looks quite dumb. So this method becomes handy allowing to create fully-pydantic config inside mapping and then pass it to "just" job with ease. And I again spent quite some time before found this method in the source code :)

small gotcha with

dagster.EnvVar

that I didn't expect: it silently fails when used for something that is not bound to a run (specifically the dagster-slack

make_slack_on_run_failure_sensor

). took me a while to figure out why!

Hello, How to keep/store materialized asset status when we stop dagster daemon ? Here a dataframe materialized yesterday as a ".parquet" file, I had to shutdown my pc to apply MS Updates, today it is not materialized. Regards,

Any plans to support multi-line docstrings for assets? (summary line, followed by empty line, followed by detailed description). At the moment Dagit drops all the newlines and displays the entire docstring as one long line in the description. I'd expect it to just use the first line of the docstring so it appears neatly in the UI. And maybe expose the rest somewhere else in the UI. Or at least not drop all the newlines! 🙂

are there any plans to migrate to SQLAlchemy 2.X? per this issue, it seems like a popular request: Add support for sqlalchemy 2.0+ · Issue #12994 · dagster-io/dagster (github.com) this is preventing me from using dagster as there's some bug with writing to parquet files

I've recently noticed that the timeline overview doesn't scale with the width of the window when you expand the window by dragging it, meaning that the hour marks will move but the runs stay in the same place. This is on firefox.

In dagster-dbt when running a job that is made up of multiple dbt models the view shows them all as one large op. Having them broken down to one op per model would make it much easier to see what is actually happening when the dag is running, and understand each models perf. It would also make it easier to re run only the models that failed when something breaks.

With software-defined assets, it was really surprising that there isn't anything that stops overlapping execution of the same asset op - for example you can just click "materialize" a bunch of times on the same asset when dagit shows it's already running. I found this out the hard way when I was trying observable source assets with auto-materializing downstream assets, which started many overlapping staging table refresh actions and my process broke down. This is all with non-argument dependencies, if that makes a difference. I'm scrambling now to figure out celery or dask to avoid this problem with multiple users potentially materializing assets on demand.

Did scrolling to zoom the DAG view go away? In the job overview DAG view, I can click to grab and move the canvas which works great. If I scroll, that also only moves the canvas up/down, and to zoom in/out I don't have a mouse control, I have to click on the +/- zoom control. I think scrolling used to zoom in/out, and I would like that to use it to zoom. (I tried shift-scroll and ctrl-scroll which didn't work either.)

Preview: New Asset Overview Page Hey all! I wanted to share a preview of a new experimental UI feature we’re launching in tomorrow’s release. The new Asset Overview tab gives you a summary of the status of every asset across your deployment and aims to help surface assets that are missing, failed or violating their freshness policy. Check out the demo here. You’ll be able to access this new Overview page tomorrow by enabling the feature from your User Settings. Give it a try and let us know what you think!

Updated to

1.3.4

and noticed a dagit regression - on the run page, long tags used to display a tooltip containing the full tag (as they are truncated in the run table) - that tooltip no longer seems to appear

I think could be useful to support config type that are in python standard lib like Path

For my understanding If i have a sensor running and a certain request fail, it will not run again as the job was successful I imagine this behavior is to avoid having to fail again in x seconds when the sensor re-run but in case of detected code change of the job could make sense to rerun the request of failing jobs?

Could someone check out my response: https://dagster.slack.com/archives/C01U954MEER/p1684376089938359?thread_ts=1684348454.801219&cid=C01U954MEER First of all, is my interpretation of how or where EnvVar() is to be used correct? In other words, the print statement would not print the key's value, just the key? If so, I would actually like to be able to use EnvVar() outside of a configurable Dagster object to obtain the value. I think I may have thought or assumed the same thing as @Sebastien did, expecting to be able to use EnvVar() outside the context of a configurable dagster object and get the key's value.

Hello, As amount of data assets managed by a Dagster instance keeps growing and thanks to the code locations concepts multiple sub-teams can independently add and enrich their set of assets, it can become harder to explore and identify an asset that can be reused for additional needs. Asset description and Dagster type are great way to be able to understand available/implemented data assets (their structure and what information they provide). The current Dagit (

1.3.4

) search seems to be limited to asset/group/job names but does not seem to allow search on description or dagster types. Is this feature expected in your backlog or will we need to use a dedicated data catalog tool (such as DataHub) to manage exploration of data assets / (search in fields / description) ?

In Dagster Cloud, when I click on a step under "Not succeeded", the Waterfall view scrolls for a second or two and then stops, it doesn't scroll all the way to my step any more.

I think Dagster needs better tools for the long-term lifecycle of a project. Let me explain with my scenario. I’m converting a bunch of old assets to a new underlying data model. Kind of a big old refactor. Instead certain assets being dataframes, they’ll be a richer type, called a

Table

. Details aren’t important - what matters is that dataframes and tables are not compatible. So I have these old assets with keys like

ztf_source_dataframe

, which are pandas dataframes. But I want to convert to `Table`s. Here are the options I see: 1. Reuse the

ztf_source_dataframe

asset key, but return a non-dataframe type. Which is confusing, and will break dependents, which can possibly be managed with asset versioning in some way? But the name confusion would be unfortunate. 2. Write new

ztf_source_table

asset, change dependents to use it, and then delete

ztf_source_dataframe.

But this would destroy all history and orphan the materialized assets. Historical runs will be… broken? I don’t know what will happen to the dagit UI for them. 3. Write the new asset, but then keep

ztf_source_dataframe

as relics of a bygone era. But they’ll clutter the UI and the codebase forever. Is there a way to mark assets as “archived” or “deprecated” or “just kept around in the attic?” Gradual migrations like this are really important. I think Dagster could provide tools to manage this, and they could be fantastically better than anything else out there, since Dagster knows so much about my computation graph. I don’t have a concrete suggestion but think this is an important area for new features.

Two pieces of feedback around using per-asset metadata to control the behavior of the io manager configured for that asset: 1. It would be nice to have a special class of metadata that is NOT passed to the IO manager and is only used for display/data dictionary purposes. Letting business-centric metadata live in the same place as operational metadata can be dangerous if a user configures an asset with a metadata key that unknowingly matches with a key that the IO manager expects. 2. It would also be nice to have deploy-time validation of the per-asset metadata that an io manager expects, similar to the validation that can be done for resources. That way, if a user forgets to configure a necessary piece of metadata for the IO manager and no default is set, we can error earlier and avoid a runtime error

Hello, 👋 I would like to suggest improving the mobile support for the web UI, specifically regarding accessing logs. Currently, when I receive Slack notifications for job, sensor, or schedule failures and try to view them in Dagit on my mobile device, it is often impossible to view the logs. Even after rotating the screen, the interface often remains unworkable, leaving me with no choice but to switch to my PC. Better mobile support would empower developers/users to effortlessly monitor failures, access logs, and enhance their overall experience and productivity. Thanks!

hello! is there any timeline to support SQLAlchemy 2.X? this is blocking my scenario from adopting Dagster as there's some bug with datetime types in 1.X that's fixed in 2.X. Prefect (yesterday) added support, and it's currently a decently +1'ed issue in Dagster

This query

SELECT
  event_logs.id,
  event_logs.event
FROM
  event_logs
WHERE
  event_logs.dagster_event_type = $1
ORDER BY
  event_logs.id DESC

Which seems to be invoked by

context: MultiAssetSensorEvaluationContext
context.latest_materialization_records_by_partition_and_asset()

Becomes very expensive as the event_logs table grows (which is indefinite I believe as it's essentially a write-only table). This is expensive even with the appropriate indexes that get leveraged by the query plan. Has there been any throughput testing on this pattern, or any ideas as to how to optimize this. Unless i'm missing something this seems to make the canonical use of multi-asset sensors non viable even at a fairly modest scale as it will only decay in performance as materializations continue until it eventually (or in the case that materializations already exist when turning on the sensor, immediately) can't complete within the hard 60 second timeout

Ran across this error message with an old reference to solids: "The node_handle property is not set on the context when a solid is directly invoked."