Hey, just a heads up - I think there should be better documentation around how dagster defines and finds it's components in a project. I spent the last few hours working through scheduler definitions and sensors, but could not find them - I even got them to run without them appearing in the dagit UI. In the end it was because my init file from when I created the project in December was using:

from .repository import my_dagster_project

and

repository.py

was:

from dagster import load_assets_from_package_module, repository
from my_dagster_project import assets


@repository
def my_dagster_project():
    return [load_assets_from_package_module(assets)]

I changed that to

from .repository import my_dagster_project

and

from dagster import Definitions, load_assets_from_package_module
from my_dagster_project import assets



all_assets = load_assets_from_package_module(assets)

defs = Definitions(
    assets=all_assets,
    schedules=[assets.dg_test_schedule.update_job_schedule]
)

And now things work However, this took me far too long 😂 - it should have been flagged as a gotcha somewhere IMO

sorry, I can't read lol

a small request: i notice that Slack hooks use

dagit_base_url

to generate a link to the job run (which is great!), but this link only works with whatever the default deployment is. i use Dagster Hybrid, so the URL

mycompany.dagster.cloud/instance/runs/abcd...

redirects to our one non-branch deployment, called

prod

, so the URL becomes

mycompany.dagster.cloud/prod/runs/abcd...

. there doesn't seem to be a way to link directly to my branch deployment.

Hello all! Dagster Product Designer here 👋 We’re exploring some changes to the Runs page in Dagit that aim to bring a few new filtering options and a more dense, scannable table layout. Here’s an early look at a prototype we are considering, I’d love to hear what you think. You can leave any comments or reactions here in Slack or right within the Loom video. One of our focus areas in the UI in the coming months is to continue to improve the scalability of pages like the Runs page and Overview pages – adding more filtering, grouping, and sorting options. This prototype is just the first of a few ideas that I’ll be sharing that focus on this theme.

What is the path to setting optional config fields in the new Config class? My instinct was to set a

None

default but that still results in a requirement to specify a string.

from dagster import Config, asset, RunConfig, materialize

class AssetConfig(Config):
    name: str = "stephen"
    optional_adverb: str = None

@asset
def foo(config: AssetConfig) -> str:
    msg = f"{config.name} is {config.optional_adverb} cool."
    print(msg)
    return msg

# these should be valid configs
materialize([foo], run_config=RunConfig({"foo": AssetConfig(optional_adverb="really")}))
materialize([foo], run_config=RunConfig({"foo": AssetConfig()}))

error

Error 1: Value at path root:ops:foo:config:optional_adverb must not be None. Expected "(String | { env: String })"

Would love to have a date somewhere in this part of the UI. It’s not clear which day I’m looking at when using pagination.

It would be helpful to have a "test" button on the alerts page, to generate a sample alert message through the integration for an alert, to make sure the integration is working as expected before waiting for a real incident. (I forgot to add the Dagster bot to the channel, and didn't find the error log about it not being there without some Dagster support [thanks!].)

I'm trying to install black as a dev dependency and am running in to the following:

❯ poetry add --group dev black
Using version ^23.1.0 for black

Updating dependencies
Resolving dependencies... (0.2s)

Because no versions of black match >23.1.0,<24.0.0
 and black (23.1.0) depends on packaging (>=22.0), black (>=23.1.0,<24.0.0) requires packaging (>=22.0).
And because dagster-cloud (1.2.1) depends on dagster-cloud-cli (1.2.1) which depends on packaging (>=20.9,<22), black (>=23.1.0,<24.0.0) is incompatible with dagster-cloud (1.2.1).
So, because ***** depends on both dagster-cloud (1.2.1) and black (^23.1.0), version solving failed.

I can get around it by pinning to the final 22.* release of black, but am wondering if there is any reason that the packaging module in

dagster-cloud-cli

needs to be pinned to

<22

?

A suggestion for a usability win: date range filtering and using less granular labels on Asset Plot pages - have a process that runs a few times every day, but this hourly depiction of the time stamp is confusing:

The hybrid quickstart Dockerfile has the following:

WORKDIR /opt/dagster/app
COPY . /opt/dagster/app

which copies the contents of the repo in to the docker image. If this is used alongside the github actions for building/deploying, it results in the

actions-repo

path being included in the dockerfile (verified using dive). In my case, adding that folder to

.dockerignore

knocked about 90MB off the total image size.

The

cloud-branch-deployments-action

is deprecated, but the https://docs.dagster.io/guides/dagster/branch_deployments refers to it when setting up the github actions for cloning of the production db,

Hello 👋 I was wondering if there was some sort of dagster cli command to load in definitions and get the dagster type checker without loading dagit. The end goal would be a pre commit hook that would run this checker. Thanks for any and all help!

Hi! I was looking in the documentation at custom run launchers (to make something for nomad). However the documentation there seems to not be very extensive. Could you improve those, or am I looking at the wrong place?

I think the MetadataValue page could benefit from a showcase of how each type looks in the Dagit UI

Hello, is there any roadmap for making MultiPartitionsDefinition work with DynamicPartitionsDefinition? Or some hack to work around it?

Hi. I think Dagster is really missing something like a

MultithreadedExecutor

which would allow executing steps in parallel within the same process. Currently, when running on Kubernetes using the

MultiprocessExecutor

, there's a massive cold start delay for each subprocess being started because, from what I understand, it has to reload the entire project and assets, which can lead to extremely long jobs even when the tasks are very lightweight.

Should

AssetIn

behave like pydantic’s Field? It would be possible to make assets more readable:

@asset
def my_asset(upstream: pd.DataFrame = AssetIn(metadata={"columns": ["A", "B"]})):
    ...

@sandy @schrockn

Hello, there is any roadmap about allowing multiple dynamic downstream ops? https://github.com/dagster-io/dagster/issues/4364#issuecomment-1474847914

I mentioned that the documentation doesn't show how to actually setup logging, however never really got a response. The links on the loggers page are sadly broken, and i cannot seem to figure out how to setup the Json_console_logger as the default logger. It seems the documentation is lacking in this area

I have a backfill over 3600 partitions that I just now tried to cancel ("Terminate remaining runs"). That takes already 5min, I am halfway through. I'd say this is a number of partitions that dagster should be able to handle well.

Something that would be nice to see is more useful/concise alerts for runs initiated by an asset reconciliation sensor. Right now, the "job" that's alerted is

__ASSET_JOB_0

, not very clear. Reporting the asset names or something similar (i understand there can be a lot of assets having runs initiated by the sensor) would be clearer.

I am trying to use a multi line string in python for my asset description. The description preview on the ui displays the triple quote instead of the description.

Hi, I currently have an asset which is statically partitioned and when I try to select the partitions I want to run on dagit I hit an issue. The dropdown box for the partitions bugs out, when I scroll down to select some of the partitions it stops so I can't access all of the partitions and the partitions in the dropdown will start disappearing and reappearing very quickly (like blinking). I have attached a screenshot and this behaviour is shown as the partitions just not appearing.

Following the message I posted above: I think the job monitoring selection for sensors is somewhat broken. • When NOT specifying the

monitored_jobs

param, sensors are triggered for all jobs across all deployments. • When specifying the

monitored_jobs

, if several jobs have the same name (e.g.

daily

) across different deployments, then the sensor will be triggered for all of those jobs. When specifying

I have a few jobs for which I want the behavior to be 'this job is always running'. Currently the way I handle this is I have a job and a sensor that makes sure the job is always running. It would be nice to have a function or something that encapsulates all of this to simplify this.

Is there a way to notify dagster that an asset materialization is missing, e.g. I am writing it to s3 into a bucket that is set to delete everything older than x weeks -> dagster somehow recognizes that it has to rematerialize the asset.

Have been using Dagster for proximally a year, currently orchestrating dozens of data pipelines for different companies/projects and over a hundred data assets. It is definitely the best data orchestration tool currently in my opinion! 🌈

It's an annoying ask, but the regex constraints on group names are a bit annoying - at my company we would like to use

"-"

&

" "

as characters in the group names. Sorry to be a pain 😅

Really would like to have

dbt tests

as bona-fide Dagster Assets. We've thought about it and they seem ideal as assets, with a catch. I want them to materialize the test result on failure (

ignore_handled_error

in

dbt_resource

works for this). But currently tests are hard coded in

dagster_dbt

to not be assets, and my hacky workarounds have short-comings 😛. But it is really nice to be able to put tests, which are really just sql models like dbt, into an asset lineage graph, see the dependencies, assign freshness policies and have a reconciliation sensor handle it all, plus the occasional asset job for batch tests.

I really like the new experimental pythonic config and resources, but it desperately needs some documentation on how to adapt tests to the new paradigm. I already migrated a few weeks ago, but remember it took a lot of playing around with how exactly to pass them to the `op`s and `asset`s, sometimes passing as an argument, sometimes sticking to the

required_resource_keys

parameter, and experimenting a lot with the

build_*_context

functions. What triggered this message was trying to add resources to a few schedules I have running and having to go through the process again. Maybe I’m trying to adopt everything a little too early, but I feel like doing that would drastically reduce the hurdle others might feel to make the jump.