Found a breaking change in dbt 1.3 when I upgraded to test python models; the

raw_sql

property in the

manifest.json

changed to

raw_code

(also in sql models), causing this line to fail.

The dagster docs should have collapsible Left Nav bar. On smaller screens, left nav bar takes up quite a lot of space.

Not sure if this is a good place to ask, but are there any case studies for running dagster hybrid in a multi-tenant/customer scenario. For instance, is there capabilities to run specific repos on specific agents? That way we can have code that is customer specific execute inside that customers cloud environments.

Some positivity this Friday: I just upgraded to 1.0.11 and have to say, I really love the new experimental workspace page!

hey friends <!here>, cross-posting from #announcements to solicit your feedback on improving Dagster. We’re going through a process of prioritizing new features, example repos, integrations, improvements to our content, etc, and it’d really help to hear from you. <https://forms.gle/xVgfJbubQpfMiBJh7%7CHere’s a <5m survey with a few questions about your team, your tech, and where you want us to focus our time>. In exchange for your input, we’ll send you some Dagster sticker swag. Woohoo! Whether you respond or not, the Dagster team thanks you for your ongoing support.

Looks like the page https://docs.dagster.io/concepts/partitions-schedules-sensors/sensors leads to a 404

The new feature

load_asset_value

added in 1.0.11 looks really helpful. I was trying it out with partitions and it seems like the method currently doesn't support that, is this correct?

The

multi_asset_sensor

doesn’t seem to be working with assets across different repositories. I have my extract-load jobs separate from my dbt transformations and only wanna trigger the dbt job when all related data is loaded. Using a

multi_asset_sensor

raises the exception

"Exception iterating responses: AssetKey(s) {'foo/bar', 'foo/baz'} were selected, but no AssetsDefinition objects supply these keys. Make sure all keys are spelled correctly, and all AssetsDefinitions are correctly added to the repository."

. This method works fine using a regular asset sensor and attaching it to only one of the assets such as

@asset_sensor(asset_key=AssetKey(["foo", "bar"]))

running 1.0.12

I am having hard time figuring out correct options to build this flow out. Between Assets/MultiAsset/Graph I feel in the weeds on how best to achieve this. Even then, the errors are super confusing. Additional Details including code and erorrs are in the 2 thread replies

Hi guys, I noticed that in the description of the release 1.0.10, the third bullet point in the section "New" refers to the

run_key

being added as an optional argument to

RunRequest

and

run_request_for_partition

. I believe it's actually the argument

asset_selection

that was added as a new optional argument to

run_request_for_partition

. This feature was very helpful to my use case where I need to run a partitioned job and simultaneously provide a

partition_key

and an

asset_selection

.

thanks for pointing this out - I passed it on to our website folks

Documentation question: why did the dbt resources lose their docs on the individual functions? as an example, https://docs.dagster.io/0.15.9/_apidocs/libraries/dagster-dbt#dagster_dbt.DbtCliResource has docs for build, run, test, etc. Those are not present on https://docs.dagster.io/_apidocs/libraries/dagster-dbt#dagster_dbt.DbtCliResource

Hi! Slight bug in the "step status for backfill" view. I'm using

end_offset = 1

in a

MonthlyPartitionsDefinition

. The current month then shows up in the start rather than the end.

Would love to have a simple function that would "smart-cast" dictionaries into metadata for assets. We have several flows where we produce a dictionary of metadata about an object, and then log it, like so:

@asset
def my_asset():
   # code to actually create the asset
   object_id = do_something()

   # code to do a /describe call from an api
   details = requests.get(f"some_url/{object_id")
   context.add_output_metadata(details)

   return object_id

However, we frequently hit errors in the ops this way, especially around array values:

dagster._core.errors.DagsterInvalidMetadata: Could not resolve the metadata value for "public_access_id" to a known type. Its type was <class 'NoneType'>. Consider wrapping the value with the appropriate MetadataValue type.

. Would be great to have a

sanitize_metadata

argument that by default drops unparseable values, or something.

it would be really cool and helpful to have a job-specific timeline page so you can get a better resolution view of when multiple runs of the same job are running at overlapping times

it'd be nice if the View Configuration button on the Runs page created a window that you could drag the sides / corners to expand. it currently pops up a really small window, which makes it a pain to navigate / view more than a couple lines of a large config at a time. it also makes wider configs difficult to view

New look of the timeline rocks 🚀 , thanks team! Tabs for repositories and a time-scale significantly improve readability there.

On the "Overview" page, would it be possible to alpha sort repositories on the jobs/schedules/sensors tabs, the same way that the timeline page is?

Some nice examples in this one for adding and using column metadata, thanks. https://dagster.io/blog/smoke-test-data-pipeline TableSchema looks like a good chance to tie in with Arrow, which then goes well with DuckDB. Article about using them together: https://arrow.apache.org/blog/2021/12/03/arrow-duckdb/ If it's applicable, there's a ton of activity happening around Arrow, improvements for partitioned datasets, etc... was thinking it might help save some work.

I might be missing something, but there doesn’t seem to be a way to subset the global asset lineage by asset groups, is there? If I have the same group in multiple repositories, I’d have to find a way to include all the asset keys pertaining to the group in the search. Would be a cool (and maybe easy?) QoL improvement.

as discussed in a #dagster-support thread....can we get repository context on dagster-daemon logs? job names/sensor names etc can be reused between repositories, which makes the current sensor and schedule tick logs a bit confusing

dagit feedback - the

Partitions

tab of a partitioned job has started showing

0

for missing partitions, even when the

Launch backfill...

button correctly identifies missing partitions

Hi there, I've started using dagster this week, and the first thing I've noticed is the Field + dagster types when configurating in assets. It is quite close to SQLAlchemy syntax. I've wondered if there you guys ever thought about integrating pydantic intrinsically, like tiangolo did at https://sqlmodel.tiangolo.com/

This is a bit different from normal patterns I think, but I've been thinking about how to make it easier for new users to understand what Dagster is doing. One great thing about dbt onboarding is that it is simply: add a SQL file, do

dbt run

, see your table come into existence. It seems like there could be some really nice asset-based tutorials if there was a

dagster asset materialize <selector>

command line function. This would allow users to write asset files and run them from the CLI without the overhead of setting up Dagit, creating jobs, etc. A

dagster asset plan <selector>

would also be useful if it listed out whether assets information actually existed (i.e. whether previous materliaizations existed).

<!here> do we have any airbyte users out there who are interested in being a design partner as we expand our integration? we’d just share the new functionality preemptively, and ask for your feedback

In the Dagster Cloud UI, if I'm viewing one deployment and then I get automatically signed out, when I sign back in I get dropped back on the default deployment. It would be handy if I dropped back in wherever I had been before.

Heard a funny optical illusion with the logo: someone mentioned to me that they read it as a left-facing face with a big nose and smile, they found it strange. They had a good laugh when I explained. Probably not a common thing, really have to squint to see it, but now I'm curious. Prior logo had more depth cues... this might be a side effect of the move to flat colour.

Developer experience feedback - type annotations for

context.resources

,

context.op_config

, etc? I have no idea if this is at all feasible (likely not), but currently all type information is lost on these objects. Just had a silly typo on a function within a custom resource - would have been much easier to catch (or not have in the first place) if auto-suggest from VS Code was functioning correctly.

it seems that when launchpad inputs are large the "View tags and config" view breaks - it displays the beginning of the config but there's no scroll bar or "copy config" button. (this was with a 2100-line config that was programmatically generated)

feature request: 1. max “queued runs” # per job 2. slack alert if we hit the max queue # this will help with our monitoring for long-running or frequently-failing + retrying jobs, especially where we have tag concurrency limits. we end up building up hundreds of runs in the queue yikes