Simrun Basuita05/14/2023, 1:35 PM
Cody Peterson05/16/2023, 5:16 PM
Zach05/16/2023, 6:55 PM
Pablo Beltran05/16/2023, 10:06 PM
Chris Comeau05/17/2023, 1:27 PM
Mark Fickett05/17/2023, 5:31 PM
josh05/17/2023, 5:38 PM
Adam Bloom05/17/2023, 6:35 PM
and noticed a dagit regression - on the run page, long tags used to display a tooltip containing the full tag (as they are truncated in the run table) - that tooltip no longer seems to appear
Francesco Piccoli05/17/2023, 7:37 PM
Francesco Piccoli05/17/2023, 8:40 PM
Daniel Kim05/18/2023, 2:20 AM
R Lucas05/19/2023, 8:52 AM
) search seems to be limited to asset/group/job names but does not seem to allow search on description or dagster types. Is this feature expected in your backlog or will we need to use a dedicated data catalog tool (such as DataHub) to manage exploration of data assets / (search in fields / description) ?
Mark Fickett05/19/2023, 1:01 PM
Spencer Nelson05/19/2023, 4:34 PM
. Details aren’t important - what matters is that dataframes and tables are not compatible. So I have these old assets with keys like
, which are pandas dataframes. But I want to convert to `Table`s. Here are the options I see: 1. Reuse the
asset key, but return a non-dataframe type. Which is confusing, and will break dependents, which can possibly be managed with asset versioning in some way? But the name confusion would be unfortunate. 2. Write new
asset, change dependents to use it, and then delete
But this would destroy all history and orphan the materialized assets. Historical runs will be… broken? I don’t know what will happen to the dagit UI for them. 3. Write the new asset, but then keep
as relics of a bygone era. But they’ll clutter the UI and the codebase forever. Is there a way to mark assets as “archived” or “deprecated” or “just kept around in the attic?” Gradual migrations like this are really important. I think Dagster could provide tools to manage this, and they could be fantastically better than anything else out there, since Dagster knows so much about my computation graph. I don’t have a concrete suggestion but think this is an important area for new features.
Leo Qin05/19/2023, 6:39 PM
Félix Tremblay05/23/2023, 10:43 PM
Cody Peterson05/24/2023, 2:06 AM
Clement Emmanuel05/24/2023, 4:01 PM
Which seems to be invoked by
SELECT event_logs.id, event_logs.event FROM event_logs WHERE event_logs.dagster_event_type = $1 ORDER BY event_logs.id DESC
Becomes very expensive as the event_logs table grows (which is indefinite I believe as it's essentially a write-only table). This is expensive even with the appropriate indexes that get leveraged by the query plan. Has there been any throughput testing on this pattern, or any ideas as to how to optimize this. Unless i'm missing something this seems to make the canonical use of multi-asset sensors non viable even at a fairly modest scale as it will only decay in performance as materializations continue until it eventually (or in the case that materializations already exist when turning on the sensor, immediately) can't complete within the hard 60 second timeout
context: MultiAssetSensorEvaluationContext context.latest_materialization_records_by_partition_and_asset()
Mark Fickett05/24/2023, 5:56 PM
Jordan05/24/2023, 9:04 PM
Chris Comeau05/25/2023, 1:40 AM
Chris Comeau05/25/2023, 2:53 AM
rex05/25/2023, 9:55 PM
(0.19.6), we added some experimental APIs to help our users create software-defined assets from their dbt projects in a more composable and ergonomic way. I’ve posted an overview of the proposed API changes in a GitHub Discussion — please chime in if you have any questions or feedback about the proposed changes! Here’s a brief run-through of the changes: “”" There are three main new interfaces that we plan to release as part of this release: 1.
(link), a decorator-based interface for users to define software-defined assets from their dbt resources. 2.
(link), an improved and lighter interface for invoking dbt commands within the Dagster framework. 3.
(link), a representation of your dbt project’s resources (models, tests, macros, etc), which will be used as arguments when invoking (1) and (2). “”"
clay05/26/2023, 5:51 PM
Matt Herling05/26/2023, 8:04 PM
Caelan Schneider05/30/2023, 4:54 PM
Pablo Beltran05/31/2023, 3:01 AM
Stephen Bailey05/31/2023, 11:06 AM
at the top, if the launchpad would otherwise be empty, or for the button to highlighted in some way. I imagine a brand new user is going to be unlikely to just explore and click those buttons.
Richard Vargas06/01/2023, 10:17 PM