Simrun Basuita
05/14/2023, 1:35 PMCody Peterson
05/16/2023, 5:16 PMZach
05/16/2023, 6:55 PMPablo Beltran
05/16/2023, 10:06 PMChris Comeau
05/17/2023, 1:27 PMMark Fickett
05/17/2023, 5:31 PMjosh
05/17/2023, 5:38 PMAdam Bloom
05/17/2023, 6:35 PM1.3.4
and noticed a dagit regression - on the run page, long tags used to display a tooltip containing the full tag (as they are truncated in the run table) - that tooltip no longer seems to appearFrancesco Piccoli
05/17/2023, 7:37 PMFrancesco Piccoli
05/17/2023, 8:40 PMDaniel Kim
05/18/2023, 2:20 AMR Lucas
05/19/2023, 8:52 AM1.3.4
) search seems to be limited to asset/group/job names but does not seem to allow search on description or dagster types.
Is this feature expected in your backlog or will we need to use a dedicated data catalog tool (such as DataHub) to manage exploration of data assets / (search in fields / description) ?Mark Fickett
05/19/2023, 1:01 PMSpencer Nelson
05/19/2023, 4:34 PMTable
. Details aren’t important - what matters is that dataframes and tables are not compatible.
So I have these old assets with keys like ztf_source_dataframe
, which are pandas dataframes. But I want to convert to `Table`s. Here are the options I see:
1. Reuse the ztf_source_dataframe
asset key, but return a non-dataframe type. Which is confusing, and will break dependents, which can possibly be managed with asset versioning in some way? But the name confusion would be unfortunate.
2. Write new ztf_source_table
asset, change dependents to use it, and then delete ztf_source_dataframe.
But this would destroy all history and orphan the materialized assets. Historical runs will be… broken? I don’t know what will happen to the dagit UI for them.
3. Write the new asset, but then keep ztf_source_dataframe
as relics of a bygone era. But they’ll clutter the UI and the codebase forever. Is there a way to mark assets as “archived” or “deprecated” or “just kept around in the attic?”
Gradual migrations like this are really important. I think Dagster could provide tools to manage this, and they could be fantastically better than anything else out there, since Dagster knows so much about my computation graph. I don’t have a concrete suggestion but think this is an important area for new features.Leo Qin
05/19/2023, 6:39 PMFélix Tremblay
05/23/2023, 10:43 PMCody Peterson
05/24/2023, 2:06 AMClement Emmanuel
05/24/2023, 4:01 PMSELECT
event_logs.id,
event_logs.event
FROM
event_logs
WHERE
event_logs.dagster_event_type = $1
ORDER BY
event_logs.id DESC
Which seems to be invoked by
context: MultiAssetSensorEvaluationContext
context.latest_materialization_records_by_partition_and_asset()
Becomes very expensive as the event_logs table grows (which is indefinite I believe as it's essentially a write-only table). This is expensive even with the appropriate indexes that get leveraged by the query plan.
Has there been any throughput testing on this pattern, or any ideas as to how to optimize this. Unless i'm missing something this seems to make the canonical use of multi-asset sensors non viable even at a fairly modest scale as it will only decay in performance as materializations continue until it eventually (or in the case that materializations already exist when turning on the sensor, immediately) can't complete within the hard 60 second timeoutMark Fickett
05/24/2023, 5:56 PMJordan
05/24/2023, 9:04 PMChris Comeau
05/25/2023, 1:40 AMChris Comeau
05/25/2023, 2:53 AMrex
05/25/2023, 9:55 PMdagster-dbt
(0.19.6), we added some experimental APIs to help our users create software-defined assets from their dbt projects in a more composable and ergonomic way.
I’ve posted an overview of the proposed API changes in a GitHub Discussion — please chime in if you have any questions or feedback about the proposed changes!
Here’s a brief run-through of the changes:
“”"
There are three main new interfaces that we plan to release as part of this release:
1. @dbt_assets
(link), a decorator-based interface for users to define software-defined assets from their dbt resources.
2. DbtCli
(link), an improved and lighter interface for invoking dbt commands within the Dagster framework.
3. DbtManifest
(link), a representation of your dbt project’s resources (models, tests, macros, etc), which will be used as arguments when invoking (1) and (2).
“”"clay
05/26/2023, 5:51 PMclay
05/26/2023, 5:51 PMMatt Herling
05/26/2023, 8:04 PMCaelan Schneider
05/30/2023, 4:54 PMPablo Beltran
05/31/2023, 3:01 AMStephen Bailey
05/31/2023, 11:06 AMops
at the top, if the launchpad would otherwise be empty, or for the button to highlighted in some way. I imagine a brand new user is going to be unlikely to just explore and click those buttons.Richard Vargas
06/01/2023, 10:17 PM