Stephen Bailey
12/21/2022, 6:49 PM@code_location
suffix is such a sneaky cruel product nudge to get me to move off repositories.Tojo Cherian
12/22/2022, 10:31 AMUnexpected GraphQL error
Operation name: CodeLocationStatusQuery
Message: Cannot query field "locationStatusesOrError" on type "DagitQuery". Did you mean "instigationStateOrError", "partitionSetsOrError", "partitionSetOrError", "executionPlanOrError" or "unloadableInstigationStatesOrError"?
Path:
Locations: [{"line":2,"column":3}]
Daniel Mosesson
12/23/2022, 1:07 PMGabe Schine
12/24/2022, 2:56 AMAssetsDefinition.from_graph()
to accept an output_required
boolean flag like @asset
does?Uri Laserson
12/24/2022, 3:47 PMMitchell Hynes
12/28/2022, 5:54 PMPeter Davidson
12/28/2022, 5:58 PMYevhenii Soboliev
12/28/2022, 9:05 PMKulanjith Deelaka
12/29/2022, 10:28 AMUPathIOManager
the method context.add_output_metadata()
does not work, rather we have to implement UPathIOManager.get_metadata()
. It would be helpful if the docs explicitly stated that calling context.add_output_metadata()
from UPathIOManager.dump_to_path()
does not work.Stephen Bailey
12/31/2022, 1:21 AMStephen Bailey
01/03/2023, 3:17 PMclaire
01/04/2023, 9:35 PMclaire
01/04/2023, 9:35 PMMartin Picard
01/05/2023, 9:43 AMRohil Badkundri
01/05/2023, 1:44 PMload_input
method of an IOManager (loading the output of the upstream op into the downstream op)?Martin Picard
01/05/2023, 3:29 PM@asset
def random_integers(context) -> Output[DataFrame]:
df = DataFrame(
{
"x": random.randint(1, 10, size=random.randint(1, 100)),
}
)
return Output(df, metadata={"num_rows": df.shape[0]})
claire
01/05/2023, 8:23 PMStephen Bailey
01/05/2023, 8:28 PMreturn RunRequest(asset_selection=list(asset_keys))
Zachary Bluhm
01/06/2023, 8:17 PMstale
even though they were backfilled.
I've now run an all-time backfill twice for a job and both times at least one asset still showed stale
in the UI
Looks like an issue has already been filed: https://github.com/dagster-io/dagster/issues/11442Zach
01/06/2023, 11:36 PMArthur
01/08/2023, 5:39 PMArthur
01/08/2023, 5:39 PMStephen Bailey
01/09/2023, 2:16 PMthis_asset
to that_asset
2. Change repository name, but not the AssetKey, from this_repo
to that_repo
.
In Terraform, you can run a command like terraform state mv ADDRESS_A ADDRESS_B
, and I think something similar here would be ideal. Otherwise, renaming an asset requires a backfill of the partition sets AFAICT.josh
01/10/2023, 4:30 PMPang Wu
01/10/2023, 10:31 PMStephen Bailey
01/11/2023, 1:42 AMJuan Blanco
01/11/2023, 2:06 PMZachary Bluhm
01/11/2023, 4:30 PM@asset(
partitions_def=HourlyPartitionsDefinition(start_date="2023-01-01-00:00", timezone=DEFAULT_TIMEZONE),
ins={
"hourly_depends_on_past": AssetIn(
partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)
)
}
)
def hourly_depends_on_past(hourly_depends_on_past):
time.sleep(3)
return 1
And backfilled 10 partitions at a time. In theory I would expect this to finish locally with no K8s overhead extremely quickly (maybe < 90 secs), but in reality this took closer to 5.5 minutes. What's interesting is that each individual OP only took ~6.5 secondsAlexander Whillas
01/11/2023, 9:49 PMCasper Weiss Bang
01/12/2023, 8:28 AM