https://dagster.io/ logo
#ask-community
Title
# ask-community
f

Frederik Hagelund

03/01/2023, 12:20 PM
Hey Everyone I am trying to create a multi partition asset that uploads a table to an tableau extract. Right now i am thinking about creating following partitions: Date and Mode (Append or overwrite). The reason is that i want people to be able to run a full refresh (overwrite), but on a daily basis it should be scheduled to Append. Is there anyway to programmatically refresh the partition UI, so if it is overwrite it will make it look like it is missing all the partitions? Hope it make sense
a

AJ Floersch

03/01/2023, 1:07 PM
@Frederik Hagelund I don't have a solution for you, but I'm curious about this as well. I have a similar case where when a user manually materializes an asset, I would like for it to do a full refresh. However, for normal scheduling I would like for it to do an incremental refresh based on it's partition definition. Haven't found an OOTB way to do this yet - aside from having the user select the full-range of backfill partitions. Unfortunately that method raises it's own set of issues.
s

sandy

03/01/2023, 8:41 PM
I think this is the best way we have right now to support this: https://github.com/dagster-io/dagster/discussions/11653
r

R. Amaral Vieira

04/11/2023, 5:48 PM
@Frederik Hagelund are you updating the extract locally or directly in the server? I haven't tried this yet, but the
datasource.update_hyper_data
method in the
tableauserverclient
package seems interesting for this (https://tableau.github.io/server-client-python/docs/api-ref#data-sources). You can choose the actions and conditions (https://help.tableau.com/current/api/rest_api/en-us/REST/rest_api_how_to_update_data_to_hyper.htm#action-batch-descriptions) so that it deletes a partition and then inserts the data for that partition from an hyper file. This could make it suitable for both incremental loads and backfills. Would so much love to see a dagster-tableau integration using this concept! 🙂
4 Views