https://dagster.io/ logo
#ask-community
Title
# ask-community
j

Jacob Marcil

03/03/2023, 3:15 PM
Do Dagster as an option where I can configure an asset that do INSERT ONLY without any partitions? The default behavior is to Delete and Recreate the table, but I would love to just insert the new rows on each runs, but without having to configure any partition. (Same question for UPSERT) I know we can configure the asset to not return anything, and then in the function I could insert the data directly in the table I want. But I was wondering if there was an option for that if I use an asset that return a set of rows.
v

Vinnie

03/03/2023, 3:16 PM
I’m not sure if any out of the box IO Managers can be configured for that, but you can always write your own custom IO Manager with this specific logic.
j

Jacob Marcil

03/03/2023, 3:17 PM
Right. It was a Snowflake specific question. (Sorry forgot to add that to the question). Pretty sure there’s not option for that right… 😞 THank you 🙂
v

Vinnie

03/03/2023, 3:22 PM
You could probably get away by just duplicating the logic from the project_fully_featured and removing the line that calls the cleanup. https://github.com/dagster-io/dagster/blob/master/examples/experimental/project_fu[…]t_fully_featured_v2_resources/resources/snowflake_io_manager.py
s

sandy

03/03/2023, 4:28 PM
The out-of-the-box IO managers don't currently support this. If you submit a Github issue, we might be able to add support at some point
🌈 1
j

Jacob Marcil

03/06/2023, 5:21 PM
Decided to use the default behavior of Dagster assets but I’ve added a dump timestamp field with a default dates of 1970-01-01 and use that as my
partition_exp
. Works well for my usecase. Thank you all again.
e

Emir Karamehmetoglu

03/18/2023, 2:38 PM
If working with actual SQL tables DBT's "incremental" mode would do what you want but I guess you are not looking to use it..
9 Views