https://dagster.io/ logo
#announcements
Title
# announcements
b

Binh Pham

04/20/2020, 5:32 PM
is there any built-in watermarking in dagster? to do incremental runs
a

alex

04/21/2020, 6:10 PM
can you go in to more detail about the nature of incremental runs you are hoping to accomplish? cc @yuhan
b

Binh Pham

04/21/2020, 7:01 PM
hi alex, i misinterpret what dagster is used for, so I don't think this question applies. but do have another question: https://stackoverflow.com/questions/61330816/how-would-you-parameterize-dagster-pipelines-to-run-same-solids-with-multiple-di could you help?
y

yuhan

04/21/2020, 8:31 PM
Hi Binh! reusable solids might be a good way to do that https://docs.dagster.io/latest/tutorial/reusable/
b

Binh Pham

04/21/2020, 10:00 PM
thanks I'll give it a try
hi @yuhan, the reusable solid method works well, thank you. But lets say I have to do this for hundreds of tables. Is it scalable/maintainable to create a large config.yaml? Or are there ways to break it out into smaller config for each table?
y

yuhan

04/22/2020, 8:47 PM
Glad it works for you! Do you plan to put your business logic in config (e.g. sql logic) or just some other ops level info like resource variables?
b

Binh Pham

04/22/2020, 8:55 PM
No business logic in config, path to SQL file and resource variables
y

yuhan

04/22/2020, 10:01 PM
I’d say config hundreds of tables should be ok scale wise. but if you are worried about maintainability or maybe even readability, you can specify multiple yamls and pass through cli