19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
is there any built-in watermarking in dagster? to ...
# announcementsb
Binh Pham
04/20/2020, 5:32 PMis there any built-in watermarking in dagster? to do incremental runs
a
alex
04/21/2020, 6:10 PMcan you go in to more detail about the nature of incremental runs you are hoping to accomplish? cc @yuhan
b
Binh Pham
04/21/2020, 7:01 PMhi alex, i misinterpret what dagster is used for, so I don't think this question applies.
but do have another question:
https://stackoverflow.com/questions/61330816/how-would-you-parameterize-dagster-pipelines-to-run-same-solids-with-multiple-di
could you help?
y
yuhan
04/21/2020, 8:31 PMHi Binh! reusable solids might be a good way to do that https://docs.dagster.io/latest/tutorial/reusable/
yuhan
04/21/2020, 8:32 PMor maybe you could try making a solid factory like this https://docs.dagster.io/latest/learn/how-tos/solids/#programmatically-generating-solids-solid-factories
b
Binh Pham
04/21/2020, 10:00 PMthanks I'll give it a try
Binh Pham
04/22/2020, 8:42 PMhi @yuhan, the reusable solid method works well, thank you. But lets say I have to do this for hundreds of tables. Is it scalable/maintainable to create a large config.yaml? Or are there ways to break it out into smaller config for each table?
y
yuhan
04/22/2020, 8:47 PMGlad it works for you! Do you plan to put your business logic in config (e.g. sql logic) or just some other ops level info like resource variables?
b
Binh Pham
04/22/2020, 8:55 PMNo business logic in config, path to SQL file and resource variables
y
yuhan
04/22/2020, 10:01 PMI’d say config hundreds of tables should be ok scale wise. but if you are worried about maintainability or maybe even readability, you can specify multiple yamls and pass through cli
yuhan
04/22/2020, 10:01 PM