https://dagster.io/ logo
#ask-community
Title
# ask-community
m

Maksym Domariev

07/22/2022, 4:43 AM
hey I've been passively exploring dagster for several weeks, docs looks clear and very well designed. However feels like docs are focused around "perfect" scenario where data is either an S3 or DBT and the volume is small. I have a super simple use case, I have DB1 and DB2 with different traits. I'd like to load from these DB's in chunks( partitioning ) and create assets for other team to experiment DB1 is time-series so dagster built-in partitioning seems super easy `QUESTION 1:`DB2 needs a different partitioning, what abstraction should i use for a simple
SELECT * FROM abc where ID=dfg LIMIT X OFFSET Y
How should I configure limit and pass offset? Any source code doing that?
QUESTION 2:
How should I rate limit such queries? I know how to do that in a "regular code" using let's say redis. Are there any standard way Do I miss anything with my questions? Should some of this problems to be solved by other solution, that's why they are not in the docs?