Hi all Just started investigating Dagster last we...
# announcements
b
Hi all Just started investigating Dagster last week and trying to figure out a couple best practices for challenges I was hoping to solve. First, is there a beat practice for dynamically generating instances of solids based on the output of an upstream solid? Use case is wanting to be able to map a common (for us) pattern over a dynamic number of outputs. Would prefer not to have to update a config file for each run. Second, is there a good way to make Dagster Date aware with dagster-cron? Basically want to run on a schedule, but have the current date as a parameter that gets passed in. Realize this is encroaching heavily on airflow territory. Maybe this is a situation where airflow is actually the best tool for the job. Curious to get feedback/opinions.
I started messing with the Fan in fan out example on GitHub for the mapping, but I didn't get the sense that this would be dynamic to each run if I were to Cron out the execution.
a
We don’t have a great answer for mapping quite yet. Our native mapping solution is still under development. For now a workaround is to do fixed width mapping - exemplified here: https://github.com/dagster-io/dagster/blob/master/examples/dagster_examples/toys/sleepy.py where you divide work between N solids using lists. Not elegant, but might get you by. As for date partitioned work - we have been actively working on this problem and @sashank can guide you on what you could do today and what is coming in the next few weeks.
b
Thanks alex. I'll dig into the examples you shared to see how they can fit the challenges I described.