King Chung Huang
10/15/2020, 4:56 PMdagster_type_materializer
to receive multiple materialization configs all at once, instead of being called once per config? For example, the following solid config materializes the Dask DataFrame facets_df
to both JSON files and a database. This config works (when given real values), but I can’t construct an optimal Dask graph because the Dask dataframe_materializer()
is called for each config separately. I’d ideally like to be called with both configs, and then compute a Dask graph for them together at the same time.
solids:
extract_info:
outputs:
- facets_df:
to:
json:
path: <s3://bucket/key>
- facets_df:
to:
sql:
name: table
uri: <postgresql://server:5432/postgres>