ChrisH07/12/2022, 11:35 PM
2. Manually run PrepFiles.py
a. Ensure the file is utf-8 encoded
b. Create a copy in
3. Manually run QuickLoadToPostgres.py
a. read csv into dataframe
b. minimal transformation
c. enrich with metadata
d. load into postgres database
i. successful files written to
ii. failed files written to
4. Manually run dbt run
a. transforms data into final table for reporting & analysis
(This always runs flawlessly! 😉)
What I'm hoping Dagster can do is replace each of the "Manually" statements.
I see how I can integrate dbt, though I haven't tried it yet.
What I'm not really seeing in the tutorial is an example of using Dagster to run existing Python scripts.
Am I thinking about this wrong? Is there a similar example I can learn from?
Qwame07/13/2022, 7:10 AM
Mark07/13/2022, 7:55 AM
yuhan07/13/2022, 8:42 PM