Hello Guys, I am new to

DAGSTER

I have just installed it and tried to execute demo code. What I am trying to achieve is as follow, Please excuse me me if I am asking silly question as I am trying to understand it 1. Download Zip file from web URL 2. Unzip it 3. Read CSV from the unzip files 4. Validate CSV (validating data like type,value range etc) and Apply Some Transformation - Join, Merge 5. Save CSV back to specified path Can this work flow possible to implement in Dagster and is there any similar example any one can point me ? Thanks

I guess that as long as you can separate the steps you mention into some sort of pipeline process (input->solid_1->intermediate_out->solid_2) You can implement it in dagster. The particular internals of the solids of course depen on your specific use case(reading the zip file, joining csv). I would say that if you want to form an idea of how to connect solids and build pipelines you can check out the tutorial https://docs.dagster.io/docs/tutorial. I guess that down the road you can do fancy stuff such as having a resource that automatically receives a link and returns alll the csvs read. Also using datatypes and hydrations, but these could be improvements over the initial version of your pipelines.

Than you @Cris for putting me on right track. I will try to implement it.

You're welcome!