https://dagster.io/ logo
Title
g

geoHeil

05/23/2023, 5:57 PM
Hi, I am using dagster in conjunction with OSM and postGIS
👋 2
:rainbow-daggy: 4
but in the past have done a lot of big geospatial analyes (outside of dagster) with spark: geomesa, and sedona
💯 1
p

Philip Orlando

05/23/2023, 5:58 PM
Not sure if you're able to share any examples, but it would be super helpful to learn more about how you're managing IO with Dagster and PostGIS.
g

geoHeil

05/23/2023, 6:01 PM
Any SQL related task is handled from DBT 😉 but I guess this is not what you are asking about. For example for putting data to postGIS using an IO manager sometimes I use hacky notebooks and the normal python interfaces i.e. geo-sqlalchemy / geo-pandas to directly write to the DB
👍 1
In cases where larger geospatial data i.e. from H3 hexagons should be loaded via d6tstack and then GIS-ified to native types using SQL
Osmosis (OSM) is pushing the stuff directly to postGIS - but I have added asset materialization events (retrofitted) to make it dagster aware
👀 1
But if you want we can have a chat on Zoom about more details
p

Philip Orlando

05/23/2023, 9:18 PM
Thanks for sharing this. I need to spend some more time getting to know Dagster right now, so having these reference points are helpful for me. I'll definitely reach back out once I feel like I'm ready to discuss more over a call.
:rainbow-daggy: 1