I wonder, if there are some more people here who are interested in spatial/geo data processing. Would love to learn how you use dagster and apply it to e.g. GeoPandas, GeoArrow, OSM or whatever datasets you are working with… If there is sufficient interest, a new channel e.g.

dagster-spatial

or

dagster-geo

might be an option!? @Gabe Schine @Philip Orlando @Fabio Picchi

Thank you! This is definitely helpful. Will join.

@Holger Bruch I would love to see more discussions happening around dagster and geospatial. I'm still in the very early stages of incorporating dagster into my stack, but so far I've been planning to use a combination of GeoPandas and PostgreSQL with the PostGIS extension. I'm really hoping that Elementl will release a new PostgreSQL and pandas integration to help me decouple business logic from IO. Extending something like this to include GeoPandas and PostGIS would make Dagster an unstoppable force within the geospatial data engineering community! Looking forward to hearing other folks' experiences working with dagster and geospatial too!

Ping me directly if you need a new channel created for this. And let me know the preferred channel name.

I’ll leave that up to @Holger Bruch. dagster-spatial has my vote though!

@Fraser Marlow thanks for the offer! So let’s use

dagster-spatial

then.

Hey, @Holger Bruch we’ve been using Dagster heavily at BestPlace, I had a at Dagster Community Meeting about our infra for spatial data analysis quite a while ago. I’m not hands-on with it right now (though @Dmitry Krylov is), but would love to connect on linkedin and help or just keep an eye on your discussions if you decide to start

dagster-spatial

@matas thanks for sharing this!

this exact talk is more about our processes of interaction between engineering and analytics teams within dagster, and not about the spatial data stack we’ve used, but I’m happy to share more if it is interesting

I'd love to learn more about what external resources/backend you're using to manage your spatial data, and how they integrate with dagster. I'm still focusing on integrating dagster with geopandas (or geopolars) and postgis, but please let me know if there are any other options worth considering.

we’ve been using mongo as our main spatial data db. we actually had several iterations of mongo-postgis back-and-forth trials 😃 but for our case (hundreds of not really big datasets from different sources with constantly changing structure) it was much better than postgis and geopandas for more complex transforms python-side

@Holger Bruch thanks for getting this convo going. And thanks @matas for sharing that video. I will gladly join a

dagster-spatial

channel. I am currently incorporating dagster into my GIS stack. Like @Philip Orlando I also hope for a Postgres-pandas integration / io manager.

Join us at #dagster-spatial

niiiice! Thanks a lot @Holger Bruch for setting this up