The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Hey I'm new to dagster and was curious if there was any guides on how to write a simple abstraction layer to easily port existing map/reduce code running in MRJob to dagster. My impression is dagster supports a more general computation paradigm than map/reduce and this should be possible, any guidance is greatly appreciated. 

hey jeffrey! sorry we didn’t get back to you here, could you pop this question into the support channel?

I think the short answer here is that we don’t have guides for this. We have made it possible for users to move their airflow tasks over to Dagster without any substantial change, but haven’t created these same abstractions for MRJob