Mattis Busse

05/11/2023, 10:20 AM
Dear Dagster-Team, is it possible or are you thinking about an AWS Glue integration with dagster? At the moment you can create an aws glue resource via boto3 and use it in dagster but with mayor restrictions. I would like to migrate my pyspark code from the aws glue console into dagster assets, use io_managers and have the glue logs integrated into dagster. A fully fletched integration such as with AWS EMR would make the decision to use dagster for all our pipelines very easy. Kind Regards, Mattis (HDI Germany)