I'm currently leaning heavily on AWS Lambda and St...
# announcements
I'm currently leaning heavily on AWS Lambda and Step Functions for pipeline compute. As expected, orchestration is a nightmare. Are there plans to revive the lambda execution engine (https://github.com/dagster-io/dagster/issues/455)? In the mean time, are there any docs on rolling your own executors with the executor API (https://docs.dagster.io/_apidocs/internals#executors)? Is executing via Lambda a dead end for now?
The hark-lang project attempts to abstract away Lambda compute, but is still very immature. https://github.com/condense9/hark-lang
The best solution may be to move away from Lambda and toward Dask running on AWS Fargate to keep infrastructure maintenance costs low.
https://docs.aws.amazon.com/lambda/latest/dg/lambda-images.html AWS recently announced Lambda Container Image, sort of like GCP Cloud Run. You can upload Docker Image on a lambda, that Docker can support an AWS Lambda Handler There are limits, such as 15 minute run-time maximum, but same limit applies to normal Lambda functions. ( I never tried it, just read about it )
Hi Luke 👋 would love to hear more about your use case (are you considering kicking off a lambda per solid?)! We have users who build their own executors and run launchers (celery executor is a good reference) and dagster is designed to enable the straightforward creation of custom executors. One of our community contributors Jordan has been making key improvements to our aws integrations, and we do intend to invest more in this integration suite.