For anyone out there who has deployed Dagster successfully to Azure and AKS, was wondering what people used for IO managers and storage locations? From what I can tell there isn't really an equivalent to the AWS s3 bucket pickle IO manager described in the AWS deployment guide. Curious if anyone else working in Azure has a successful pattern for deploying to AKS and using k8s job executors. thanks!
09/07/2022, 8:27 PM
Hi @Mark Atkins i haven't personally tried to deploy dagster to azure so hopefully someone else will be able to add some advice, but we do have an ADLS2 IO manager (source code here api docs here). i'll be the first to admit it's not our best documented library, but i'm happy to help out! at a high level, using the adls io manager should be conceptually identical to the s3 io manager
09/07/2022, 9:04 PM
thanks Jamie! I was looking through the docs today and just saw that. I think the last time I had read through the Azure integration docs was a few months ago so I didn't realize that existed. I'll give it a go with our deployment and reach out if anything comes up! thanks again for the quick response