Hi all :wave: I'm currently looking to teach mysel...
# dagster-plus
a
Hi all 👋 I'm currently looking to teach myself Dagster and am deciding whether to use Serverless or Hybrid. I wanted to ask: 1. Are there any security advantages of using the Hybrid model (and storing the secrets as AWS secrets) vs using Serverless and storing secrets as environment variables in the Dagster Cloud UI. 2. If using Serverless is it best practise to store all secrets in Dagster Cloud UI or to store secrets in AWS secrets and only storing one secret in Dagster Cloud UI (to access AWS Secrets) Many thanks 🙂
j
welcome @Adam Samuel!
So one of the main reasons we see customers choose hybrid over serverless is that it provides much stronger security gaurentees. Mainly it can be boiled down to: With serverless you are ok with Dagster Cloud having access to the entire lifecycle of your application. With hybrid the execution of your jobs can be isolated from Dagster cloud entirely. In terms of secrets, Dagster Cloud encrypts anything you provide but you are accepting the risk that we might make some blunder (we use KMS for encryption). If you're more confident in your own internal process then hybrid secrets backed by Secrets Manager can be more secure.
If you go with serverless I'd recommend just using the Dagster Cloud UI/cli for secrets instead of an intermediate secret store
if you want an over view of how cloud handles secrets this post https://dagster.io/blog/dagster-managing-secrets will be useful
a
Thank you @Joe, that's super helpful. With serverless does that also mean any requirements.txt packages will need to be installed each time a dagster job is run?
j
no in serverless you'll either deploy a docker image or a PEX which will be used for runs
a
Brilliant, thank you for your help! I think I'm going to try Serverless to start with but just wanted to be aware of security risks 🙂