@daniel Does dagster integrates with kerberos/ldap? Does it allows you to set user roles for dagit web?

I know of at least one company using tags to manually attribute pipeline runs to people, but nothing automatic

Re: loading new repositories: how often would you think you would need to do this?

I'm looking at using the venv approach for user code isolation, trying to figure out how to push updates in that scenario

am I going to footgun myself by live updating a running server like that (while user code is actively running)? don't think so, though I can imagine race conditions if it happens while the grpc process is starting up...

and the grpc server wont shutdown til its active executions complete

this bit didn't make sense to me btw: https://docs.dagster.io/overview/repositories-workspaces/workspaces#executing-runs-against-a-grpc-server

but it could probably be improved either way cc @daniel

so that's another execution remote approach besides celery/k8s?

out of curiosity, what's the life cycle for a local grpc process? a new one for each solid execution, or one per dagit, or...?

one per “repository location” which is roughly equivalent to

load_from

entry in

workspace.yaml

tbh, grpc seems preferable to celery if you're not using k8s IMO

and I'm probably missing something in the docs, but not seeing any way to control parallelism at all - ie, "only run at most 3 instances of a solid at the same time".

Ray does this with arbitrary named resources attached to a task (ie, "this task reserves 1 bigquery connection while running")

that might be a deal-breaker for me 😞

I have a client that will eventually have only a handful of pipelines, but will run each one with 100s/1000s different parameters

is there parallelism control at the pipeline level?

(next release is ~November yeah)

PLEEEEASE DON'T MAKE ME USE AIRFLOW 😛

thx, very helpful and much appreciated

this is a really big issue for me with backfill - client wants to backfill 2 years of data occasionally, with daily partitions (don't ask me, I'm just a consultant here), which AFAICT would launch >800 simultaneous solid instances...

the semaphore approach is... kinda blah - launch a bunch of process only to have them immediately block on a lockfile or start polling a database in a spin loop

has no one encountered this before?

theres also this example of setting up a scheduler to progress through a backfill, using

should_execute

to check for currently active runs before firing the next one cumbersome - but i figured worth sharing https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-test/dagster_test/toys/schedules.py#L9-L86

just to clarify, is the parallelism control in the next version only for pipelines, or also at the solid level?

re: following progress, we're not using issues for tracking planned work for the release but will absolutely be posting updates as plans crystallize there. It sounds like using celery for either type of parallelism control is a deal-breaker for you which is helpful to know, would you be willing to share more about why that is? Is it just extra complexity/needing to manage yet another piece of deployed infra?

and yeah, aversion to celery (and curiosity about grpc as an alternative) is mainly infra overhead