I have dagit deployed on a Kubernetes cluster and it's regularly being restarted due to failure to respond to health checks (liveness probes), but there's no logs to indicate what's going on inside dagit's web server. is there any way to enable those logs?

Hi all, I'm trying to get dagster to run through dask; I have a dask scheduler and worker running via docker locally. I can't seem to get the config right, any pointers?

Hi all. Had a quick question on coding patterns for expectations. Suppose I have a data verification step within a solid which should be treated as terminal or non-terminal, depending on the configuration of the pipeline (eg., maybe in dev mode, all failed expectations are terminal, whereas in production runs a failed expectation is only recorded). I can obviously write code that either raises a Failure or yields an ExpectationResult based on the mode that pipeline is running in. Just wondering if there is already API for doing that that I'm overlooking?

How do I actually configure which run launcher to use?

Looks like it's a repo level config thing?

Also is there a formal API spec for run launchers or should I just copy all the functions from Default? 🙂

Ahh looks like even fewer are needed, sync-in-memory has a pretty minimal API

It is configured on the instance. dagster.yaml

Finally looping back around to trying my hand at a CeleryLauncher

There actually isn’t a celery run launcher – you’re probably looking for the celery executor. More info here about executing on celery here: https://docs.dagster.io/deploying/celery. An explanation of run launchers and executors here: https://docs.dagster.io/overview/run-launchers-executors/run-launcher. More info about the instance and configuring run launchers here: https://docs.dagster.io/overview/instances/dagster-instance#instance-configuration-yaml

What is inst_data actually for?

Nothing seems to actually use it

From the name I'm guessing it's some data about the instance 🙂

How would a launcher store some k/v pair data into the run that it can access later? It seems like a lot of plugins store that all in memory but that would mean losing the associations if dagit restarts.

I could mutate the run_config I guess?

All the examples I can find seem transient 😞

@Noah K we use tags for this purpose in the GrpcRunLauncher

I'm looking to write a solid which runs a job in a Kubernetes cluster. is there any existing code which would make this easier? i'm guessing in dagster-k8s, but not sure where to look

Hi, for the hello, dagster example the code looks as follows: execute_pipeline(hello_pipeline)

if someone is running this line of code from his own virtualenv or his own docker container, is there a way to run such command to execute_pipeline on a separate centralized server, different from the one he runs the execute_pipeline command

You mean launching a pipeline via the API?

@Noah K very good point

yes that would probably work with one caveat, when one execute_pipeline, is there also a way to schedule it to run at certain times regularly?

do you have examples of the API i can look if it's exactly what i mean

https://docs.dagster.io/overview/graphql-api#launch-a-pipeline-run shows how to use the API

https://docs.dagster.io/overview/scheduling-partitions/schedules shows how to set up scheduled runs

this probably works. and then i could create a python wrapper to make it custom to our settings... also is there a way to define a pipeline w/o executing and later scheduling it?

Yes? Pipelines are defined when it starts up, you don't need to run anything

but when i call the API with execute_pipeline it does run the pipeline... i m refering the first time I am defining a pipeline

Are you using Dagit? "defining" the pipeline happens when it starts up