Hi Dagster team! I'm trying to use docker-compose + docker runner to run locally a dummy pipeline. This is something that I got working last week, but now it has stopped working... I can see in dagit

[DockerRunLauncher] Launching run in container five_oracle_pipeline_dagster_oracle_pipelines with ID ceacbfd6707e3cfa09dce15898d1923b3e4edb14de818a9cf39bfc0390a4cfe1

And in the docker-compose output:

dagster_docker_daemon       | 2021-01-27 14:56:41 - SchedulerDaemon - INFO - Not checking for any runs since no schedules have been started.
dagster_docker_daemon       | 2021-01-27 14:56:41 - SensorDaemon - INFO - Not checking for any runs since no sensors have been started.
dagster_docker_daemon       | 2021-01-27 14:56:41 - QueuedRunCoordinatorDaemon - INFO - Retrieved 1 queued runs, checking limits.
dagster_docker_daemon       | 2021-01-27 14:56:43 - QueuedRunCoordinatorDaemon - INFO - Launched 1 runs.
dagster_docker_daemon       | 2021-01-27 14:56:47 - QueuedRunCoordinatorDaemon - INFO - Poll returned no queued runs.
dagster_docker_daemon       | 2021-01-27 14:56:47 - QueuedRunCoordinatorDaemon - INFO - Launched 0 runs.

But nothing actually runs... Any hints what could be happening?

Hi Fran - is there anything useful in the logs for the container that gets launched? Something might be failing early enough in the container that it doesn't make its way back into dagit

That's the thing, I don't see any container being launched...

hmmmm, strange. At that point all that's left to do is call container.start() in the docker API - i'd expect that to raise some kind of exception if it failed to start, which should then appear in the logs. This launcher is fairly new though, so it's possible there's a failure scenario that isn't getting surfaced the way that it should

Mmm, there is something interesting, when I execute

dagster instance info

in the container I get an error

dagster.check.CheckError: Failure condition: Couldn't import module dagster_postgres.run_storage when attempting to load the configurable class dagster_postgres.run_storage.PostgresRunStorage

So it seems that

dagster-postgres

isn't installed

Got it - generally we could be better about handling errors when the resource that a run launcher spins up fails early enough that it can't even create the instance (e.g. a missing module error like that). I'm still surprised that there weren't any docker logs for the container though

I'd have expected it to fail when launching the grpc server as the configured RunLauncher has missing dependencies

I would too. I'll see if I can reproduce by taking dagster-postgres out of the pipeline container in the example

It's working now 👍

Ah, so I think it doesn't actually need to load anything on the instance in order to display the pipelines in dagit / launch the gRPC server. So that's why it didn't fail there.

Nothing for me... that's weird...

weird!

I checked using

docker logs

and just to triple-check, it's 'docker logs ceacbfd6707e3cfa09dce15898d1923b3e4edb14de818a9cf39bfc0390a4cfe1' (from your example above)?

No, actually the logs of that one show the error

ahhh, OK. It's a bit confusing - in that example the gRPC server is just used to serve the pipeline metadata. The DockerRunLauncher that the example uses doesn't actually hit the gRPC server, it launches a new container using the same image that was used to launch the gRPC server. (That way each run can use its own container)

So, dagit makes the request to dagster_daemon, and dagster_daemon interacts with the docker API to create the container, is that right?

that's right. Well technically dagit queues the run and then the daemon polls for queued runs, but the docker API part is exactly right.

Makes sense, for some reason I thought it would be the gRPC server the one creating the container

Yeah, exactly. What you thought was very reasonable since there's already a container with that image running, i'll tweak the message on the DockerRunLauncher to emphasize that it's a new container.

And then, if I configure the RunLauncher to run in the same process, does the queue commands the gRPC server to run a pipeline?

Yeah, the DefaultRunLauncher (the one that is used if you don't change your instance) calls out to the gRPC server and tells it to execute the pipeline there

Gotcha. I think with this I have enough to keep experimenting 😃

awesome, thanks for the feedback and for trying it out

You guys are doing an awesome job with Dagster 👏