But for me in my setup - it fails. Even though I have mapped the docker socket in the exact same way. Do you have any idea what is causing the problem here?

Though:

USER dagster:dagster

the user in the dockerfile is not root

does the user inside the dockerfile need some special permissions?

I have changed the user - but still get a permission denied

Interestingly I also see failures like:

WARNING:root:Retrying failed database connection: (psycopg2.OperationalError) connection to server at "postgresql" (172.31.0.3), port 5432 failed: Connection refused

one step further: ImageNotFound: 404 Client Error for http+docker://localhost/v1.41/images/create?tag=latest&fromImage=other: Not Found ("pull access denied for other, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")

but locally the client (on the mac side ) is logged in

ok - the docker stuff is fixed now

but I am stuck at: `dagster.core.errors.DagsterInstanceSchemaOutdated: Raised an exception that may indicate that the Dagster database needs to be be migrated. Database is at revision None, head is b601eb913efa. To migrate, run

dagster instance migrate

.`

when spinning up an empty container - I would expect that the migration is run automatically

and in fact, this works for the default example - so what is the difference in my code?

and:

(psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "pg_class_relname_nsp_index"

DETAIL:  Key (relname, relnamespace)=(secondary_indexes_id_seq, 2200) already exists.

[SQL:

CREATE TABLE secondary_indexes (

id SERIAL NOT NULL,

name VARCHAR(512),

create_timestamp TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP,

migration_completed TIMESTAMP WITHOUT TIME ZONE,

PRIMARY KEY (id),

UNIQUE (name)

)

is found in the logs

postgres is also showing me several: ERROR: duplicate key value violates unique constraint "instigators_selector_id_key" errors

Strangely, the SFTP connection to the container name:

sftp

fails: unable to connect to port 2222 on 192.168.48.5

(but works nicely for localhost)

furthermore: NotFound: 404 Client Error for http+docker://localhost/v1.41/containers/ffd4684f2076cc41d455b8928ba1126cdf0e605d0f39c4ce38316bf91450fc7c/start: Not Found ("network docker_example_network not found") the docker error is now again quite irritating

Wondering if this is some initialization race condition between

dagit

and

dagster-daemon

I think I fixed a couple of mistakes - nonetheless I still cannot get the DIND queued docker launcher to work. Meanwhile 1) It no longer fails instantly but is stuck (without any further logs) when trying to spin up the container 2) dagster daemon fails to instantiate the SFTP connection 3) postgres is throwing duplicate_key constraint validation errors

docker rm <postgres container ID>

possibly with a -f

and try re-deploying?

to replicate:

git clone <https://github.com/geoHeil/dagster-ssh-demo.git>

cd dagster-ssh-demo

docker compose --profile dagster up --build

go to: http://localhost:3000/workspace/deploy_docker_repository@other/jobs/my_job/playground and try to launch

(2) is fixed now

but now all (manual dummy and sensor initiated runs) are stuck in the startup phase.

10 runs are currently in progress. Maximum is 10, won't launch more.

And postgres keeps showing logs like: ERROR: duplicate key value violates unique constraint "instigators_selector_id_key" postgresql | 2022-04-22 172216.774 UTC [1140] DETAIL: Key (selector_id)=(40ae1d09616324124ae8fab93494603eee744f81) already exists. postgresql | 2022-04-22 172216.774 UTC [1140] STATEMENT: INSERT INTO instigators (selector_id, repository_selector_id, status, instigator_type, instigator_body) VALUES ('40ae1d09616324124ae8fab93494603eee744f81', '22ed74ad3bf3c735dd23193f2387c48c5a8cc556', 'AUTOMATICALLY_RUNNING', 'SENSOR', '{"__class__": "InstigatorState", "job_specific_data": {"__class__": "SensorInstigatorData", "cursor": null, "last_run_key": null, "last_tick_timestamp": 1650648132.730657, "min_interval": 30}, "job_type": {"__enum__": "InstigatorType.SENSOR"}, "origin": {"__class__": "ExternalJobOrigin", "external_repository_origin": {"__class__": "ExternalRepositoryOrigin", "repository_location_origin": {"__class__": "GrpcServerRepositoryLocationOrigin", "host": "ssh-demo", "location_name": "ssh-demo", "port": 4000, "socket": null}, "repository_name": "SSH_DEMO"}, "job_name": "foo_scd2_asset_sensor"}, "status": {"__enum__": "InstigatorStatus.AUTOMATICALLY_RUNNING"}}') RETURNING instigators.id

the example that is working for you has the former

i'm a little confused why they aren't waiting for the postgres container to spin up even though they have depends_on: set

actually taking this line out is making postgres behave better for me:

-    volumes:
-      - ./postgres-dagster:/var/lib/postgresql/data

the broad advice that I have is to start with the example that is working and then incrementally add things that are different until it stops working - i think it will be a lot easier to isolate problems to a specific change that way

also postgres 11 is showing the duplicate key warnings/errors

Are you able to actually get runs to finish running (and not have them stuck in the getting started phase?

but I am not running into the same DB consistency checks that you are running into as the daemon is running

However, runs are still stuck launching: dagster.daemon.QueuedRunCoordinatorDaemon - INFO - Launched 2 runs. but the logs show that they clearly have been received

When commenting out this block:

# run_launcher:
#   module: dagster_docker
#   class: DockerRunLauncher
#   config:
#     env_vars:
#       - DAGSTER_POSTGRES_USER
#       - DAGSTER_POSTGRES_PASSWORD
#       - DAGSTER_POSTGRES_DB
#     network: dagster_network
#     container_kwargs:
#       auto_remove: true

the tasks start to execute.

(in dagster.yaml )

This seems strage though:

dagster-daemon    | 2022-04-23 10:19:16 +0000 - dagster.daemon.QueuedRunCoordinatorDaemon - INFO - Retrieved 3 queued runs, checking limits.
dagster-daemon    | 2022-04-23 10:19:19 +0000 - dagster.daemon.QueuedRunCoordinatorDaemon - INFO - Launched 3 runs.
dagster-daemon    | INFO  [dagster.daemon.QueuedRunCoordinatorDaemon] Launched 3 runs.
dagster-daemon    | DEBUG [dagster.daemon.SchedulerDaemon] Not checking for any runs since no schedules have been started.
dagster-daemon    | DEBUG [dagster.daemon.QueuedRunCoordinatorDaemon] Poll returned no queued runs.

unfortunately, so far I did not find any clues yet

I can only see: [DockerRunLauncher] Launching run in a new container 337b6b03bc2dc28bda38318700100de882ababb349388c587acb318b829c6cc3 with image dagster-ssh-demo_other

though I set these here: x-app-vars: &default-app-vars DAGSTER_POSTGRES_HOSTNAME: "postgresql"

(and pass it to all the containers)

But why does the link you sent not include the hostname?

(and is working)

this fixes the run launcher

let me re-enable the other services successively and then work on mounting the volumes.

But I am curious - you mentioned that dropping the volume mount makes your postgres behave better - how did this have any influence here at all?

It seemed like having it in the volume was keeping the postgres container from starting up correctly

I am on to the next step now: The pyspark resource (in local mode) is not coming up:

RuntimeError: Java gateway process exited before sending its port number
  File "/opt/conda/lib/python3.9/site-packages/dagster/core/errors.py", line 184, in user_code_error_boundary
    yield
  File "/opt/conda/lib/python3.9/site-packages/dagster/core/execution/resources_init.py", line 298, in single_resource_event_generator
    resource_def.resource_fn(context)
  File "/opt/conda/lib/python3.9/site-packages/dagster_pyspark/resources.py", line 53, in pyspark_resource
    return PySparkResource(init_context.resource_config["spark_conf"])
  File "/opt/conda/lib/python3.9/site-packages/dagster_pyspark/resources.py", line 20, in __init__
    self._spark_session = spark_session_from_config(spark_conf)
  File "/opt/conda/lib/python3.9/site-packages/dagster_pyspark/resources.py", line 15, in spark_session_from_config
    return builder.getOrCreate()
  File "/opt/conda/lib/python3.9/site-packages/pyspark/sql/session.py", line 228, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
  File "/opt/conda/lib/python3.9/site-packages/pyspark/context.py", line 392, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "/opt/conda/lib/python3.9/site-packages/pyspark/context.py", line 144, in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
  File "/opt/conda/lib/python3.9/site-packages/pyspark/context.py", line 339, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
  File "/opt/conda/lib/python3.9/site-packages/pyspark/java_gateway.py", line 108, in launch_gateway
    raise RuntimeError("Java gateway process exited before sending its port number")

But so far I did not see anything suspicious in the logs of the container.

this looks like JAVA_HOME is not set error

(but the logs) do not show this directly

This looks like https://github.com/geoHeil/dagster-ssh-demo/blob/master/environment.yml#L12 (conda pyspark) is not bringing in the downstream JDK dependency and not setting java home

spark is starting now

Is the postgres error/warning: ERROR: duplicate key value violates unique constraint "instigators_selector_id_key" anything I should worry about?

I can confirm that a lot of the dummy example jobs are working now! However the state_handling for the mappend volumes is not yet working as expected

Also strange: too many retries for DB connection sometimes dagit & the dagster-daemon fail to start with this error- albeit they are set to depend-on / wait for the DB container

@daniel is my suspicion correct that the ``DockerRunLauncher`` is not using the volume mappings defined in docker-compose? How can I set up the volume mappings for the run launcher so they are applied when starting the container?

Even when mounting an absolute path:

/path/to/dagster-ssh-demo/warehouse_location_dagster:/opt/dagster/dagster_home/warehouse_location

no files are written to this directory

is the github repo that you posted earlier up to date with the code that you're using that repros this?

what I think is happening with the postgres volume thing is that making it mount that data as a volume on startup causes it to take a lot longer to start up, so other services that depends on it start spewing some errors while they are waiting for postgres to be ready (adding

depends_on

in your docker-compose file just makes the containers wait for the container to start, they don't make it wait to be fully ready). For me the daemon and dagit eventually reach an OK place and are able to run correctly - there's just some spew at the beginning while they wait to be able to connect to postgres

there are some tips in the docker docs for controlling service order more explicitly if you want to make the other services specifically wait for postgres to be ready: https://docs.docker.com/compose/startup-order/

The "duplicate key value violates unique constraint" errors are actually expected currently (it adds the row if it doesn't exist, then updates it if it does exist and that constraint fires) but we'll see what we can do to make that less spew-y

that leaves the issue you described about mounting volumes in the run launcher not working - can you share more details about what exactly I should do to reproduce that? Which job i should run, what the exact expected behavior is vs. what you're seeing, and what code is supposed to be writing to the volume?

In particular the IO managers are writing to this location https://github.com/geoHeil/dagster-ssh-demo/blob/master/SSH_DEMO/resources/parquet_io_manager.py#L99 the ingest assets https://github.com/geoHeil/dagster-ssh-demo/blob/master/SSH_DEMO/assets/ingest_assets.py read from t he SFTP docker container - and store it into the warehouse but they store it locally in the container which gets deleted (as the volume mappings which are applied a) from docker-compose and b) from the launch configuration of dagit.yaml for the docker-based executor somehow seem to work in a a different way

You do not need to run any job.

git clone <https://github.com/geoHeil/dagster-ssh-demo.git>
cd dagster-ssh-demo

make start 
# or alternatively without make
docker compose --profile dagster up --build

is all what is needed - the sensors start to automatically poll the SFTP resource for ingestable files

i.e. if you changed it from

volumes:
-        - warehouse_location_dagster:/opt/dagster/dagster_home/warehouse_location

to something like (this is my absolute path, yours is probably different):

volumes:
         - /Users/dgibson/dagster-ssh-demo/warehouse_location_dagster:/opt/dagster/dagster_home/warehouse_location

I think it would be more likely to work. I tried that and am still getting an error in your job, but i think that may be logic in your modified IO manager now? I'd hope that the volume would work as expectedn ow

when I ran

docker inspect <container ID>

on a launched container, I saw

"Mounts": [
            {
                "Type": "bind",
                "Source": "/Users/dgibson/dagster-ssh-demo/warehouse_location_dagster",
                "Destination": "/opt/dagster/dagster_home/warehouse_location",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],

which matched the path on the gRPC server

I see how needing to specify the absolute path is annoying - it's tricky because dagster.yaml is getting loaded and interpreted inside the Docker container, where it doesn't have any way to know what the base directory to use for a relative path that's referring to the filesystem outside of Docker

let me re try/check with docker inspect

I get:

"Mounts": [ { "Type": "bind", "Source": "/Users/geoheil/Downloads/fooo/dagster-ssh-demo/warehouse_location_dagster", "Destination": "/opt/dagster/dagster_home/warehouse_location", "Mode": "", "RW": true, "Propagation": "rprivate" } ]

for:

- /Users/geoheil/Downloads/fooo/dagster-ssh-demo/warehouse_location_dagster:/opt/dagster/dagster_home/warehouse_location

But still get: Path does not exist: file:/opt/dagster/dagster_home/src/warehouse_location/foo_asset.

I think I need to adapt the path mapping to:

/Users/geoheil/Downloads/fooo/dagster-ssh-demo/warehouse_location_dagster:/opt/dagster/dagster_home/src/warehouse_location

(include the src)

Interestingly:

Error: No arguments given and workspace.yaml not found.

I get this error then from dagster daemon (did not have that one before this change)

Cool - this error indeed is solved with the absolute path and including the

src

in the mapping (see latest commit)

Except for the combined_assset_sensor everything else works fine. This sensor however does not fire from within docker.