https://dagster.io/ logo
Title
j

Jakob Thrane

05/11/2023, 2:23 PM
Hello Dagster people! I'm in the process of moving our code locations from development, to a "staging" service. The staging environment is largely created using the guide provided in https://docs.dagster.io/deployment/guides/docker. It's been pretty smooth sailing so far, however, I have suddenly been facing the issue of the launched docker container exiting before finishing the job (with no feedback in dagit). Note: Another code location works fine with zero issues. The error as described by the logs of the exited docker container:
Usage: dagster api execute_run [OPTIONS] INPUT_JSON
Try 'dagster api execute_run -h' for help.

Error: Got unexpected extra arguments ("ExecuteRunArgs", "instance_ref": {"__class__": "InstanceRef", "compute_logs_data": {"__class__": "ConfigurableClassData", "class_name": "LocalComputeLogManager", "config_yaml": "base_dir: /opt/dagster/dagster_home/storage\n", "module_name": "dagster.core.storage.local_compute_log_manager"}, "custom_instance_class_data": null, "event_storage_data": {"__class__": "ConfigurableClassData", "class_name": "PostgresEventLogStorage", "config_yaml": "postgres_db:\n db_name:\n env: DAGSTER_POSTGRES_DB\n hostname:\n env: DAGSTER_POSTGRES_HOST\n password:\n env: DAGSTER_POSTGRES_PASSWORD\n port: 5432\n username:\n env: DAGSTER_POSTGRES_USER\n", "module_name": "dagster_postgres"}, "local_artifact_storage_data": {"__class__": "ConfigurableClassData", "class_name": "LocalArtifactStorage", "config_yaml": "base_dir: /ai4xray_2/orchestration/data_storage\n", "module_name": "dagster.core.storage.root"}, "run_coordinator_data": {"__class__": "ConfigurableClassData", "class_name": "QueuedRunCoordinator", "config_yaml": "{}\n", "module_name": "dagster.core.run_coordinator"}, "run_launcher_data": {"__class__": "ConfigurableClassData", "class_name": "DockerRunLauncher", "config_yaml": "container_kwargs:\n volumes:\n - /var/run/docker.sock:/var/run/docker.sock\n - /ai4xray_2/orchestration/data_storage:/ai4xray_2/orchestration/data_storage\n - /ai4xray_1/data_vault:/ai4xray_1/data_vault\nenv_vars:\n- DAGSTER_POSTGRES_HOST\n- DAGSTER_POSTGRES_USER\n- DAGSTER_POSTGRES_PASSWORD\n- DAGSTER_POSTGRES_DB\n- POSTGRES_DAGSTER_PASSWORD\n- DATA_ROOT\n- DAGSTER_MODE\n- BASE_SCHEMA\nnetwork: postgres_network\n", "module_name": "dagster_docker"}, "run_storage_data": {"__class__": "ConfigurableClassData", "class_name": "PostgresRunStorage", "config_yaml": "postgres_db:\n db_name:\n env: DAGSTER_POSTGRES_DB\n hostname:\n env: DAGSTER_POSTGRES_HOST\n password:\n env: DAGSTER_POSTGRES_PASSWORD\n port: 5432\n username:\n env: DAGSTER_POSTGRES_USER\n", "module_name": "dagster_postgres"}, "schedule_storage_data": {"__class__": "ConfigurableClassData", "class_name": "PostgresScheduleStorage", "config_yaml": "postgres_db:\n db_name:\n env: DAGSTER_POSTGRES_DB\n hostname:\n env: DAGSTER_POSTGRES_HOST\n password:\n env: DAGSTER_POSTGRES_PASSWORD\n port: 5432\n username:\n env: DAGSTER_POSTGRES_USER\n", "module_name": "dagster_postgres"}, "scheduler_data": {"__class__": "ConfigurableClassData", "class_name": "DagsterDaemonScheduler", "config_yaml": "{}\n", "module_name": "dagster.core.scheduler"}, "secrets_loader_data": null, "settings": {}, "storage_data": {"__class__": "ConfigurableClassData", "class_name": "DagsterPostgresStorage", "config_yaml": "postgres_db:\n db_name:\n env: DAGSTER_POSTGRES_DB\n hostname:\n env: DAGSTER_POSTGRES_HOST\n password:\n env: DAGSTER_POSTGRES_PASSWORD\n port: 5432\n username:\n env: DAGSTER_POSTGRES_USER\n", "module_name": "dagster_postgres"}}, "pipeline_origin": {"__class__": "PipelinePythonOrigin", "pipeline_name": "take_snapshot", "repository_origin": {"__class__": "RepositoryPythonOrigin", "code_pointer": {"__class__": "ModuleCodePointer", "fn_name": "definitions", "module": "annotator.orchestration", "working_directory": "/opt/pysetup"}, "container_context": {}, "container_image": "annotator_backend_image", "entry_point": ["dagster"], "executable_path": "/opt/pysetup/.venv/bin/python"}}, "pipeline_run_id": "13b11cf8-b134-4e68-a52b-72781de304be", "set_exit_code_on_failure": null})
The command looks a bit messy, but since it's a
Got unexpected extra arguments
I'm guessing it could be a change of API? However I cannot find anything related in the changelog. Can anyone point me in the direction for debugging this? Thanks! :)
:dagster-bot-resolve: 1
d

daniel

05/11/2023, 2:28 PM
Hi Jakob - does the image that you're using contain an ENTRYPOINT?
Actually, another idea - have you checked what version of dagster is running in the container that launched the job?
j

Jakob Thrane

05/11/2023, 2:29 PM
Hi daniel 🙂 yes, ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-m", "annotator.orchestration"] in my image
d

daniel

05/11/2023, 2:29 PM
Try setting that to the CMD instead of the ENTRYPOINT
j

Jakob Thrane

05/11/2023, 2:29 PM
Yes, just found theres a version difference between the container, and the daemon
d

daniel

05/11/2023, 2:30 PM
what are the two versions?
j

Jakob Thrane

05/11/2023, 2:30 PM
1.3.3 in the daemon and 1.3.1 in my image
d

daniel

05/11/2023, 2:30 PM
ok, that shouldn't matter - probably the CMD thing
dagster updates the CMD but not the ENTRYPOINT, so i think it might be smashing them together?
❤️ 1
j

Jakob Thrane

05/11/2023, 2:33 PM
Ah! That makes sense! I think that's the solution. Cheers daniel 🙂
:condagster: 1