19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
announcements
dagster-airbyte
dagster-airflow
dagster-bigquery
dagster-cloud
dagster-cube
dagster-dask
dagster-dbt
dagster-de
dagster-ecs
dagster-feedback
dagster-kubernetes
dagster-machine-learning
dagster-noteable
dagster-releases
dagster-serverless
dagster-showcase
dagster-snowflake
dagster-spatial
dagster-support
dagster-wandb
dagstereo
data-platform-design
events
faq-read-me-before-posting
gigs-freelance
github-discussions
introductions
jobs
random
tools
豆瓣酱帮
Title
d
Denis Rakhimov
06/04/2021, 10:32 AMQueue problem
Very few tasks in progress in the queue
Runs only 2-5 tasks
d
daniel
06/04/2021, 11:39 AMHi Denis - is it possible to share the logs from your dagster-daemon process / are there any errors listed on the Status tab next to “Run Queue”?
d
Denis Rakhimov
06/04/2021, 11:49 AMThere are no errors in the daemon.
Dagster launched in docker
j
johann
06/04/2021, 11:53 AMCould you share a ‘docker top’ result during this?
d
Denis Rakhimov
06/04/2021, 12:00 PMd
daniel
06/04/2021, 12:58 PMWhat about in the stdout/stderr logs from the dagster-daemon process/container? If runs are getting dropped for some reason there might be some clues there. Also what version of dagster is this?
d
Denis Rakhimov
06/04/2021, 2:12 PMdagster 11.11
stderr
04.06.2021 17:09:02[2021-06-04 14:09:02,839: INFO/MainProcess] dagster-d264f7@26deac8967de ready.
04.06.2021 17:09:042021-06-04 14:09:04 - dagster - DEBUG - nsk_process_sending - 97618e27-74fd-4cfe-941d-48180a81be8e - 200 - PIPELINE_START - Started execution of pipeline "nsk_process_sending".
04.06.2021 17:09:042021-06-04 14:09:04 - dagster - DEBUG - nsk_process_sending - 97618e27-74fd-4cfe-941d-48180a81be8e - 200 - ENGINE_EVENT - Executing steps in process (pid: 200)
04.06.2021 17:09:042021-06-04 14:09:04 - dagster - DEBUG - nsk_process_sending - 97618e27-74fd-4cfe-941d-48180a81be8e - 200 - ENGINE_EVENT - Starting initialization of resources [io_manager].
04.06.2021 17:09:042021-06-04 14:09:04 - dagster - DEBUG - nsk_process_sending - 97618e27-74fd-4cfe-941d-48180a81be8e - 200 - ENGINE_EVENT - Finished initialization of resources [io_manager].
04.06.2021 17:09:062021-06-04 14:09:06 - dagster - DEBUG - nsk_process_sending - 97618e27-74fd-4cfe-941d-48180a81be8e - 200 - ENGINE_EVENT - Finished steps in process (pid: 200) in 1.15s
04.06.2021 17:09:062021-06-04 14:09:06 - dagster - DEBUG - nsk_process_sending - 97618e27-74fd-4cfe-941d-48180a81be8e - 200 - PIPELINE_SUCCESS - Finished execution of pipeline "nsk_process_sending".
04.06.2021 17:09:072021-06-04 14:09:07 - dagster - DEBUG - process_receive - f30c6e65-bd3d-4231-bf89-6c03aad87c89 - 337 - PIPELINE_START - Started execution of pipeline "process_receive".
04.06.2021 17:09:072021-06-04 14:09:07 - dagster - DEBUG - process_receive - f30c6e65-bd3d-4231-bf89-6c03aad87c89 - 337 - ENGINE_EVENT - Executing steps in process (pid: 337)
04.06.2021 17:09:072021-06-04 14:09:07 - dagster - DEBUG - process_receive - f30c6e65-bd3d-4231-bf89-6c03aad87c89 - 337 - transfer_receipt - ENGINE_EVENT - Starting initialization of resources [io_manager].
04.06.2021 17:09:072021-06-04 14:09:07 - dagster - DEBUG - process_receive - f30c6e65-bd3d-4231-bf89-6c03aad87c89 - 337 - transfer_receipt - ENGINE_EVENT - Finished initialization of resources [io_manager].
04.06.2021 17:09:082021-06-04 14:09:07 - dagster - DEBUG - process_receive - f30c6e65-bd3d-4231-bf89-6c03aad87c89 - 337 - transfer_receipt - LOGS_CAPTURED - Started capturing logs for solid: transfer_receipt.
04.06.2021 17:09:082021-06-04 14:09:07 - dagster - DEBUG - process_receive - f30c6e65-bd3d-4231-bf89-6c03aad87c89 - 337 - transfer_receipt - STEP_START - Started execution of step "transfer_receipt".
04.06.2021 17:09:102021-06-04 14:09:10 - dagster - DEBUG - process_shipment - dc151dc3-bde5-4296-9d0f-976d09b78810 - 487 - PIPELINE_START - Started execution of pipeline "process_shipment".
04.06.2021 17:09:102021-06-04 14:09:10 - dagster - DEBUG - process_shipment - dc151dc3-bde5-4296-9d0f-976d09b78810 - 487 - ENGINE_EVENT - Executing steps in process (pid: 487)
04.06.2021 17:09:102021-06-04 14:09:10 - dagster - DEBUG - process_shipment - dc151dc3-bde5-4296-9d0f-976d09b78810 - 487 - ENGINE_EVENT - Starting initialization of resources [io_manager].
04.06.2021 17:09:102021-06-04 14:09:10 - dagster - DEBUG - process_shipment - dc151dc3-bde5-4296-9d0f-976d09b78810 - 487 - ENGINE_EVENT - Finished initialization of resources [io_manager].
04.06.2021 17:09:122021-06-04 14:09:12 - dagster - DEBUG - nsk_process_shipment - 0de686bb-6051-419e-b95a-9d2d9d846afd - 587 - PIPELINE_START - Started execution of pipeline "nsk_process_shipment".
04.06.2021 17:09:122021-06-04 14:09:12 - dagster - DEBUG - nsk_process_shipment - 0de686bb-6051-419e-b95a-9d2d9d846afd - 587 - ENGINE_EVENT - Executing steps in process (pid: 587)
04.06.2021 17:09:122021-06-04 14:09:12 - dagster - DEBUG - nsk_process_shipment - 0de686bb-6051-419e-b95a-9d2d9d846afd - 587 - ENGINE_EVENT - Starting initialization of resources [io_manager].
04.06.2021 17:09:122021-06-04 14:09:12 - dagster - DEBUG - nsk_process_shipment - 0de686bb-6051-419e-b95a-9d2d9d846afd - 587 - ENGINE_EVENT - Finished initialization of resources [io_manager].
04.06.2021 17:09:122021-06-04 14:09:12 - dagster - DEBUG - process_shipment - dc151dc3-bde5-4296-9d0f-976d09b78810 - 487 - ENGINE_EVENT - Finished steps in process (pid: 487) in 2.28s
04.06.2021 17:09:122021-06-04 14:09:12 - dagster - DEBUG - process_shipment - dc151dc3-bde5-4296-9d0f-976d09b78810 - 487 - PIPELINE_SUCCESS - Finished execution of pipeline "process_shipment".
04.06.2021 17:09:132021-06-04 14:09:13 - dagster - DEBUG - process_sending - c568b6a3-4b0e-427e-974e-4a52481152ef - 696 - PIPELINE_START - Started execution of pipeline "process_sending".
04.06.2021 17:09:132021-06-04 14:09:13 - dagster - DEBUG - process_sending - c568b6a3-4b0e-427e-974e-4a52481152ef - 696 - ENGINE_EVENT - Executing steps in process (pid: 696)
04.06.2021 17:09:142021-06-04 14:09:14 - dagster - DEBUG - process_sending - c568b6a3-4b0e-427e-974e-4a52481152ef - 696 - ENGINE_EVENT - Starting initialization of resources [io_manager].
04.06.2021 17:09:142021-06-04 14:09:14 - dagster - DEBUG - process_sending - c568b6a3-4b0e-427e-974e-4a52481152ef - 696 - ENGINE_EVENT - Finished initialization of resources [io_manager].
04.06.2021 17:09:142021-06-04 14:09:14 - dagster - DEBUG - process_sending - c568b6a3-4b0e-427e-974e-4a52481152ef - 696 - ENGINE_EVENT - Finished steps in process (pid: 696) in 513ms
04.06.2021 17:09:142021-06-04 14:09:14 - dagster - DEBUG - process_sending - c568b6a3-4b0e-427e-974e-4a52481152ef - 696 - PIPELINE_SUCCESS - Finished execution of pipeline "process_sending".
04.06.2021 17:09:152021-06-04 14:09:15 - dagster - DEBUG - nsk_process_shipment - 0de686bb-6051-419e-b95a-9d2d9d846afd - 587 - ENGINE_EVENT - Finished steps in process (pid: 587) in 3.31s
04.06.2021 17:09:152021-06-04 14:09:15 - dagster - DEBUG - pipe_line_sql_process_data_exch - c85b68a0-6473-4602-aae4-25d7c3099e72 - 763 - PIPELINE_START - Started execution of pipeline "pipe_line_sql_process_data_exch".
04.06.2021 17:09:152021-06-04 14:09:15 - dagster - DEBUG - nsk_process_shipment - 0de686bb-6051-419e-b95a-9d2d9d846afd - 587 - PIPELINE_SUCCESS - Finished execution of pipeline "nsk_process_shipment".
04.06.2021 17:09:152021-06-04 14:09:15 - dagster - DEBUG - pipe_line_sql_process_data_exch - c85b68a0-6473-4602-aae4-25d7c3099e72 - 763 - ENGINE_EVENT - Executing steps in process (pid: 763)
04.06.2021 17:09:152021-06-04 14:09:15 - dagster - DEBUG - pipe_line_sql_process_data_exch - c85b68a0-6473-4602-aae4-25d7c3099e72 - 763 - reader_old - ENGINE_EVENT - Starting initialization of resources [connector, io_manager].
04.06.2021 17:09:152021-06-04 14:09:15 - dagster - INFO - resource:connector - c85b68a0-6473-4602-aae4-25d7c3099e72 - reader_old - Register callback
04.06.2021 17:09:152021-06-04 14:09:15 - dagster - INFO - resource:connector - c85b68a0-6473-4602-aae4-25d7c3099e72 - reader_old - Register callback
04.06.2021 17:09:152021-06-04 14:09:15 - dagster - INFO - resource:connector - c85b68a0-6473-4602-aae4-25d7c3099e72 - reader_old - Register callback
04.06.2021 17:09:152021-06-04 14:09:15 - dagster - INFO - resource:connector - c85b68a0-6473-4602-aae4-25d7c3099e72 - reader_old - Register callback
04.06.2021 17:09:162021-06-04 14:09:16 - dagster - DEBUG - pipe_line_sql_process_data_exch - c85b68a0-6473-4602-aae4-25d7c3099e72 - 763 - reader_old - ENGINE_EVENT - Finished initialization of resources [connector, io_manager].
04.06.2021 17:09:172021-06-04 14:09:16 - dagster - DEBUG - pipe_line_sql_process_data_exch - c85b68a0-6473-4602-aae4-25d7c3099e72 - 763 - reader_old - LOGS_CAPTURED - Started capturing logs for solid: reader_old.
04.06.2021 17:09:172021-06-04 14:09:16 - dagster - DEBUG - pipe_line_sql_process_data_exch - c85b68a0-6473-4602-aae4-25d7c3099e72 - 763 - reader_old - STEP_START - Started execution of step "reader_old".
04.06.2021 17:09:172021-06-04 14:09:16 - dagster - INFO - resource:connector - c85b68a0-6473-4602-aae4-25d7c3099e72 - reader_old - Connected to MQ
04.06.2021 17:09:172021-06-04 14:09:16 - dagster - INFO - resource:connector - c85b68a0-6473-4602-aae4-25d7c3099e72 - reader_old - -> Start consume queue: khd, route_key: ['torg_ekb_analytics.main.nomencl', 'ut_msk.main.nomencl', 'ut_msk.main.price', 'pim.main.products'] during
04.06.2021 17:09:182021-06-04 14:09:18 - dagster - DEBUG - process_sending - 41d6f501-ebcd-4563-a81d-c1df59fbee05 - 888 - PIPELINE_START - Started execution of pipeline "process_sending".
04.06.2021 17:09:182021-06-04 14:09:18 - dagster - DEBUG - process_sending - 41d6f501-ebcd-4563-a81d-c1df59fbee05 - 888 - ENGINE_EVENT - Executing steps in process (pid: 888)
04.06.2021 17:09:182021-06-04 14:09:18 - dagster - DEBUG - process_sending - 41d6f501-ebcd-4563-a81d-c1df59fbee05 - 888 - ENGINE_EVENT - Starting initialization of resources [io_manager].
04.06.2021 17:09:182021-06-04 14:09:18 - dagster - DEBUG - process_sending - 41d6f501-ebcd-4563-a81d-c1df59fbee05 - 888 - ENGINE_EVENT - Finished initialization of resources [io_manager].
04.06.2021 17:09:192021-06-04 14:09:19 - dagster - DEBUG - process_sending - 41d6f501-ebcd-4563-a81d-c1df59fbee05 - 888 - ENGINE_EVENT - Finished steps in process (pid: 888) in 639ms
04.06.2021 17:09:192021-06-04 14:09:19 - dagster - DEBUG - process_sending - 41d6f501-ebcd-4563-a81d-c1df59fbee05 - 888 - PIPELINE_SUCCESS - Finished execution of pipeline "process_sending".
04.06.2021 17:09:202021-06-04 14:09:20 - dagster - DEBUG - process_sending - a9d07222-118a-4675-b56f-d08be01cbfcc - 954 - PIPELINE_START - Started execution of pipeline "process_sending".
04.06.2021 17:09:202021-06-04 14:09:20 - dagster - DEBUG - process_sending - a9d07222-118a-4675-b56f-d08be01cbfcc - 954 - ENGINE_EVENT - Executing steps in process (pid: 954)
04.06.2021 17:09:202021-06-04 14:09:20 - dagster - DEBUG - process_sending - a9d07222-118a-4675-b56f-d08be01cbfcc - 954 - ENGINE_EVENT - Starting initialization of resources [io_manager].
04.06.2021 17:09:202021-06-04 14:09:20 - dagster - DEBUG - process_sending - a9d07222-118a-4675-b56f-d08be01cbfcc - 954 - ENGINE_EVENT - Finished initialization of resources [io_manager].
04.06.2021 17:09:212021-06-04 14:09:21 - dagster - DEBUG - process_sending - a9d07222-118a-4675-b56f-d08be01cbfcc - 954j
johann
06/04/2021, 2:13 PMThanks! Could we also get
docker statsd
Denis Rakhimov
06/04/2021, 2:14 PMj
johann
06/04/2021, 2:15 PMAre you running dagster in a single container? Or using the compose setup
d
Denis Rakhimov
06/04/2021, 2:16 PMsingle container
j
johann
06/04/2021, 2:42 PMGotcha. Is it possible to tell if the container is being constrained for compute resources at all?
d
daniel
06/04/2021, 2:44 PMin addition to johann's point, is using the DockerRunLauncher to run each run in its own container something you might be interested in? It's possible that could help if you're running into some kind of per-container resource limit
j
Jamic Juraev
06/04/2021, 3:59 PM@daniel not quite clear, have we can fix it in configurations?
d
daniel
06/04/2021, 4:02 PMHi Jamic - you can configure your run_launcher to launch each run in its own container using the DockerRunLauncher like the deploy_docker example here: https://github.com/dagster-io/dagster/blob/master/examples/deploy_docker/dagster.yaml
That should cause it to launch each run in its own docker container. That example is running in a docker-compose environment though which might be different than your current setup.
The big tricky thing about using that run launcher is that the container that runs it needs to be able to run docker commands - the way we do that in our example is by giving it access to the docker socket here: https://github.com/dagster-io/dagster/blob/master/examples/deploy_docker/docker-compose.yml#L59