https://dagster.io/ logo
Join Slack
Powered by
Hello, came across a bug in the daemon CLI, opened...
# ask-community
y
Hello, came across a bug in the daemon CLI, opened an issue explaining it: https://github.com/dagster-io/dagster/issues/16029 Did anyone encounter such issue on v1.4.7 as well?
d
Hi Yuval, I'm having trouble reproducing this, so I suspect it may have something to do with a specific operating system. Can you share more about what machine you're running dagster on?
y
We are trying to run it locally on our intel based Macs
d
Has this started happening recently on a new version? Or has it always been happening?
y
No we are trying to upgrade from 1.2.6 to 1.4.7
d
my attempt to reproduce: • check out dagster 1.4.7 (on my non M1 mac) • write hi.py:
Copy code
from dagster import Definitions, job, op


@op
def my_op():
    pass


@job
def my_job():
    my_op()


defs = Definitions(jobs=[my_job])
• run both 
dagster dev -f hi.py
and 
dagster-daemon run -f hi.py
- both start without issue So something strange must be happening if those same steps are leading to a port/socket error for you
y
Copy code
@repository
def demo_repo():
    return [demo_dev_job]
this is my repository file
d
Are you able to see what subprocess it's trying to open that has 'dagster api grpc' in it? might be tricky to find if its failing right away
y
ill repro and send full logs
d
can you try one other thing for me - what's the output of this Python script?
Copy code
from dagster._utils import safe_tempfile_path_unmanaged
print(safe_tempfile_path_unmanaged())
y
/var/folders/k5/dlhcjkks0y7d0xg5pth56dth0000gn/T/tmpxz4c8as6
d
ok, no problems there
y
demo.log
demo.log
demo2.log
demo2.log
d
Do you possibly have the DAGSTER_GRPC_PORT environment variable set?
echo $DAGSTER_GRPC_PORT
y
nope, as you can see in the first log file i ran it without the port
and in the second log i run it with the port
d
mind posting the output of that command just to rule it out?
y
Copy code
dagster-landing-service % echo $DAGSTER_GRPC_PORT
 %
d
What happens if you run this command locally:
Copy code
/Users/yuvalgrinberg/envs/dagster-landing-service/bin/python -m dagster api grpc --lazy-load-user-code --socket /var/folders/k5/dlhcjkks0y7d0xg5pth56dth0000gn/T/tmp5r7qtgie --heartbeat --heartbeat-timeout 20 --fixed-server-id bd4da9d1-80e4-4049-b743-dd3f888ed855 --log-level warning --location-name dev.py -f /Users/yuvalgrinberg/PycharmProjects/dagster-landing-service/dagster_landing_service/repository/dev.py -d /Users/yuvalgrinberg/PycharmProjects/dagster-landing-service"
y
i opened the issue after struggling for two days with this 🙃
d
and if you go back to 1.2.6 does the problem go away?
y
yep
Copy code
(dagster-landing-service) yuvalgrinberg@Yuvals-MacBook-Pro dagster-landing-service % /Users/yuvalgrinberg/envs/dagster-landing-service/bin/python -m dagster api grpc --lazy-load-user-code --socket /var/folders/k5/dlhcjkks0y7d0xg5pth56dth0000gn/T/tmp5r7qtgie --heartbeat --heartbeat-timeout 20 --fixed-server-id bd4da9d1-80e4-4049-b743-dd3f888ed855 --log-level warning --location-name dev.py -f /Users/yuvalgrinberg/PycharmProjects/dagster-landing-service/dagster_landing_service/repository/dev.py -d /Users/yuvalgrinberg/PycharmProjects/dagster-landing-service


(dagster-landing-service) yuvalgrinberg@Yuvals-MacBook-Pro dagster-landing-service %
i found a work around to run without the daemon
only running GRPC and webserver
d
i would expect that command to either open the server and keep it running (with logs) or to raise an exception - I wouldn't expect it to not have any ouput or return immediately without an exception
is that different from what you're seeing?
i.e. i would expect to see a log line like this:
Copy code
2023-08-23 10:44:55 -0400 - dagster.code_server - INFO - Started Dagster code server for file hi.py on port 4000 in process 7524
Or an error like this:
Copy code
Traceback (most recent call last):
  File "/Users/dgibson/.pyenv/versions/dagster-3.10.11/bin/dagster", line 33, in <module>
    sys.exit(load_entry_point('dagster', 'console_scripts', 'dagster')())
  File "/Users/dgibson/dagster/python_modules/dagster/dagster/_cli/__init__.py", line 48, in main
    cli(auto_envvar_prefix=ENV_PREFIX)  # pylint:disable=E1123
  File "/Users/dgibson/.pyenv/versions/3.10.11/envs/dagster-3.10.11/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/dgibson/.pyenv/versions/3.10.11/envs/dagster-3.10.11/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/dgibson/.pyenv/versions/3.10.11/envs/dagster-3.10.11/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/dgibson/.pyenv/versions/3.10.11/envs/dagster-3.10.11/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/dgibson/.pyenv/versions/3.10.11/envs/dagster-3.10.11/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/dgibson/.pyenv/versions/3.10.11/envs/dagster-3.10.11/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/dgibson/dagster/python_modules/dagster/dagster/_cli/api.py", line 688, in grpc_command
    server = DagsterGrpcServer(
  File "/Users/dgibson/dagster/python_modules/dagster/dagster/_grpc/server.py", line 947, in __init__
    check.invariant(
  File "/Users/dgibson/dagster/python_modules/dagster/dagster/_check/__init__.py", line 1589, in invariant
    raise CheckError(f"Invariant failed. Description: {desc}")
dagster._check.CheckError: Invariant failed. Description: You must pass one and only one of `port` or `socket`.
y
maybe the code opens those tmp files / dirs and now they dont exist
Copy code
Stack Trace:
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_core/host_representation/grpc_server_registry.py", line 205, in _get_grpc_endpoint
    server_process = GrpcServerProcess(
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_grpc/server.py", line 1222, in __init__
    server_process = open_server_process(
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_grpc/server.py", line 1129, in open_server_process
    wait_for_grpc_server(server_process, client, subprocess_args, timeout=startup_timeout)
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_grpc/server.py", line 1063, in wait_for_grpc_server
    raise Exception(

  warnings.warn(
2023-08-23 17:35:35 +0300 - dagster.daemon - INFO - Instance is configured with the following daemons: ['AssetDaemon', 'BackfillDaemon', 'SchedulerDaemon', 'SensorDaemon']
2023-08-23 17:35:35 +0300 - dagster.daemon.SensorDaemon - WARNING - Could not load location dev.py to check for sensors due to the following error: Exception: gRPC server exited with return code 1 while starting up with the command: "/Users/yuvalgrinberg/envs/dagster-landing-service/bin/python -m dagster api grpc --lazy-load-user-code --socket /var/folders/k5/dlhcjkks0y7d0xg5pth56dth0000gn/T/tmp5r7qtgie --heartbeat --heartbeat-timeout 20 --fixed-server-id bd4da9d1-80e4-4049-b743-dd3f888ed855 --log-level warning --inject-env-vars-from-instance --instance-ref {"__class__": "InstanceRef", "compute_logs_data": {"__class__": "ConfigurableClassData", "class_name": "LocalComputeLogManager", "config_yaml": "base_dir: /Users/yuvalgrinberg/dagster_home/storage\n", "module_name": "dagster.core.storage.local_compute_log_manager"}, "custom_instance_class_data": null, "event_storage_data": {"__class__": "ConfigurableClassData", "class_name": "SqliteEventLogStorage", "config_yaml": "base_dir: /Users/yuvalgrinberg/dagster_home/history/runs/\n", "module_name": "dagster.core.storage.event_log"}, "local_artifact_storage_data": {"__class__": "ConfigurableClassData", "class_name": "LocalArtifactStorage", "config_yaml": "base_dir: /Users/yuvalgrinberg/dagster_home\n", "module_name": "dagster.core.storage.root"}, "run_coordinator_data": {"__class__": "ConfigurableClassData", "class_name": "DefaultRunCoordinator", "config_yaml": "{}\n", "module_name": "dagster.core.run_coordinator"}, "run_launcher_data": {"__class__": "ConfigurableClassData", "class_name": "DefaultRunLauncher", "config_yaml": "{}\n", "module_name": "dagster"}, "run_storage_data": {"__class__": "ConfigurableClassData", "class_name": "SqliteRunStorage", "config_yaml": "base_dir: /Users/yuvalgrinberg/dagster_home/history/\n", "module_name": "dagster.core.storage.runs"}, "schedule_storage_data": {"__class__": "ConfigurableClassData", "class_name": "SqliteScheduleStorage", "config_yaml": "base_dir: /Users/yuvalgrinberg/dagster_home/schedules\n", "module_name": "dagster.core.storage.schedules"}, "scheduler_data": {"__class__": "ConfigurableClassData", "class_name": "DagsterDaemonScheduler", "config_yaml": "{}\n", "module_name": "dagster.core.scheduler"}, "secrets_loader_data": null, "settings": {}, "storage_data": {"__class__": "ConfigurableClassData", "class_name": "DagsterSqliteStorage", "config_yaml": "base_dir: /Users/yuvalgrinberg/dagster_home\n", "module_name": "dagster.core.storage.sqlite_storage"}} --location-name dev.py -f /Users/yuvalgrinberg/PycharmProjects/dagster-landing-service/dagster_landing_service/repository/dev.py -d /Users/yuvalgrinberg/PycharmProjects/dagster-landing-service"

Stack Trace:
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_core/host_representation/grpc_server_registry.py", line 205, in _get_grpc_endpoint
    server_process = GrpcServerProcess(
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_grpc/server.py", line 1222, in __init__
    server_process = open_server_process(
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_grpc/server.py", line 1129, in open_server_process
    wait_for_grpc_server(server_process, client, subprocess_args, timeout=startup_timeout)
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_grpc/server.py", line 1063, in wait_for_grpc_server
    raise Exception(

2023-08-23 17:35:35 +0300 - dagster.daemon.SchedulerDaemon - WARNING - Could not load location dev.py to check for schedules due to the following error: Exception: gRPC server exited with return code 1 while starting up with the command: "/Users/yuvalgrinberg/envs/dagster-landing-service/bin/python -m dagster api grpc --lazy-load-user-code --socket /var/folders/k5/dlhcjkks0y7d0xg5pth56dth0000gn/T/tmp5r7qtgie --heartbeat --heartbeat-timeout 20 --fixed-server-id bd4da9d1-80e4-4049-b743-dd3f888ed855 --log-level warning --inject-env-vars-from-instance --instance-ref {"__class__": "InstanceRef", "compute_logs_data": {"__class__": "ConfigurableClassData", "class_name": "LocalComputeLogManager", "config_yaml": "base_dir: /Users/yuvalgrinberg/dagster_home/storage\n", "module_name": "dagster.core.storage.local_compute_log_manager"}, "custom_instance_class_data": null, "event_storage_data": {"__class__": "ConfigurableClassData", "class_name": "SqliteEventLogStorage", "config_yaml": "base_dir: /Users/yuvalgrinberg/dagster_home/history/runs/\n", "module_name": "dagster.core.storage.event_log"}, "local_artifact_storage_data": {"__class__": "ConfigurableClassData", "class_name": "LocalArtifactStorage", "config_yaml": "base_dir: /Users/yuvalgrinberg/dagster_home\n", "module_name": "dagster.core.storage.root"}, "run_coordinator_data": {"__class__": "ConfigurableClassData", "class_name": "DefaultRunCoordinator", "config_yaml": "{}\n", "module_name": "dagster.core.run_coordinator"}, "run_launcher_data": {"__class__": "ConfigurableClassData", "class_name": "DefaultRunLauncher", "config_yaml": "{}\n", "module_name": "dagster"}, "run_storage_data": {"__class__": "ConfigurableClassData", "class_name": "SqliteRunStorage", "config_yaml": "base_dir: /Users/yuvalgrinberg/dagster_home/history/\n", "module_name": "dagster.core.storage.runs"}, "schedule_storage_data": {"__class__": "ConfigurableClassData", "class_name": "SqliteScheduleStorage", "config_yaml": "base_dir: /Users/yuvalgrinberg/dagster_home/schedules\n", "module_name": "dagster.core.storage.schedules"}, "scheduler_data": {"__class__": "ConfigurableClassData", "class_name": "DagsterDaemonScheduler", "config_yaml": "{}\n", "module_name": "dagster.core.scheduler"}, "secrets_loader_data": null, "settings": {}, "storage_data": {"__class__": "ConfigurableClassData", "class_name": "DagsterSqliteStorage", "config_yaml": "base_dir: /Users/yuvalgrinberg/dagster_home\n", "module_name": "dagster.core.storage.sqlite_storage"}} --location-name dev.py -f /Users/yuvalgrinberg/PycharmProjects/dagster-landing-service/dagster_landing_service/repository/dev.py -d /Users/yuvalgrinberg/PycharmProjects/dagster-landing-service"

Stack Trace:
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_core/host_representation/grpc_server_registry.py", line 205, in _get_grpc_endpoint
    server_process = GrpcServerProcess(
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_grpc/server.py", line 1222, in __init__
    server_process = open_server_process(
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_grpc/server.py", line 1129, in open_server_process
    wait_for_grpc_server(server_process, client, subprocess_args, timeout=startup_timeout)
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_grpc/server.py", line 1063, in wait_for_grpc_server
    raise Exception(
this is the log of your command inside demo.log
d
Right- i'm trying to see what happens if you run that same command locally, to try to get to the bottom of what's happening
y
I thought its fairly simple
you can’t run both -f and --grpc-port together
and also you can’t run -f without providing grpc-port or socket
i worked around it like this:
Copy code
dagster api grpc --port 3030 -f /Users/yuvalgrinberg/PycharmProjects/dagster-landing-service/dagster_landing_service/repository/dev.py &

dagster-webserver --port 3000 --grpc-port 3030 &

dagster daemon run --grpc-port 3030
d
The command that i see your daemon running is providing -f and --socket, which I would expect to work
y
problem with my work-around is that the daemon doesn’t load sensors code
so maybe its something indeed with the mac OS temporary socket
Copy code
--socket /var/folders/k5/dlhcjkks0y7d0xg5pth56dth0000gn/T/tmp5r7qtgie
d
I'm still confused about this output that you posted:
Copy code
(dagster-landing-service) yuvalgrinberg@Yuvals-MacBook-Pro dagster-landing-service % /Users/yuvalgrinberg/envs/dagster-landing-service/bin/python -m dagster api grpc --lazy-load-user-code --socket /var/folders/k5/dlhcjkks0y7d0xg5pth56dth0000gn/T/tmp5r7qtgie --heartbeat --heartbeat-timeout 20 --fixed-server-id bd4da9d1-80e4-4049-b743-dd3f888ed855 --log-level warning --location-name dev.py -f /Users/yuvalgrinberg/PycharmProjects/dagster-landing-service/dagster_landing_service/repository/dev.py -d /Users/yuvalgrinberg/PycharmProjects/dagster-landing-service


(dagster-landing-service) yuvalgrinberg@Yuvals-MacBook-Pro dagster-landing-service %
Any idea why it isn't logging anything there? It just returned right away without outputing anything?
y
was this added 1.2.6-1.4.7
d
No, nothing significant has changed here that i'm aware of between these two versions, which is another thing that makes this confusing
y
Copy code
ls /var/folders/k5/dlhcjkks0y7d0xg5pth56dth0000gn/T/tmp5r7qtgie
ls: /var/folders/k5/dlhcjkks0y7d0xg5pth56dth0000gn/T/tmp5r7qtgie: No such file or directory
d
this is working for me though:
Copy code
(dagster-3.10.11) dgibson@Daniels-MacBook-Pro dagster % dagster api grpc -f hi.py --socket /tmp/ijustwroteanythinghereletsseeifitworks 
2023-08-23 10:50:59 -0400 - dagster.code_server - INFO - Started Dagster code server for file hi.py in process 8014
(note "ijustwroteanythinghereletsseeifitworks")
y
hmm and dagster daemon?
d
dagster-daemon run
doesn't let you specify a socket like that
y
aha correct
d
so just to confirm - that command I asked you to run earlier is neither logging anything nor raising an exception?
it just returns immediately with no output?
y
Copy code
(dagster-landing-service) yuvalgrinberg@Yuvals-MacBook-Pro dagster-landing-service % dagster api grpc --socket /var/folders/k5/dlhcjkks0y7d0xg5pth56dth0000gn/T/tmp5r7qtgie -f /Users/yuvalgrinberg/PycharmProjects/dagster-landing-service/dagster_landing_service/repository/dev.py
2023-08-23 17:52:48 +0300 - dagster.code_server - INFO - Started Dagster code server for file /Users/yuvalgrinberg/PycharmProjects/dagster-landing-service/dagster_landing_service/repository/dev.py in process 9491
so without all the mess it works like that
d
Ahhh right, i'm sorry, that was the --log-level warning
OK, so the mystery is why it is working when you run it locally, but not when the daemon runs a very similar command
y
yeah
Copy code
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/__main__.py", line 3, in <module>
    main()
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_cli/__init__.py", line 48, in main
    cli(auto_envvar_prefix=ENV_PREFIX)  # pylint:disable=E1123
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_cli/api.py", line 688, in grpc_command
    server = DagsterGrpcServer(
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_grpc/server.py", line 947, in __init__
    check.invariant(
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_check/__init__.py", line 1589, in invariant
    raise CheckError(f"Invariant failed. Description: {desc}")
dagster._check.CheckError: Invariant failed. Description: You must pass one and only one of `port` or `socket`.
/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_core/workspace/context.py:610: UserWarning: Error loading repository location dev.py:Exception: gRPC server exited with return code 1 while starting up with the command: "/Users/yuvalgrinberg/envs/dagster-landing-service/bin/python -m dagster api grpc --lazy-load-user-code --socket /var/folders/k5/dlhcjkks0y7d0xg5pth56dth0000gn/T/tmp5r7qtgie --heartbeat --heartbeat-timeout 20 --fixed-server-id bd4da9d1-80e4-4049-b743-dd3f888ed855 --log-level warning --inject-env-vars-from-instance --instance-ref {"__class__": "InstanceRef", "compute_logs_data": {"__class__": "ConfigurableClassData", "class_name": "LocalComputeLogManager", "config_yaml": "base_dir: /Users/yuvalgrinberg/dagster_home/storage\n", "module_name": "dagster.core.storage.local_compute_log_manager"}, "custom_instance_class_data": null, "event_storage_data": {"__class__": "ConfigurableClassData", "class_name": "SqliteEventLogStorage", "config_yaml": "base_dir: /Users/yuvalgrinberg/dagster_home/history/runs/\n", "module_name": "dagster.core.storage.event_log"}, "local_artifact_storage_data": {"__class__": "ConfigurableClassData", "class_name": "LocalArtifactStorage", "config_yaml": "base_dir: /Users/yuvalgrinberg/dagster_home\n", "module_name": "dagster.core.storage.root"}, "run_coordinator_data": {"__class__": "ConfigurableClassData", "class_name": "DefaultRunCoordinator", "config_yaml": "{}\n", "module_name": "dagster.core.run_coordinator"}, "run_launcher_data": {"__class__": "ConfigurableClassData", "class_name": "DefaultRunLauncher", "config_yaml": "{}\n", "module_name": "dagster"}, "run_storage_data": {"__class__": "ConfigurableClassData", "class_name": "SqliteRunStorage", "config_yaml": "base_dir: /Users/yuvalgrinberg/dagster_home/history/\n", "module_name": "dagster.core.storage.runs"}, "schedule_storage_data": {"__class__": "ConfigurableClassData", "class_name": "SqliteScheduleStorage", "config_yaml": "base_dir: /Users/yuvalgrinberg/dagster_home/schedules\n", "module_name": "dagster.core.storage.schedules"}, "scheduler_data": {"__class__": "ConfigurableClassData", "class_name": "DagsterDaemonScheduler", "config_yaml": "{}\n", "module_name": "dagster.core.scheduler"}, "secrets_loader_data": null, "settings": {}, "storage_data": {"__class__": "ConfigurableClassData", "class_name": "DagsterSqliteStorage", "config_yaml": "base_dir: /Users/yuvalgrinberg/dagster_home\n", "module_name": "dagster.core.storage.sqlite_storage"}} --location-name dev.py -f /Users/yuvalgrinberg/PycharmProjects/dagster-landing-service/dagster_landing_service/repository/dev.py -d /Users/yuvalgrinberg/PycharmProjects/dagster-landing-service"

Stack Trace:
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_core/host_representation/grpc_server_registry.py", line 205, in _get_grpc_endpoint
    server_process = GrpcServerProcess(
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_grpc/server.py", line 1222, in __init__
    server_process = open_server_process(
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_grpc/server.py", line 1129, in open_server_process
    wait_for_grpc_server(server_process, client, subprocess_args, timeout=startup_timeout)
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_grpc/server.py", line 1063, in wait_for_grpc_server
    raise Exception(
comes back to this
d
what if you unset DAGSTER_HOME before running it?
(before running 
dagster dev
that is - the daemon might not like that)
y
daemon didnt like it
d
how about dagster dev?
y
Copy code
2023-08-23 17:58:42 +0300 - dagster - INFO - Launching Dagster services...
Usage: python -m dagster._daemon run [OPTIONS]
Try 'python -m dagster._daemon run --help' for help.

Error: Invalid set of CLI arguments for loading repository/job. See --help for details.
same for webserver
Copy code
DAGSTER_STORAGE=~"/dagster_home" DAGSTER_HOME="" dagster dev -f /Users/yuvalgrinberg/PycharmProjects/dagster-landing-service/dagster_landing_service/repository/dev.py
🤷
I don’t want to waste your time, this happens only when deploying locally and we were able to run the GRPC and DAGIT without sensors locally at the moment.
If you guys can try to check that in your next sprint i will be willing to help you reproduce
d
or how about
Copy code
unset DAGSTER_HOME
dagster dev -f /Users/yuvalgrinberg/PycharmProjects/dagster-landing-service/dagster_landing_service/repository/dev.py
you're not wasting my time, particularly if others might be hitting this problem - but you're also not obligated at all to keep debugging if you don't need or want to
I haven't seen any other reports of this issue though - very strange
y
Copy code
2023-08-23 18:03:26 +0300 - dagster - INFO - To persist information across sessions, set the environment variable DAGSTER_HOME to a directory to use.
Traceback (most recent call last):
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/bin/dagster", line 8, in <module>
    sys.exit(main())
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_cli/__init__.py", line 48, in main
    cli(auto_envvar_prefix=ENV_PREFIX)  # pylint:disable=E1123
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_cli/dev.py", line 114, in dev_command
    with get_possibly_temporary_instance_for_cli("dagster dev", logger=logger) as instance:
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_cli/utils.py", line 83, in get_possibly_temporary_instance_for_cli
    with _get_temporary_instance(cli_command, logger) as instance:
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_cli/utils.py", line 42, in _get_temporary_instance
    with DagsterInstance.from_ref(
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_core/instance/__init__.py", line 583, in from_ref
    unified_storage = instance_ref.storage
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_core/instance/ref.py", line 496, in storage
    return self.storage_data.rehydrate(as_type=DagsterStorage) if self.storage_data else None
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_serdes/config_class.py", line 122, in rehydrate
    return klass.from_config_value(self, check.not_none(result.value))
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_core/storage/legacy_storage.py", line 118, in from_config_value
    run_storage = ConfigurableClassData(
  File "/Users/yuvalgrinberg/envs/dagster-landing-service/lib/python3.10/site-packages/dagster/_serdes/config_class.py", line 117, in rehydrate
    raise DagsterInvalidConfigError(
dagster._core.errors.DagsterInvalidConfigError: Errors whilst loading configuration for {'base_dir': <dagster._config.source.StringSourceType object at 0x110234d90>}.
    Error 1: Post processing at path root:base_dir of original value {'env': 'DAGSTER_STORAGE'} failed:
dagster._config.errors.PostProcessingError: You have attempted to fetch the environment variable "DAGSTER_STORAGE" which is not set. In order for this execution to succeed it must be set in this environment.
d
ah sorry can you try running it in a folder without your dagster.yaml
y
Copy code
(dagster-landing-service) yuvalgrinberg@Yuvals-MacBook-Pro dagster-landing-service % unset DAGSTER_HOME
dagster dev -f /Users/yuvalgrinberg/PycharmProjects/dagster-landing-service/dagster_landing_service/repository/dev.py
2023-08-23 18:07:06 +0300 - dagster - INFO - Loaded environment variables from .env file: DAGSTER_GRPC_PORT,DAGIT_PORT,DAGSTER_MODULE_NAME,SSH_PORT,IMAGE_NAME_ROOT,DEPLOYMENT_NAME,IMAGE_OWNER,REGISTRY_PREFIX,REGISTRY_PREFIX_GCR,GKE_NODE_POOL_SIZE,GKE_NODE_POOL_RUNNER,CPU,MEMORY
2023-08-23 18:07:06 +0300 - dagster - INFO - Using temporary directory /Users/yuvalgrinberg/PycharmProjects/dagster-landing-service/tmpr0gx2dmq for storage. This will be removed when dagster dev exits.
2023-08-23 18:07:06 +0300 - dagster - INFO - To persist information across sessions, set the environment variable DAGSTER_HOME to a directory to use.
2023-08-23 18:07:07 +0300 - dagster - INFO - Launching Dagster services...
Usage: python -m dagster._daemon run [OPTIONS]
Try 'python -m dagster._daemon run --help' for help.

Error: Invalid set of CLI arguments for loading repository/job. See --help for details.
2023-08-23 18:07:09 +0300 - dagster-webserver - INFO - Loaded environment variables from .env file: DAGSTER_GRPC_PORT,DAGIT_PORT,DAGSTER_MODULE_NAME,SSH_PORT,IMAGE_NAME_ROOT,DEPLOYMENT_NAME,IMAGE_OWNER,REGISTRY_PREFIX,REGISTRY_PREFIX_GCR,GKE_NODE_POOL_SIZE,GKE_NODE_POOL_RUNNER,CPU,MEMORY
Usage: python -m dagster_webserver [OPTIONS]
Try 'python -m dagster_webserver --help' for help.

Error: Invalid set of CLI arguments for loading repository/job. See --help for details.
2023-08-23 18:07:12 +0300 - dagster - INFO - Shutting down Dagster services...
2023-08-23 18:07:12 +0300 - dagster - INFO - Dagster services shut down.
d
lastly if there's any chance you could past the contents of your dagster.yaml file that might help us reproduce - i'm wondering if something about the formatting is causing that '--instance-ref' argument to mess things up
Oh hmmm, this line jumps out:
Copy code
Loaded environment variables from .env file: DAGSTER_GRPC_PORT,DAGIT_PORT,DAGSTER_MODULE_NAME,SSH_PORT,IMAGE_NAME_ROOT,DEPLOYMENT_NAME,IMAGE_OWNER,REGISTRY_PREFIX,REGISTRY_PREFIX_GCR,GKE_NODE_POOL_SIZE,GKE_NODE_POOL_RUNNER,CPU,MEMORY
Try taking DAGSTER_GRPC_PORT out of there
Pretty sure that's it - we'll get that fixed
y
ohohoh
you da man
checking the daemon now
seem to be working
d
nice - that was a tricky one
y
so many ways to pass those params
thank you so very much mate, i’ll close the issue
d
all good - thanks for the help debugging
y
perhaps we should add some remarks to the CLI docs about how to pass those params
d
I think what we'll do is unset that environment variable in the subprocess
y
yeah thats also a great idea
i thought it is being used for the K8s deployment alone
@Eldan Hamdani
d
Here's a fix: https://github.com/dagster-io/dagster/pull/16037/files - thanks again for the report
114 Views
Open in Slack
PreviousNext
Powered by