Hi, I have been developing an internal data pipeli...
# ask-community
d
Hi, I have been developing an internal data pipeline using daster. Ths project includes several internal projects maintained by other teams. To make life easy in deployment and maintenance, I separated dagit UI and actual dagster code in separate docker containers. Dagit UI loads a list of jobs through ``grpc_server`` defined in ``workspace.yaml` and the code is loaded in a separate container using ``dagster api grpc`` One issue I faced is that ``location_name`` field in ``workspace.yaml`` does not seem to behavior as what I expected. Based on dagster documentation, the ``location_name`` field is used to distinguish multiple code locations in case that there are. For a single code locataion, when I specified the job name as ``location_name``, it works fine. Then, I changed the location_name to something else other than ``job_name``, it failed to load code. Is this expected? Another issue is that when multiple jobs from a Defintions object need to be loaded , what do I need to specify in ``location_name``? Documentation says that users can specify a custom location name. I tried to put "test_job" or some thing sles. But this does not seem to be working. A workaround I found is to drop ``location_name`` from the ``grpc_server`` section of the ``workspace.yaml``. This loads all jobs successfully. But I see a warning in the terminal which bothers me. Does anyone have an insight on these issues?
d
Hi Dong - this does not sounds expected. What error are you seeing when it fails to load your code?
d
I am seeing ``gRPC:UNAVAILABLE `` in the first case when it fails to load. For the second case, ``gRPC Error code: UNAVAILABLE`` is thrown out when I drop ``location_name``. But all jobs are loaded successfully in both ``dagit`` and ``dagster`` containers. I confirmed by running ``dagster job list`` as well as in the dagit UI.
d
Can you post an example workspace.yaml that reproduces the first issue? And maybe the full stack trace of the error in the first case?
d
@daniel Sure, I will get those for you
d
i wouldn't expect location_name to matter at all for whether or not the code is able to load - I suspect something else may be going on here
d
That is my suspicion too
d
if you reload the code location in dagit, does it load?
i'm wondering if the issue might be that the code server is taking a bit to start up, and dagit tries to access it before its loaded
d
Before, dagit complained grpc unavailable. But I cleaned up all dangling docker images and rebuild images.
Yes, dagit loads jobs now.
So, this is simply a matter of timing issue; dagit trying to access the code server before it is ready.
d
that sounds right - believe you can add a depends_on to docker-compose to make it wait
the daemon should automatically reload it every minute or so
d
That makes sense to me. Thanks Daniel
condagster 1