I m currently running a fanned out graph aseet job that spli

Question

I m currently running a fanned out graph aseet job that splits into 60 different pieces using the QueuedRunCoordinator We have the dagster daemon dagster yaml as ```run coordinator module dagster core run coordinator class QueuedRunCoordinator config max concurrent runs 25 tag concurrency limits key database value sql limit 8 key ml value score limit 10``` However only 4 ops are generated at a time We previously had the concurrency limit for sql set to 4 Is there a way to confirm that those updated limits are currently loaded in the daemon also how do those keys and values become associated with a run I m using a sql server resource in my fanned out op does it do some sort of name checking with the resource

daniel · Answer

Hi Danny those tags will strictly refer to tags on the job not on individual ops or assets We don t currently have a way to supply global concurrency limits at the op level but its come up a lot recently and it s something we re starting to take a closer look at If you want to apply op level concurrency limits within a single run you can configure that in code on the executor for your job following the example here <https docs dagster io concepts ops jobs graphs job execution op concurrency limits>

Danny Steffy · Answer

I see so my issue is unrelated to the concurrency limits

daniel · Answer

if you want more than 4 at once I think that might be changing the max concurrent parameter of the executor

daniel · Answer

in that same doc link I believe the default is based on the number of CPUs available

Danny Steffy · Answer

hm I see Is there any way to set the executor for a graph defined asset to be materialized via the UI

daniel · Answer

Yeah you can also specify that same config in the launchpad UI

daniel · Answer

```execution config multiprocess max concurrent 20```

daniel · Answer

If you Shift click on Materialize it will open the launchpad UI

Danny Steffy · Answer

ah I see Can I specify to use the docker executor instead of the multiprocess

daniel · Answer

you specify which executor to use in code but you can configure that executor in the launchpad

Danny Steffy · Answer

I didn t think I could define the executor for the assets or graphs in code just for jobs

daniel · Answer

oh i see the definitions or repository object can also specify a default executor which would be used there

daniel · Answer

there s an executor arg <https docs dagster io apidocs definitions dagster Definitions>

Danny Steffy · Answer

ah thank you

Danny Steffy · Answer

do I need the same dagster yaml config that I have for the daemon to be set in the dagit dagster yaml as well Specifically the run launcher and all the configuration settings for the DockerRunLauncher Getting an error about an environment variable not being set when I try to materialize the asset in the UI with the new docker executor being set in the Definition

daniel · Answer

Yeah you want a single shared dagster yaml

daniel · Answer

or rather your dagster yaml should be the same everywhere

Danny Steffy · Answer

hmmm getting this error still ```dagster core errors DagsterInvalidConfigError Errors whilst loading configuration for image Field lt dagster config source StringSourceType object at 0x7f7999fa8910 gt default= is required=False network Field lt dagster config source StringSourceType object at 0x7f7999fa8910 gt default= is required=False registry Field lt dagster config field utils Shape object at 0x7f7993b66d70 gt default= is required=False env vars Field lt dagster config config type Array object at 0x7f7993b9b130 gt default= is required=False container kwargs Field lt dagster config field utils Permissive object at 0x7f7999fa8580 gt default= is required=False networks Field lt dagster config config type Array object at 0x7f7993b986d0 gt default= is required=False Error 1 Post processing at path root registry password of original value env DOCKER REG PASSWORD failed dagster config errors PostProcessingError You have attempted to fetch the environment variable DOCKER REG PASSWORD which is not set In order for this execution to succeed it must be set in this environment ```

Danny Steffy · Answer

I ve made sure the dagit dagster code and dagster daemon containers all have that env var and it s also being set in the run launcher config that all three containers are now using in the dagster yaml file

daniel · Answer

Where are you seeing that error exactly

Danny Steffy · Answer

image png

daniel · Answer

Looks like that s in the container it tried to spin up to run the op step you re using the DockerRunLauncher

Danny Steffy · Answer

yep

Danny Steffy · Answer

oh wait interesting

daniel · Answer

Typically the env vars you put on the run launcher would also be passed through to the step containers

Danny Steffy · Answer

when I go to the configuration tab it doesn t have that env var set in the run launcher config

Danny Steffy · Answer

but it s in there on my dagster yaml maybe I need to restart my dagit container going to give that a try

daniel · Answer

Depending on the way you have it set up it may need to be rebuilt not just restarted

Danny Steffy · Answer

ah So if we copy the dagster yaml file over to the image and then run the dagit command in the Dockerfile would mounting the file location via docker compose actually do anything or do we need to rebuild the image each time

daniel · Answer

If they re both in the DAGSTER HOME folder I m not actually certain how docker compose handles that my guess would be the docker compose would break the tie but im not certain

Danny Steffy · Answer

hm so I m still getting this error I connected via bash to the different containers and they all have that environment variable mentioned above I m not sure what to look at next for this

daniel · Answer

Can you paste the full stack trace Including the red part at the bottom there that got cut off

daniel · Answer

This is a recent version of dagster I assume

Danny Steffy · Answer

```dagster core errors DagsterInvalidConfigError Errors whilst loading configuration for image Field lt dagster config source StringSourceType object at 0x7f9abe7148e0 gt default= is required=False network Field lt dagster config source StringSourceType object at 0x7f9abe7148e0 gt default= is required=False registry Field lt dagster config field utils Shape object at 0x7f9ab82ed990 gt default= is required=False env vars Field lt dagster config config type Array object at 0x7f9ab82ef5b0 gt default= is required=False container kwargs Field lt dagster config field utils Permissive object at 0x7f9abe714550 gt default= is required=False networks Field lt dagster config config type Array object at 0x7f9ab82ee290 gt default= is required=False Error 1 Post processing at path root registry password of original value env DOCKER REG PASSWORD failed dagster config errors PostProcessingError You have attempted to fetch the environment variable DOCKER REG PASSWORD which is not set In order for this execution to succeed it must be set in this environment Stack Trace File usr local lib python3 10 site packages dagster config post process py line 79 in post process new value = context config type post process config value File usr local lib python3 10 site packages dagster config source py line 42 in post process return str ensure env variable cfg File usr local lib python3 10 site packages dagster config source py line 16 in ensure env variable raise PostProcessingError File usr local lib python3 10 site packages dagster core execution api py line 992 in pipeline execution iterator for event in pipeline context executor execute pipeline context execution plan File usr local lib python3 10 site packages dagster core executor step delegating step delegating executor py line 305 in execute list File usr local lib python3 10 site packages dagster docker docker executor py line 217 in launch step container context = self get docker container context step handler context File usr local lib python3 10 site packages dagster docker docker executor py line 145 in get docker container context run launcher = step handler context instance run launcher File usr local lib python3 10 site packages dagster core instance init py line 660 in run launcher launcher = cast InstanceRef self ref run launcher File usr local lib python3 10 site packages dagster core instance ref py line 491 in run launcher return self run launcher data rehydrate if self run launcher data else None File usr local lib python3 10 site packages dagster serdes config class py line 96 in rehydrate raise DagsterInvalidConfigError ```

Danny Steffy · Answer

I think we have it pinned but I m not sure what version we re currently running

daniel · Answer

Got it so this error is implying pretty strongly that DOCKER REG PASSWORD is not set in the container that it spun up to run the job in Is it possible to paste the contents of your dagster yaml

Danny Steffy · Answer

```storage postgres postgres db username env POSTGRES USER password env POSTGRES PASSWORD hostname env POSTGRES HOST db name env POSTGRES DB port 5432 local artifact storage module dagster core storage root class LocalArtifactStorage config base dir opt dagster local run launcher module dagster docker class DockerRunLauncher config env vars POSTGRES USER POSTGRES PASSWORD POSTGRES DB POSTGRES HOST DAGSTER CURRENT IMAGE POSTGRES HOST POSTGRES USER POSTGRES PASSWORD POSTGRES DB FLUSH USER FLUSH PASS FLUSH DISPATCH SERVER FLUSH DISPATCH DB AIRBYTE PASSWORD AIRBYTE HOST DAGSTER DEPLOYMENT DOCKER REG PASSWORD networks dagster flush registry url xxx username xxxx password env DOCKER REG PASSWORD container kwargs volumes ssd dagster 01 data opt dagster local ssd dagster 01 code opt dagster app dagster io manager tmp io manager storage var run docker sock var run docker sock run coordinator module dagster core run coordinator class QueuedRunCoordinator config max concurrent runs 25 tag concurrency limits key database value sql limit 8 key ml value score limit 10 telemetry enabled false```

daniel · Answer

Is it possible to docker inspect and paste the command that it used for that container the one that it spun up for the run

daniel · Answer

er wait maybe that doesn t print the command one sec

daniel · Answer

er yeah it should be in there

Danny Steffy · Answer

ah I just checked what I believe is the container that got spun up for the run

Danny Steffy · Answer

and it doesn t have that env var

daniel · Answer

aha

Danny Steffy · Answer

but I m not sure why it wouldn t

Danny Steffy · Answer

isnt that being set by the dagster yaml

daniel · Answer

It should be yeah when you inspect it can you paste the key under Cmd

daniel · Answer

My suspicion is still that the dagster yaml is out of date somewhere

daniel · Answer

specifically my guess is that it s out of date on the daemon container

daniel · Answer

since that is where the run is launched from

Danny Steffy · Answer

``` Cmd dagster api execute run class ExecuteRunArgs instance ref class InstanceRef compute logs data class ConfigurableClassData class name LocalComputeLogManager config yaml base dir opt dagster dagster home storage n module name dagster core storage local compute log manager custom instance class data null event storage data class ConfigurableClassData class name PostgresEventLogStorage config yaml postgres db n db name n env POSTGRES DB n hostname n env POSTGRES HOST n password n env POSTGRES PASSWORD n port 5432 n username n env POSTGRES USER n module name dagster postgres local artifact storage data class ConfigurableClassData class name LocalArtifactStorage config yaml base dir opt dagster local n module name dagster core storage root run coordinator data class ConfigurableClassData class name QueuedRunCoordinator config yaml max concurrent runs 25 ntag concurrency limits n key database n limit 8 n value sql n key ml n limit 10 n value score n module name dagster core run coordinator run launcher data class ConfigurableClassData class name DockerRunLauncher config yaml container kwargs n volumes n ssd dagster 01 data opt dagster local n ssd dagster 01 code opt dagster app n var run docker sock var run docker sock nenv vars n POSTGRES USER n POSTGRES PASSWORD n POSTGRES DB n POSTGRES HOST n DAGSTER CURRENT IMAGE n POSTGRES HOST n POSTGRES USER n POSTGRES PASSWORD n POSTGRES DB n FLUSH USER n FLUSH PASS n FLUSH DISPATCH SERVER n FLUSH DISPATCH DB n AIRBYTE PASSWORD n AIRBYTE HOST n DAGSTER DEPLOYMENT nnetworks n dagster n flush nregistry n password n env DOCKER REG PASSWORD n url n username flcontainers n module name dagster docker run storage data class ConfigurableClassData class name PostgresRunStorage config yaml postgres db n db name n env POSTGRES DB n hostname n env POSTGRES HOST n password n env POSTGRES PASSWORD n port 5432 n username n env POSTGRES USER n module name dagster postgres schedule storage data class ConfigurableClassData class name PostgresScheduleStorage config yaml postgres db n db name n env POSTGRES DB n hostname n env POSTGRES HOST n password n env POSTGRES PASSWORD n port 5432 n username n env POSTGRES USER n module name dagster postgres scheduler data class ConfigurableClassData class name DagsterDaemonScheduler config yaml n module name dagster core scheduler secrets loader data null settings telemetry enabled false storage data class ConfigurableClassData class name DagsterPostgresStorage config yaml postgres db n db name n env POSTGRES DB n hostname n env POSTGRES HOST n password n env POSTGRES PASSWORD n port 5432 n username n env POSTGRES USER n module name dagster postgres pipeline origin class PipelinePythonOrigin pipeline name ASSET JOB repository origin class RepositoryPythonOrigin code pointer class FileCodePointer fn name defs python file repo py working directory opt dagster app container context container image entry point dagster executable path usr local bin python pipeline run id 40a3c5f8 074f 46e6 b2e6 7d6461b15a91 set exit code on failure null ```

Danny Steffy · Answer

this was working with the daemon before it s when I changed the configuration on dagit that this error starting occurring

daniel · Answer

That command is a big mess but the relevant bit is buried in there `env vars n POSTGRES USER n POSTGRES PASSWORD n POSTGRES DB n POSTGRES HOST n DAGSTER CURRENT IMAGE n POSTGRES HOST n POSTGRES USER n POSTGRES PASSWORD n POSTGRES DB n FLUSH USER n FLUSH PASS n FLUSH DISPATCH SERVER n FLUSH DISPATCH DB n AIRBYTE PASSWORD n AIRBYTE HOST n DAGSTER DEPLOYMENT n` no DOCKER REG PASSWORD that s the daemon serializing its dagster yaml and passing it into the command

Danny Steffy · Answer

is there a way to restart the daemon from inside the container

daniel · Answer

Hmm not that i m aware of possibly a way to do it in docker but i m not sure

Danny Steffy · Answer

because I can see that the dagster yaml has the DOCKER REG PASSWORD inside the daemon container

daniel · Answer

ah is that because its mounted from an external file

Danny Steffy · Answer

it definitely is mounted from a file on the host vm

daniel · Answer

Got it that would require a restart when that file changes yeah

daniel · Answer

which I would typically do by stopping and restarting the container

Danny Steffy · Answer

hm I think I must have had a stuck container that was spun up I killed the dagster code code container that had been spun up for the run restarted the daemon and then reexecuted the run It finished successfully now

Danny Steffy · Answer

thanks for the help