Hi all we have managed to get our ECS Task running We have a

Question

Hi all we have managed to get our ECS Task running We have a slightly more complicated design than what it seems dagster is designed to do we have a process manager inside the container that goes and collects the secrets that we need and sets the names as env values as we need them We noticed that dagster passes in the commands that it wants to run as a CMD in the task We used this to our advantage and collected it and stored it in a file then collected our secrets and then ran the code that was passed in the CMD However we seem to be getting an error from dagster as follows ```Error Got unexpected extra arguments ExecuteRunArgs instance ref JSON BLOB ``` The code that we are running is exactly ```eval usr local bin python m dagster api execute run JSON BLOB ``` We use eval because it is being run from a bash script We are not sure what the extra arguments are If you need more information about this please ask but any help would be great

jordan · Answer

That error is actually coming from Click which is what the `dagster` CLI uses under the hood Does your Dockerfile also have a `CMD` or `ENTRYPOINT` specified See <https aws amazon com blogs opensource demystifying entrypoint cmd docker >

jordan · Answer

Also you might be interested in these changes available as of Dagster 0 13 10 <https docs dagster io deployment guides ecs secrets management>

Randy Coburn · Answer

We do have a ENTRYPOINT specified it uses a process manager as explained We use it to collect secrets as we collect many of them and change the names that are presented in secret manager The way we do this is to first check to see if there was a CMD passed in we don t use it as we place everything in the ENTRYPOINT of the docker file If there is one we write exactly what was given into a file collect our secrets and then execute the passed in CMD

Randy Coburn · Answer

We collect many secrets that have the same internal names `username password database` however we change the names to be `service a username` `service b username`

jordan · Answer

Can I see your `ENTRYPOINT` I still suspect it has something to do with how ECS is concatenating `ENTRYPOINT` and `CMD`

Randy Coburn · Answer

sure 1 min

Randy Coburn · Answer

In the dockerfile ``` CMD allows this to be overridden from run launchers or executors that want to run other commands against your repository ENTRYPOINT bin bash cmd run sh CMD ``` In the cmd run sh ``` bin bash if z $ echo n $ | base64 then When we have a cmd passed in we will use it We do this by checking if the array of passed in values is more than zero in length LAUNCH WITH CMD= launch with cmd yaml echo Got a CMD passed in CMD SCRIPT= cmd sh cat lt lt EOF gt $CMD SCRIPT bin bash echo Starting CMD eval $ EOF launch f $LAUNCH WITH CMD else Else we will just run the GRPC container launch f launch grpc code yaml fi```

Randy Coburn · Answer

In the ECS console I see only a CMD passed in I can also confirm that the container is running that code as expected

jordan · Answer

what s in the two yaml files

jordan · Answer

This answer seems to suggest whitespacing in your JSON blob could also be at play <https stackoverflow com a 50921236> So definitely try that as well But there s enough indirection happening within the ENTRYPOINT and the CMD that I have a feeling we might find we just have some subtle inconsistency with how the two are concatenated together

Randy Coburn · Answer

They describe the run for launch `launch with cmd yaml` ```processes secret processes name SERVICE A RDS CREDENTIALS command aws secret reader arguments secret env SERVICE A RDS SECRET HINT upper case prepend with SERVICE A RDS name SERVICE B CREDENTIALS command aws secret reader arguments secret env SERVICE B SECRET HINT upper case prepend with SERVICE B main processes name CMD command bin bash arguments cmd sh``` `launch grpc code yaml` ```processes secret processes name SERVICE A RDS CREDENTIALS command aws secret reader arguments secret env SERVICE A RDS SECRET HINT upper case prepend with SERVICE A RDS name SERVICE B CREDENTIALS command aws secret reader arguments secret env SERVICE B SECRET HINT upper case prepend with SERVICE B main processes name DAG REPO GRPC command usr local bin dagster arguments api grpc h 0 0 0 0 p 4000 f opt dagster app repositories datalake repositories py ```

Randy Coburn · Answer

Are you suggesting that the white space is in the JSON itself

jordan · Answer

It could be I still think it s more likely that the ENTRYPOINT and CMD are getting concatenated together incorrectly But it s at least worth ruling out that the JSON isn t malformed

Randy Coburn · Answer

```eval usr local bin python m dagster api execute run class ExecuteRunArgs instance ref class InstanceRef compute logs data class ConfigurableClassData class name LocalComputeLogManager config yaml base dir home service user dagster storage n module name dagster core storage local compute log manager custom instance class data null event storage data class ConfigurableClassData class name PostgresEventLogStorage config yaml postgres db n db name dagster n hostname n password xxx n port 5432 n username xxx n module name dagster postgres event log local artifact storage data class ConfigurableClassData class name LocalArtifactStorage config yaml base dir home service user dagster n module name dagster core storage root run coordinator data class ConfigurableClassData class name QueuedRunCoordinator config yaml n module name dagster core run coordinator run launcher data class ConfigurableClassData class name EcsRunLauncher config yaml container name dlakepipe pipeline code ntask definition arn aws ecs eu west 1 123 task definition dlakepipe pipeline code 1 n module name dagster aws ecs run storage data class ConfigurableClassData class name PostgresRunStorage config yaml postgres db n db name dagster n hostname n password xxx n port 5432 n username xxx n module name dagster postgres run storage schedule storage data class ConfigurableClassData class name PostgresScheduleStorage config yaml postgres db n db name dagster n hostname n password xxx n port 5432 n username xxx n module name dagster postgres schedule storage scheduler data class ConfigurableClassData class name DagsterDaemonScheduler config yaml n module name dagster core scheduler settings telemetry enabled false pipeline origin class PipelinePythonOrigin pipeline name my job repository origin class RepositoryPythonOrigin code pointer class FileCodePointer fn name deploy docker repository python file opt dagster app repo test py working directory opt dagster app container image null executable path usr local bin python pipeline run id 8b3de847 95ba 4b98 97b5 f35733e18260 ``` After the capture this is what we have I have removed the username and password stuff However the JSON is valid I tried to get eval to use it as to try stop any bash crazy stuff

jordan · Answer

Yeah at a glance that looks correct to me At this point I d recommend checking all the places your ENTRYPOINT and CMD can be set in the Dockerfile in the ECS Task Definition and in the ECS Task Container Overrides and stepping through the logic described here to see what the full command ECS is running is <https aws amazon com blogs opensource demystifying entrypoint cmd docker >

Randy Coburn · Answer

Can you point me to the line that ECS Runner sets the CMD I m pretty sure that we have the ENTRYPOINT and CMD right That snip comes from the container itself after it has captured the `$ ` which should be the CMD

jordan · Answer

There are 3 different places it can source the ENTRYPOINT From and 3 different places it can source the CMD from ENTRYPOINT in the Dockerfile in the ECS Task Definition s Container Definitions in the ECS Task s Overrides CMD in the Dockerfile in the ECS Task Definition s Container Definitions in the ECS Task s Overrides By default the ECS Run Launcher passes its command as an ECS Task s Override <https github com dagster io dagster blob c24f76950b8c80de3ed538e091fda9df7ee3e66c python modules libraries dagster aws dagster aws ecs launcher py L154 L157> If your ENTRYPOINT is being set in your Dockerfile it still might be getting overridden by something in your Task Definition or in your Task Or things might not be being concatenated the way you expect it to particularly because of the combination of arrays and strings being used here

Randy Coburn · Answer

Hi < jordan> We finally managed to figure this out We shelved it over the holiday season but picked it up again a few days ago It was basically multiple different shells parsing the text and stripping it of the quotes This then made it seem like arguments rather than a JSON blob to the dagster process I wonder if it would be useful for other to see how we do this as we execute code in the dagster task containers BEFORE we start the actual task We do this to collect secrets and setup the environment using a tool called Launch