https://dagster.io/ logo
Join Slack
Powered by
Hi all, We are getting an error when starting our ...
# deployment-ecs
r
Hi all, We are getting an error when starting our ECS jobs. 
dagster.check.CheckError: Invariant failed. Description: Must include either pipeline_run or pipeline_name and run_id
Can someone please point us in the the direction of where to look for this.
d
Hi Randy, do you have a full stack trace for this?
r
That is all we get
This is the stack trace that we see:
Copy code
/usr/local/lib/python3.9/site-packages/dagster_aws/ecs/launcher.py:97: ExperimentalWarning: "EcsRunLauncher" is an experimental class. It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at <https://docs.python.org/3/library/warnings.html#describing-warning-filters>.
return EcsRunLauncher(inst_data=inst_data, **config_value)
Traceback (most recent call last):
File "/usr/local/bin/dagster", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.9/site-packages/dagster/cli/__init__.py", line 50, in main
cli(auto_envvar_prefix=ENV_PREFIX) # pylint:disable=E1123
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/dagster/cli/api.py", line 61, in execute_run_command
return_code = _execute_run_command_body(
File "/usr/local/lib/python3.9/site-packages/dagster/cli/api.py", line 87, in _execute_run_command_body
instance.report_engine_event(
File "/usr/local/lib/python3.9/site-packages/dagster/core/instance/__init__.py", line 1356, in report_engine_event
check.invariant(
File "/usr/local/lib/python3.9/site-packages/dagster/check/__init__.py", line 176, in invariant
raise CheckError(f"Invariant failed. Description: {desc}")
dagster.check.CheckError: Invariant failed. Description: Must include either pipeline_run or pipeline_name and run_id
d
Got it, thanks - and what version of dagster is this?
And what are you using for storage for the dagster DB? We'll improve this not-so-great error message, but what appears to be happening here is that the task that the EcsRunLauncher spins up can't find the run that it's supposed to be executing in your DB
r
@daniel Thanks for that I think I might have an idea in this case. I need add an extra security group to the mix. Unfortunatelly we are in very distance timezones. So this conversation might have some lag. We use a postgres DB but I suspect that the container it spins doesn't get attached to the security group that it needs to access the DB
d
That's possible yeah - I'd be surprised that the error would manifest in this specific way if it couldn't connect to your database though. It's almost as though it's successfully connecting to some other db that doesn't contain your run. If you could share the full command that the task is running that might have some clues (it would start with dagster api execute_run and have a lot of JSON in it)
r
I sent you the full command that Dagster is getting in this task
Just a follow up for anyone else reading this. We did not have RDS setup at the time while we were doing our conversion. This error was because the task runner was not able to read the data it required from the database which was on sqlite in the scheduler container. Once we got the RDS running, it was fine agian.
condagster 1
2 Views
Open in Slack
PreviousNext
Powered by