I m having some difficulties using the `execute job` to test

Question

I m having some difficulties using the `execute job` to test my jobs I m using databricks step launcher and it requires a reconstructable pipeline so I m unable to use `job execute in process` From the docs I m doing ```instance = DagsterInstance get execute job reconstructable my job run config=config instance=instance ``` But I get this error ```dagster check CheckError Failure condition Unexpected return value from child process lt class collections ChildProcessStartEvent gt Stack Trace File opt conda envs dagster env lib python3 9 site packages dagster core execution api py line 991 in pipeline execution iterator for event in pipeline context executor execute pipeline context execution plan File opt conda envs dagster env lib python3 9 site packages dagster core executor multiprocess py line 240 in execute event or none = next step iter File opt conda envs dagster env lib python3 9 site packages dagster core executor multiprocess py line 364 in execute step out of process check failed Unexpected return value from child process format type ret File opt conda envs dagster env lib python3 9 site packages dagster check init py line 1687 in failed raise CheckError f Failure condition desc ``` Am I missing something

Dinis Rodrigues · Answer

I reported this in a github issue and I also started a thread a few months ago about this But I lost the thread due to the 90 day message retention I only saved this message from that time with < daniel> But as I stated using databricks step launcher I cannot use execute in process

yuhan · Answer

hey sorry for the late response we re looking into this issue if im recalling it correctly does the issue occur only when you re using it with `op retry policy`

daniel · Answer

Hi Dinis in the short term is it possible to move the code with side effects to a different file I think your problem is coming from the fact that your code within the ```if name == main ``` block is also executing within the step launcher which is probably not what you want

daniel · Answer

putting your definitions in one file and your testing code in another that imports the job from the definitions file is one quick way to get around that

Dinis Rodrigues · Answer

< yuhan> If I remove the retry policy the job proceeds although giving the same error < daniel> I m trying to implement your suggestion but I m getting the same result I built an example test that matches what I m doing If you could take a look pray

daniel · Answer

Does it work if you take out main py altogether

Dinis Rodrigues · Answer

You mean running myjob test py directly With that I get a multiprocessing error and then the same error again

daniel · Answer

I meant removing the main py file entirely my suspicion is that something about the fact that code is running in an if main block could be causing the problem

Dinis Rodrigues · Answer

Okok yes by doing that I get the above image error

daniel · Answer

What if you run the job using the dagster job launch command Does that work better

Dinis Rodrigues · Answer

Using the CLI the error doesn t seem to happen

Dinis Rodrigues · Answer

But with the cli it seems this would require a major refactoring right From what I understand I would need to create new graphs since I have more than one job running in the test pipeline and then I would need to specify the run config directly in the job decorator

Dinis Rodrigues · Answer

By doing this I only have 2 logs Doesn t seem to fully run I would expect some logs from databricks launcher

Dinis Rodrigues · Answer

Iv e managed to get the logs by doing `dagster job execute` instead of launch But I m still unable to get a workaround on inserting the run config directly in the decorator The reason being my run config depends on environment variables So when this goes to databricks it tries to read environment variables that don t yet exist on the cluster

Dinis Rodrigues · Answer

< daniel> < yuhan> Based on all this information and for future reference this is the solution I came up with for the test setup 1 Read a pre defined job config 2 Fill it based on environment variables 3 Dump it into a new generated job config yaml 4 Launch jobs using the dagster CLI with the generated config This is a crude sample of what we are going to develop but it s the baseline of the implementation In this way we entirely skip the python execution api and only use the dagster CLI On a side note the documentation about the CLI is pretty scattered I took a really long time until I found out I could use the config flag Thank you call me hand