@rex even though the old API was parsing the manifest files on demand it was approx. 33 seconds faster. The only major difference is that now I also extract the affected row numbers - they were not available in the old API.

i wonder if this is because dbt is not using partial parsing since the target path is now dynamic. Do you have links to Dagster runs, before and after deploying the experimental APIs?

I saw these logs before. Let me search the logs and then share the links

Yeah, this looks related to project parsing. Because we’re dynamically setting the target path, I believe some optimizations are being skipped. I can see if there’s a way to continue to take advantage of this. We could probably do something like copy the

partial_parse.msgpack

file to the dynamic target path before invoking the CLI. This is like the equivalent of

Reusing Objects

in the dbt programatic invocation case. But since this API isn’t stable yet, there’s no first-class way to do this right now.

so once this gets merged it should be sped up automatically?

You would get speed up in instances where partial parsing is possible.

To truly confirm the root of the issue with the performance degradation you’re seeing, we would need to see a pyspy profile

Would I somehow have the possibility to get this from the dagit UI? I guess not but this might be a feature request. If I connect directly to the docker image and start a container can I easily run a pipeline with the credentials from dagster cloud?

Let’s wait for the partial parsing change to land before going down this rabbithole. The partial parsing is probably the cause of your problems, since I don’t see how there could be any other slow down from calling the dbt CLI.

I am now seeing: /partial_parse.msgpack` to take advantage of partial parsing. and an improved time. I still have to run for several more days to tell exactly if it is working well and if it is now overall faster than before or if more needs to be done. But this certainly is a step in the right direction