The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Execution_of_pipeline____import_google_news_pipeline____failed_

Hi all, I’m setting up dagster to manage a data pipeline to write to google cloud storage and google bigquery. Locally it runs but when I deploy it to a simple server I get the following error, any clue what could be causing this?

Hmm - it looks like there is an error happening during a different error situation

Ah! you hit a bug we have since fixed (but not pushed a release for) from this line <https://github.com/dagster-io/dagster/blob/0.6.6/python_modules/libraries/dagster-ssh/dagster_ssh/solids.py#L84>

To workaround you can
* Install a nightly version of dagster-ssh <https://pypi.org/project/dagster-ssh-nightly/>
* Fork the dagster-ssh solid locally

Hi Alex, thanks for your reply. I found that my issue was actually caused by setting an incorrect location of my google service account auth file. The error log unfortunately does not indicate or hint to this issue.

Just to clarify, once you are no longer hitting the `KeyError`, the real underlying error is not surfaced in a useful way? Can you share what you do end up seeing?

The `KeyError` was caused by incorrect location of the google service account, once I fixed the location issue the pipeline ran as expected without errors.
So real underlying error of the incorrect file location was not shown and `KeyError` showed instead

Ok thank you  - that issue is already fixed in master so will be resolved in the next release