Hi! I’m just starting to explore Dagster (it seems...
# dagster-plus
e
Hi! I’m just starting to explore Dagster (it seems perfect for my use case) and attempting to set up the “Dagster + GCP starter kit” (screenshot 1), however hitting some issues: 1. In the screen in screenshot 2 it doesn’t specify in what format the credentials should be added. From reading docs I believe this is the GCP JSON key file for a service account -- base64 encoded. It would be helpful to just mention this here, or link to docs (I had to find docs separately, and found these) 2. Even so, these credentials don’t seem to work, or I’m doing it wrong, as I’m getting an exception
google.auth.exceptions.DefaultCredentialsError: Your default credentials were not found.
(see attached for full error). I’m currently blocked on #2 and unsure how to proceed. Can I upload a JSON somewhere to Dagster Cloud and set
GOOGLE_APPLICATION_CREDENTIALS
in the UI..? Thanks so much for any help.
w
Welcome Emanuel!
e
Curious if there is any update on how to make things work with BigQuery?
d
Hey Emanuel - sorry for the delay here. Do the tips in this discussion help? https://github.com/dagster-io/dagster/discussions/12183
e
@daniel That link did not help unfortunately. But based on docs I modified the code as per this diff which seems to work. Basically the GCP template on your website contains a bug in that it doesn’t properly use the newer
BigQueryPandasIOManager
. Might want to fix that for other first-time users exploring Dagster. It would also be great to document what permissions the BigQuery user actually needs (I couldn’t find a list anywhere).
d
Got it - glad you're unblocked, The link you posted is 404ing for me though
e
My bad, apparently I set it as private. But this is the diff:
Basically, the website template asks you to fill in
BIGQUERY_SERVICE_ACCOUNT_CREDENTIALS
but this is never used in the repo that’s bootstrapped. Once that env var is actually passed into the IO Manager everything works fine.
So basically, it seems the repo that’s bootstrapped from the website template should be updated
d
got it - thanks for digging into that, we'll get that updated