https://dagster.io/ logo
#integration-bigquery
Title
# integration-bigquery
j

jamie

02/27/2023, 4:30 PM
debugging thread for @Benedikt Buchert
1
context from this thread: I just ran your code locally and tried materializing and got the following error with the example above:
Copy code
dagster._core.errors.DagsterExecutionHandleOutputError: Error occurred while handling output "result" of step "iris_data":

  File "/opt/homebrew/lib/python3.10/site-packages/dagster/_core/execution/plan/execute_plan.py", line 269, in dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
  File "/opt/homebrew/lib/python3.10/site-packages/dagster/_core/execution/plan/execute_step.py", line 386, in core_dagster_event_sequence_for_step
    for evt in _type_check_and_store_output(step_context, user_event):
  File "/opt/homebrew/lib/python3.10/site-packages/dagster/_core/execution/plan/execute_step.py", line 439, in _type_check_and_store_output
    for evt in _store_output(step_context, step_output_handle, output):
  File "/opt/homebrew/lib/python3.10/site-packages/dagster/_core/execution/plan/execute_step.py", line 640, in _store_output
    for elt in iterate_with_context(
  File "/opt/homebrew/lib/python3.10/site-packages/dagster/_utils/__init__.py", line 469, in iterate_with_context
    with context_fn():
  File "/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/opt/homebrew/lib/python3.10/site-packages/dagster/_core/execution/plan/utils.py", line 85, in op_execution_error_boundary
    raise error_cls(

The above exception was caused by the following exception:
AttributeError: 'NoneType' object has no attribute 'write'

  File "/opt/homebrew/lib/python3.10/site-packages/dagster/_core/execution/plan/utils.py", line 55, in op_execution_error_boundary
    yield
  File "/opt/homebrew/lib/python3.10/site-packages/dagster/_utils/__init__.py", line 471, in iterate_with_context
    next_output = next(iterator)
  File "/opt/homebrew/lib/python3.10/site-packages/dagster/_core/execution/plan/execute_step.py", line 630, in _gen_fn
    gen_output = output_manager.handle_output(output_context, output.value)
  File "/opt/homebrew/lib/python3.10/site-packages/dagster/_core/storage/db_io_manager.py", line 139, in handle_output
    self._handlers_by_type[obj_type].handle_output(context, table_slice, obj, conn)
  File "/opt/homebrew/lib/python3.10/site-packages/dagster_gcp_pandas/bigquery/bigquery_pandas_type_handler.py", line 41, in handle_output
    pandas_gbq.to_gbq(
  File "/opt/homebrew/lib/python3.10/site-packages/pandas_gbq/gbq.py", line 1220, in to_gbq
    connector.load_data(
  File "/opt/homebrew/lib/python3.10/site-packages/pandas_gbq/gbq.py", line 614, in load_data
    chunks = tqdm.tqdm(chunks)
  File "/opt/homebrew/lib/python3.10/site-packages/tqdm/std.py", line 1109, in __init__
    self.refresh(lock_args=self.lock_args)
  File "/opt/homebrew/lib/python3.10/site-packages/tqdm/std.py", line 1361, in refresh
    self.display()
  File "/opt/homebrew/lib/python3.10/site-packages/tqdm/std.py", line 1509, in display
    self.sp(self.__str__() if msg is None else msg)
  File "/opt/homebrew/lib/python3.10/site-packages/tqdm/std.py", line 350, in print_status
    fp_write('\r' + s + (' ' * max(last_len[0] - len_s, 0)))
  File "/opt/homebrew/lib/python3.10/site-packages/tqdm/std.py", line 343, in fp_write
    fp.write(_unicode(s))
  File "/opt/homebrew/lib/python3.10/site-packages/tqdm/utils.py", line 89, in __getattr__
    return getattr(self._wrapped, name)
Does the io manager create the dataset and table automatically? Also I adjusted bellow so i can use these as the column headers in Bigquery:
Copy code
names=[
            "sepal_length_cm",
            "sepal_width_cm",
            "petal_length_cm",
            "petal_width_cm",
            "species",
        ],
I’m having trouble replicating this error, but i have a few follow up questions/things for you to check: 1. Are you authenticated with GCP - ie have you run
google auth …
or have
GOOGLE_APPLICATION_CREDENTIALS
set 2. how are you running the code - ie materializing via the UI, running via the CLI, etc 3. can you update the asset to look like this just to confirm that the dataset is being downloaded correctly?
Copy code
@asset
def iris_data() -> pd.DataFrame:
    df = pd.read_csv(
        "<https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data>",
        names=[
            "sepal_length_cm",
            "sepal_width_cm",
            "petal_length_cm",
            "petal_width_cm",
            "species",
        ],
    )
    print(df)
    return df
❤️ 1
b

Benedikt Buchert

02/27/2023, 5:59 PM
I think I opened the server with the wrong port locally There is an old server running on port 3000: Running it with the port given in the cli it seems to be working.
Thanks for the help.
🎉 1
3 Views