Dusty Shapiro
12/22/2022, 3:09 PMload_assets_from_airbyte_instance
to load_assets_from_connections
?
I tried to match all source/dest/connection settings, but still seeing ValueError: Airbyte connections are not in sync with provided configuration
AFTER dagster-airbyte applying that reconciler.ben
12/22/2022, 5:33 PMdagster-airbyte check
?Dustin Shapiro
12/22/2022, 6:08 PMDusty Shapiro
12/22/2022, 7:29 PMop 'airbyte_sync_63c3b' did not fire outputs
dagster._core.errors.DagsterStepOutputNotFoundError: Core compute for op "airbyte_sync_63c3b" did not return an output for non-optional output "foo"
load_assets_from_airbyte_instance
method alleviates it.load_assets_from_connections
for a connection that already existed.TypeError: reduce() of empty sequence with no initial value
File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/server.py", line 245, in __init__
self._container_image,
File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/server.py", line 120, in __init__
repo_def = recon_repo.get_definition()
File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/reconstruct.py", line 117, in get_definition
return repository_def_from_pointer(self.pointer, self.repository_load_data)
File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/reconstruct.py", line 787, in repository_def_from_pointer
repo_def = repository_def_from_target_def(target, repository_load_data)
File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/reconstruct.py", line 776, in repository_def_from_target_def
return target.compute_repository_definition()
File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/repository_definition.py", line 1549, in compute_repository_definition
return self._get_repository_definition(repository_load_data)
File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/repository_definition.py", line 1529, in _get_repository_definition
default_logger_defs=self._default_logger_defs,
File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/repository_definition.py", line 857, in from_list
default_executor_def=default_executor_def,
File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/unresolved_asset_job_definition.py", line 159, in resolve
asset_selection=self.selection.resolve([*assets, *source_assets]),
File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/asset_selection.py", line 157, in resolve
return self.resolve_inner(asset_graph)
File "/usr/local/lib/python3.7/site-packages/dagster/_core/definitions/asset_selection.py", line 230, in resolve_inner
for asset_key in selection
ben
12/22/2022, 9:24 PMDusty Shapiro
12/22/2022, 9:25 PMairbyte_config_assets = load_assets_from_connections(
airbyte=airbyte,
connections=[payments_to_warehouse],
key_prefix="payments_backend",
)
ben
12/22/2022, 9:26 PMDusty Shapiro
12/22/2022, 9:26 PMload_assets_from_airbyte_instance
# connection_to_group_fn=lambda group_name: "payments_backend_replication",
connection_to_group_fn
group_name value eh?ben
12/22/2022, 9:31 PMconnection_to_group_fn
on the load_assets_from_connections
call too? I think what’s happening is that job is not finding the new assets bc they’re missing the group nameDusty Shapiro
12/22/2022, 9:32 PMben
12/22/2022, 10:42 PMDustin Shapiro
12/22/2022, 10:56 PMben
12/23/2022, 3:45 PMDusty Shapiro
12/23/2022, 3:46 PMben
01/05/2023, 8:56 PMDusty Shapiro
01/05/2023, 9:03 PM