Is there a best practice to migrate from loading assets usin

Question

Is there a best practice to migrate from loading assets using `load assets from airbyte instance` to `load assets from connections` I tried to match all source dest connection settings but still seeing `ValueError Airbyte connections are not in sync with provided configuration` AFTER dagster airbyte applying that reconciler

ben · Answer

Hi Dusty that s odd are you able to see a diff running `dagster airbyte check`

Dustin Shapiro · Answer

Yea I can run through the process again and document it as it s just our dev environment I reverted it but can pick this back up after lunch

Dusty Shapiro · Answer

One thing I did notice is that when I turned off streams in Airbyte the asset definitions weren t reloading in Dagster and when I went to materialize those assets in Dagster it throws the ```op airbyte sync 63c3b did not fire outputs dagster core errors DagsterStepOutputNotFoundError Core compute for op airbyte sync 63c3b did not return an output for non optional output foo ```

Dusty Shapiro · Answer

a no op diff for the `load assets from airbyte instance` method alleviates it

Dusty Shapiro · Answer

Starting the process now of loading assets from the `load assets from connections` for a connection that already existed

Dusty Shapiro · Answer

Looks ilke the above was caused by the assets not being reloaded unless I did a no op deploy but once I aligned the stream config with the existing Airbyte asses I get this in the deployment gt ```TypeError reduce of empty sequence with no initial value ```

Dusty Shapiro · Answer

``` File usr local lib python3 7 site packages dagster grpc server py line 245 in init self container image File usr local lib python3 7 site packages dagster grpc server py line 120 in init repo def = recon repo get definition File usr local lib python3 7 site packages dagster core definitions reconstruct py line 117 in get definition return repository def from pointer self pointer self repository load data File usr local lib python3 7 site packages dagster core definitions reconstruct py line 787 in repository def from pointer repo def = repository def from target def target repository load data File usr local lib python3 7 site packages dagster core definitions reconstruct py line 776 in repository def from target def return target compute repository definition File usr local lib python3 7 site packages dagster core definitions repository definition py line 1549 in compute repository definition return self get repository definition repository load data File usr local lib python3 7 site packages dagster core definitions repository definition py line 1529 in get repository definition default logger defs=self default logger defs File usr local lib python3 7 site packages dagster core definitions repository definition py line 857 in from list default executor def=default executor def File usr local lib python3 7 site packages dagster core definitions unresolved asset job definition py line 159 in resolve asset selection=self selection resolve assets source assets File usr local lib python3 7 site packages dagster core definitions asset selection py line 157 in resolve return self resolve inner asset graph File usr local lib python3 7 site packages dagster core definitions asset selection py line 230 in resolve inner for asset key in selection```

ben · Answer

Hmm it looks like the asset job is resolving no assets What s the asset job that s triggering this

Dusty Shapiro · Answer

The definition is ```airbyte config assets = load assets from connections airbyte=airbyte connections= payments to warehouse key prefix= payments backend ```

ben · Answer

is there a job in your repository that relies on these assets

ben · Answer

I m wondering if maybe the asset names or prefix are mismatched and it s causing a job to resolve as empty

Dusty Shapiro · Answer

Good thought let me check

Dusty Shapiro · Answer

We do define an asset job that has a group name value that matches this line of the previous `load assets from airbyte instance` ``` connection to group fn=lambda group name payments backend replication ```

Dusty Shapiro · Answer

It would seem that it s defining an asset job based on that `connection to group fn` group name value eh

ben · Answer

Yup I think you ll want `connection to group fn` on the `load assets from connections` call too I think what s happening is that job is not finding the new assets bc they re missing the group name

Dusty Shapiro · Answer

Good catch let me add that in and take it do dev

Dusty Shapiro · Answer

Looks like that did the trick thank you Ben

Dusty Shapiro · Answer

I think my original issue had to do with the asset reloading which was alleviated by the no op d the asset definition

ben · Answer

Yeah that s very odd something another user encountered recently

ben · Answer

We are hoping to figure out why that data seems to be cached even after reload

Dustin Shapiro · Answer

Yea probably my colleague Guy We re just causing all sorts of problems I ll leave y all alone for the rest of the year Happy holidays

ben · Answer

thank you for the detailed write up this should make it easier for us to try to replicate on our end pray happy holidays

Dusty Shapiro · Answer

Of course I m wondering if it related to our infrastructure K8s but I ll try not to think about it until after Xmas

Dusty Shapiro · Answer

FWIW our instance of Dagster is deployed on K8s via Helm Chart

Dusty Shapiro · Answer

< ben> would it make sense for me to open an issue for the above

ben · Answer

That would be great I will have some time to try to replicate this today+tomorrow

Dusty Shapiro · Answer

sounds good should be able to write it up tonight

Dusty Shapiro · Answer

Opened the issue <https github com dagster io dagster issues 11531|here> Happy to help provide any other info needed to reproduce