Shane Kalepp
06/19/2023, 3:12 PMload_assets_from_dbt_cloud_job
does not bring in the automatic dependency as the load_assets_from_dbt_project
function does. Is this intended?Shane Kalepp
06/19/2023, 3:23 PMrex
06/20/2023, 3:01 PMAssetKey
for the open_weather
dbt source that’s ingested. The `AssetKey`’s have to be exactly the same for the dependency to be understood properly.
cc @claire, who is working on utilities to make this more ergonomicShane Kalepp
06/22/2023, 7:47 PM@asset
looks like:
@asset(
group_name="weather",
compute_kind="API",
io_manager_key="snowflake_io_manager",
non_argument_deps="open_weather"
)
def open_weather() -> pd.DataFrame:
cities_data = []
for city in cities:
url = f"<https://api.openweathermap.org/data/2.5/weather?q={city}&units=imperial&appid=apikey>"
weather = requests.get(url).json()
cities_data.append(weather)
df = pd.json_normalize(cities_data)
df = df.drop(columns=["weather"])
df["dt"] = pd.to_datetime(df["dt"], unit="s").dt.date # Convert to date which will be used to partition by
return df
Here's what my `sources.yml`file looks like:
version: 2
sources:
- name: weather
database: dev
schema: skalepp
tables:
- name: open_weather
And this is what it loads. Any help would be appreciated.claire
06/22/2023, 10:06 PMnon_argument_deps
arg accepts a set instead of a string. In this case, it's interpreting your string as a set. We really should raise an error for this, but I think you can define your asset instead with:
@asset(non_argument_deps={AssetKey("open_weather")})
def open_weather():
...