Andrew Smith
08/18/2022, 8:18 PMwith_resources
with define_asset_job
, but I am getting a strange error from with_resources. I honestly can't tell if this is an error on my side or Dagster's end. The docs for with_resources
states, "Adds dagster resources to copies of resource-requiring dagster definitions." It seems define_asset_job
should be a resource requiring dagster definition.
The code:
fin_and_ops_job = with_resources(
define_asset_job(
name="fin_and_ops_job",
selection=AssetSelection.assets(fetched_table_updates, pushed_table_updates),
partitions_def=table_name_partitions,
config=db_config
),
resource_defs={
"source_db": sql_resources.sql_res,
"sink_db": sql_resources.sql_res
}
)
The Error:
AttributeError: 'str' object has no attribute 'with_resources'
File "/home/dev/.local/share/virtualenvs/_-26LfFVFv/lib/python3.10/site-packages/dagster/_grpc/server.py", line 227, in __init__
self._loaded_repositories = LoadedRepositories(
File "/home/dev/.local/share/virtualenvs/_-26LfFVFv/lib/python3.10/site-packages/dagster/_grpc/server.py", line 101, in __init__
loadable_targets = get_loadable_targets(
File "/home/dev/.local/share/virtualenvs/_-26LfFVFv/lib/python3.10/site-packages/dagster/_grpc/utils.py", line 53, in get_loadable_targets
else loadable_targets_from_python_package(package_name, working_directory)
File "/home/dev/.local/share/virtualenvs/_-26LfFVFv/lib/python3.10/site-packages/dagster/_core/workspace/autodiscovery.py", line 48, in loadable_targets_from_python_package
module = load_python_module(
File "/home/dev/.local/share/virtualenvs/_-26LfFVFv/lib/python3.10/site-packages/dagster/_core/code_pointer.py", line 136, in load_python_module
return importlib.import_module(module_name)
File "/usr/lib64/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/home/dev/PycharmProjects/_/_/__init__.py", line 1, in <module>
from .repository import emperitas
File "/home/dev/PycharmProjects/_/_/repository.py", line 6, in <module>
from .assets.oxford_template import fetch_table_updates, push_table_updates, fin_and_ops_job
File "/home/dev/PycharmProjects/_/_/assets/oxford_template.py", line 63, in <module>
fin_and_ops_job = with_resources(
File "/home/dev/.local/share/virtualenvs/_-26LfFVFv/lib/python3.10/site-packages/_/_core/execution/with_resources.py", line 101, in with_resources
transformed_defs.append(cast(T, definition.with_resources(resource_defs)))
sandy
08/18/2022, 8:33 PMdefine_asset_job
to with_resources
. instead, you'd do something like:
@repository
def repo():
return [with_resources(assets, resource_defs=...), define_asset_job(...)]