The `with_resources` function doesn't work for `jo...
# ask-community
d
The
with_resources
function doesn't work for
job
objects. Is it possible to define a
job
once and then provide different resources for different repositories (e.g. dev and prod)? Normally this could be done via defining a
job
from
graph
, but my job is defined using
build_assets_job
from assets. What's the best solution here?
s
Hey Daniel - are you able to use
define_asset_job
?
d
Thanks, seems like this is what I need. However, it fails with
Copy code
dagster._check.CheckError: Invariant failed. Description: Assets defined for node 'catalog_session_events' have a partitions_def of Daily, starting 2022-05-01 Europe/Moscow., but job 'daily_etl' has non-matching partitions_def of Daily, starting 2022-05-31 Europe/Moscow..
These assets have a different start_date since some of the downstream assets depend on multiple partitions of the upstream assets (and therefore should start later). This error doesn't seem logical, especially considering I'm using partitions mapping and Dagster should know the assets are mapped correctly, but most importantly, this forbids sharing an asset between two jobs with different partitioning:. Consider: 1. Training job has a daily partitioning with an offset of 1 day, because we need the current partition to make targets for the previous one. 2. Inference job has a normal daily partitioning. Currently. it's impossible to share
data
and
model
assets between these two jobs (created via
define_asset_job
), because they required different partitioning. Anyway, after changing all the partitions_def to be the same just to try
define_asset_job
out, I got this error:
Copy code
dagster._check.CheckError: Invariant failed. Description: Duplicate definition found for daily_etl_partition_set
Seems like this is a bug (I can't find any duplicated definitions), I found it reported elsewhere. So seems like I'll have to go with
build_assets_job
for now (as it allows to override the asset partitioning when building the job). I'll just have to call it once per every
repository
, every time with different resources. Maybe
define_asset_job
should let the user to override the partitioning in the same way?
j
@sandy I think something similar is happening with our dag. @Daniel Gafni did you resolve this?
s
Hey @Jeremy - are you seeing an error like this?
Copy code
dagster._check.CheckError: Invariant failed. Description: Duplicate definition found for daily_etl_partition_set
if so, that's fixed in a recent version of Dagster
d
I didn't try this in the new version yet. Will do today.
Nope - still can't use assets with different partitions
start_date
for the same
define_asset_job
Copy code
dagster._check.CheckError: Invariant failed. Description: Assets defined for node 'line_items_stats_with_aggs' have a partitions_def of Daily, starting 2022-05-15 Europe/Moscow., but job 'daily_etl' has non-matching partitions_def of Daily, starting 2022-05-01 Europe/Moscow..
This means I can't use the same asset - ML model - both for training and inference jobs as they have different partitions
Copy code
dagster._check.CheckError: Invariant failed. Description: Assets defined for node 'ranking_model' have a partitions_def of Daily, starting 2022-05-29 Europe/Moscow. End offsetted by -1 partitions., but job 'catalog_ranking_inference' has non-matching partitions_def of Daily, starting 2022-05-01 Europe/Moscow..
j
upgrading worked for me.