https://dagster.io/ logo
Join Slack
Powered by
Is it possible to schedule the same job with diffe...
# ask-community
c
Is it possible to schedule the same job with different config for each schedule? e.g at 4AM with config1 and 10AM with config2
o
hi @Chris Histe! yep this is possible by creating two separate schedules, each pointing at the same job
c
I tried that already, and it didn’t work, can you share a bit more technical details because I’m getting a 
Duplicate definition found for X
o
can you share your code?
c
Copy code
TOOL_REFRESH_DBT_DATASET = "tool_refresh_dbt_dataset"
asset_definition_refresh_dbt_dataset = AssetsDefinition.from_graph(
    refresh_dbt_dataset,
    group_name=TOOL_REFRESH_DBT_DATASET,
    resource_defs={
        "refresh_dbt_dataset_config": make_values_resource(
            source_dataset=str,
            destination_dataset=str,
        )
    },
)
asset_job_refresh_dbt_dataset = define_asset_job(
    TOOL_REFRESH_DBT_DATASET,
    AssetSelection.groups(TOOL_REFRESH_DBT_DATASET),
    get_config(),
)
schedule_refresh_dbt_chris = ScheduleDefinition(
    job=asset_job_refresh_dbt_dataset, cron_schedule="0 */6 * * *" 
)
schedule_refresh_landing_development = ScheduleDefinition(
    job=asset_job_refresh_dbt_dataset, cron_schedule="0 1 * * *"
)
Something like this
This is missing the different 
run_config
that I will need
o
ah yeah you'll want to pass in different `name`s for each of your 
ScheduleDefinition
s to disambiguate them (the default name is just 
<name of job>_schedule
, so these end up with the same name)
c
Copy code
DagsterInvalidDefinitionError: schedule 'schedule_refresh_landing_development' targets unresolved asset job 'tool_refresh_dbt_dataset', but a different unresolved asset job with the same name was provided. Disambiguate between these by providing a separate name to one of them.
with
Copy code
schedule_refresh_dbt_chris = ScheduleDefinition(
    name="schedule_refresh_dbt_chris",
    job=asset_job_refresh_dbt_dataset,
    cron_schedule="0 */6 * * *",
)
schedule_refresh_landing_development = ScheduleDefinition(
    name="schedule_refresh_landing_development",
    job=asset_job_refresh_dbt_dataset,
    cron_schedule="0 1 * * *",
)
Do I need to create an asset job for each?
I love Dagster but this is so painful, I always lose time on those APIs
o
hm I'm not able to replicate this (you shouldn't have to create separate asset jobs) -- are you gathering these definitions into a Definitions() object? and is it possible that there's another 
tool_refresh_dbt_dataset
existing somewhere? quite strange...
c
Let me double check my code
Facepalm
Alright, I fixed the code but now I’m getting another error. For 
make_values_resource
it’s through 
run_config
right?
e.g.
Copy code
schedule_refresh_landing_development = ScheduleDefinition(
    name="schedule_refresh_landing_development",
    job=asset_job_refresh_dbt_dataset,
    cron_schedule="0 8 * * *",  # 4am EST
    run_config={
        "resources": {
            "refresh_dbt_dataset_config": {
                "config": {
                    "destination_dataset": "landing_development",
                    "source_dataset": "landing_production",
                }
            }
        }
    },
)
Copy code
Error 1: Missing required config entry "resources" at the root. Sample config for missing entry: {'resources': {'schedule_refresh_landing_development': {'config': {'destination_dataset': '...', 'source_dataset': '...'}}}}
o
when are you getting that error?
c
dagster dev
o
is it when it's attempting to launch a job though? I wouldn't expect to see an error about the config unless a run was requested from somewhere
c
Copy code
/Users/christopherhiste/Library/Caches/pypoetry/virtualenvs/pipelines-4jD-Dlvi-py3.10/lib/python3.10/site-packages/dagster/_core/workspace/context.py:602: UserWarning: Error loading repository location pipelines:dagster._core.errors.DagsterInvalidConfigError: Error in config when building job 'tool_refresh_dbt_dataset'
    Error 1: Missing required config entry "resources" at the root. Sample config for missing entry: {'resources': {'refresh_dbt_dataset_config': {'config': {'destination_dataset': '...', 'source_dataset': '...'}}}}
No that’s when I try to load Dagster
Actually that might be me again
My brain is fried
🍟 1
Lol fixed
Let me try
I think it’s good. Thanks for the help. Sorry for wasting your time
o
no problem 🫡
16 Views
Open in Slack
PreviousNext
Powered by