https://dagster.io/ logo
Title
k

Kyle Downey

11/16/2021, 8:18 PM
I am following the hacker_news example, specifically https://github.com/dagster-io/dagster/blob/master/examples/hacker_news/hacker_news/jobs/hacker_news_api_download.py. I have replicated the hourly_download_config function, but when I replace a hard-coded start/end time config with the function like shown in the example, it fails with an error: dagster.core.errors.DagsterInvalidConfigError: Error in config for job Error 1: Missing required config entry "resources" at the root. Sample config for missing entry: {'resources': {'partition_bounds': {'config': {'end': '...', 'start': '...'}}}}
The commented-out code works, but the code shown (the same as hacker_news prod job) does not):
load_uniswap_trades_local_job = <http://load_uniswap_trades.to|load_uniswap_trades.to>_job(
resource_defs={ 'partition_bounds': partition_bounds, 'moonstream_client': live_moonstream_client.configured({'api_key': os.getenv('MOONSTREAM_API_KEY')}) }, config=hourly_download_config, # config={ # 'resources': { # 'partition_bounds': { # 'config': { # 'start': '2021-11-13 00:00:00', # 'end': '2021-11-13 01:00:00', # } # }, # } # } )
@hourly_partitioned_config(start_date=datetime(2021, 11, 16))
def hourly_download_config(start: datetime, end: datetime):
return {
'resources': {
'partition_bounds': {
'config': {
'start': start.strftime(PARTITION_BOUND_DATE_FMT),
'end': end.strftime(PARTITION_BOUND_DATE_FMT),
}
},
}
}
Possible hint: this happens when I run the job with "dagster job execute" but with dagit it seems fine. I am prompted to select the hourly partition and then it creates a config similar to the one above with start/end date and runs it.
Pretty sure this is just a confusing error message resulting from a problem my end. I created a partitioned job in the repo similar to the HN example and instead of running it with "job execute" in dagster I ran with "job backfill" -- this prompted me with a list of partitions not yet backfilled and then ran fine.
s

sandy

11/16/2021, 9:20 PM
Hey Kyle - I believe that behavior is expected with
dagster job execute
. Because you're providing partitioned config to your job (via
hourly_download_config
), Dagster needs to know what partition to use to generate config when launching the job
dagster job execute
doesn't allow you to provide a partition, so no partition is provided, so Dagster can't use the config from
hourly_download_config
Does that make sense? In the future, we should probably add a
--partition
arg to
dagster job execute
k

Kyle Downey

11/17/2021, 10:44 PM
Hi Sandy, yes, a --partition argument for dagster job execute makes sense.