https://dagster.io/ logo
Powered by
#deployment-kubernetes
Title
# deployment-kubernetes
t

trevenrawr

04/22/2022, 9:37 PM
Howdy, folks! A somewhat related question: Is there a way to change the 
container_config
on a per-run basis? Depending on how a particular job runs gets configured (through a 
PartitionedConfig
), I’d like to be able to update a pod’s resource requests, as opposed to always allocating based on the “worst case” resource usage scenario (at the job scope).
d

daniel

04/22/2022, 9:42 PM
Hi Trevor - there's a 
tags_for_partition_fn
arg on PartitionedConfig that you could use for this: https://docs.dagster.io/_apidocs/partitions#partitions
(setting the dagster-k8s/config in that function)
t

trevenrawr

04/22/2022, 9:43 PM
Aha! That looks like the perfect path forward; thank you!
Hahah, I was wondering why I didn’t notice these sooner and realized that I had started this project at 
0.14.6
and they were introduced shortly thereafter! I’m constantly amazed at how you folks are always one step ahead of my needs; thank you again!
condagster 1
Quick followup: Since I’m seeing this error 
Value in dict mismatches expected type for key dagster-k8s/config. Expected value of type <class 'str'>. Got value {'container_config': {'resources': {'requests': {'cpu': '4', 'memory': '48Gi'}, 'limits': {'cpu': '16', 'memory': '48Gi'}}}, 'job_spec_config': {'ttl_seconds_after_finished': 300}} of type <class 'dict'>.
I assume I should turn the dict into a string in the 
tags_for_partition_fn
?
d

daniel

04/22/2022, 11:32 PM
Hmm I think it is expecting it to be json encoded, yeah.
t

trevenrawr

04/22/2022, 11:48 PM
Ahh, yup! Good ol’ 
json.dumps
did the trick; thanks!
d

daniel

04/25/2022, 2:17 PM
Passing in a dict should be possible too though IMO, so sent out a PR for that here: https://github.com/dagster-io/dagster/pull/7554
partywizard 1
ty thankyou 2
c

Charlie Bini

05/03/2022, 7:00 PM
@daniel did this make it into the latest release? just ran into this issue today
d

daniel

05/03/2022, 7:00 PM
It did - what issue are you seeing?
c

Charlie Bini

05/03/2022, 7:02 PM
setting up a schedule with the 
dagster-k8s/config
tag, and getting this:
Copy code
dagster.check.CheckError: Value in dict mismatches expected type for key dagster-k8s/config. Expected value of type <class 'str'>. Got value {'container_config': {'resources': {'limits': {'cpu': '1750m', 'memory': '2500Mi'}}}} of type <class 'dict'>.
d

daniel

05/03/2022, 7:03 PM
got it - possible to share the full stack trace?
c

Charlie Bini

05/03/2022, 7:03 PM
just searched the error and found this thread, trying to see if 
json.dumps
fixes it
d

daniel

05/03/2022, 7:03 PM
that'll most likely fix it, might need one more PR targeted at schedules
c

Charlie Bini

05/03/2022, 7:04 PM
Copy code
dagster.check.CheckError: Value in dict mismatches expected type for key dagster-k8s/config. Expected value of type <class 'str'>. Got value {'container_config': {'resources': {'limits': {'cpu': '1750m', 'memory': '2500Mi'}}}} of type <class 'dict'>.
  File "/root/app/__pypackages__/3.10/lib/dagster/grpc/server.py", line 214, in __init__
    self._loaded_repositories = LoadedRepositories(
  File "/root/app/__pypackages__/3.10/lib/dagster/grpc/server.py", line 94, in __init__
    loadable_targets = get_loadable_targets(
  File "/root/app/__pypackages__/3.10/lib/dagster/grpc/utils.py", line 53, in get_loadable_targets
    else loadable_targets_from_python_package(package_name, working_directory)
  File "/root/app/__pypackages__/3.10/lib/dagster/core/workspace/autodiscovery.py", line 48, in loadable_targets_from_python_package
    module = load_python_module(
  File "/root/app/__pypackages__/3.10/lib/dagster/core/code_pointer.py", line 136, in load_python_module
    return importlib.import_module(module_name)
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/root/app/teamster/local/__init__.py", line 1, in <module>
    from teamster.local.repository import *  # noqa: F401, F403
  File "/root/app/teamster/local/repository.py", line 4, in <module>
    from teamster.local.schedules import powerschool as pss
  File "/root/app/teamster/local/schedules/powerschool.py", line 8, in <module>
    full_schedule = ScheduleDefinition(
  File "/root/app/__pypackages__/3.10/lib/dagster/core/definitions/schedule_definition.py", line 305, in __init__
    check_tags(tags, "tags")
  File "/root/app/__pypackages__/3.10/lib/dagster/core/storage/tags.py", line 67, in check_tags
    check.opt_dict_param(obj, name, key_type=str, value_type=str)
  File "/root/app/__pypackages__/3.10/lib/dagster/check/__init__.py", line 276, in opt_dict_param
    return _check_mapping_entries(obj, key_type, value_type, mapping_type=dict)
  File "/root/app/__pypackages__/3.10/lib/dagster/check/__init__.py", line 1578, in _check_mapping_entries
    raise CheckError(
yup that works
d

daniel

05/03/2022, 10:09 PM
OK, https://github.com/dagster-io/dagster/pull/7720 should hopefully handle this class of things once and for all
c

Charlie Bini

05/20/2022, 5:29 PM
hey @daniel just caught that this is still an issue in the Docker agent. Not sure if that's expected there, but I'm using it for my staging site
if there isn't a hard reason for Docker tags to only accept string values in tags, it would be great to allow dicts so I can avoid having to change code between staging and prod
it's only an issue for Schedules still, Job tags accept it
d

daniel

05/20/2022, 6:06 PM
Hey Charlie - that fix should have gone out in dagster 0.14.14 - is your code at that version or greater?
if so would you mind posting or DMing a snippet of the schedule code that's hitting the issue?
c

Charlie Bini

05/20/2022, 6:13 PM
yeah Im on 0.14.16
Copy code
full_tables = ScheduleDefinition(
    name="full_tables",
    job=powerschool_extract,
    cron_schedule=" 40 14 * * *",
    execution_timezone=LOCAL_TIME_ZONE,
    run_config=config_from_files(
        ["./teamster/local/config/powerschool/query-full.yaml"]
    ),
    tags={
        "dagster-k8s/config": {
            "container_config": {
                "resources": {"limits": {"cpu": "1750m", "memory": "3Gi"}}
            }
        }
    },
)
lost the exception, but that's an example of a schedule that threw the error
d

daniel

05/20/2022, 6:16 PM
hmmm a stack trace of the exception would be really helpful I think. When I check out 0.14.13 and load that I get:
Copy code
dagster.check.CheckError: Value in dict mismatches expected type for key dagster-k8s/config. Expected value of type <class 'str'>. Got value {'container_config': {'resources': {'limits': {'cpu': '1750m', 'memory': '3Gi'}}}} of type <class 'dict'>.
but in 0.14.16 it loads without an error
c

Charlie Bini

05/20/2022, 6:18 PM
hang on I think I might know what's up
think my local image is out of date
that explains it, false alarm
d

daniel

05/20/2022, 6:30 PM
hooray!
Powered by