https://dagster.io/ logo
#ask-community
Title
# ask-community
p

Peter Davidson

11/26/2022, 8:03 AM
Trying to set the local_artifact_storage.config.base_dir and comput_logs.config.base_dir parameters at run time. Something like:
Copy code
run_config = {'local_artifact_storage':
                  {'config':
                       {'base_dir': r'C:\Users\peter\dagster'
                        }
                   }
              }
How can I add this to an @repository ?
Copy code
@repository
def my_repo():
    return [test_asset1, test_asset2]
q

Qwame

11/26/2022, 3:28 PM
Can this be configured in
dagster.yaml
?
p

Peter Davidson

11/26/2022, 4:36 PM
yes it can. but, this requires the yaml to be in the project folder, or to be pointed to using dagit -w <path_to_yaml>/dagster.yaml Typically all my config files are in subfolders of the users $home folder -> this allows platform agnostic install (windows & unix) and keeps config files out of the git repo. It also requires the path to be known when dagster is launched. What I would like is a dynamic path e.g. via a get_config() function.
o

owen

11/28/2022, 7:36 PM
for local artifact storage, assuming that your goal is to control where the pickled asset outputs are stored, you can configure the default io manager with:
Copy code
run_config = {"resources": {"io_manager": {"config": {"base_dir": "<some_base_dir>"}}}}
or
Copy code
from dagster import fs_io_manager
@repository
def my_repo():
    return with_resources([test_asset1, test_asset2], resource_defs={"io_manager": fs_io_manager.configured({"base_dir": "some_base_dir"})
p

Peter Davidson

11/28/2022, 7:36 PM
Awesome
exactly the syntax I was struggling with
o

owen

11/28/2022, 7:36 PM
for compute logs, unfortunately it is only possible to configure their location at the instance level
🌈 1
p

Peter Davidson

11/28/2022, 7:37 PM
the context config stuff is a concept I'm still learning tbh
yeah, this is a good start though, means within the team we can share a common file system (even if run logs and instantiation isn't fully shared)
🎉 1
o

owen

11/28/2022, 7:40 PM
one more note (not sure if relevant), but if the user has a DAGSTER_HOME environment variable set (pointing at a directory w/ a dagster.yaml file in it), then you don't need to pass that path into dagster-daemon or dagit invocations
p

Peter Davidson

11/28/2022, 7:41 PM
yep I saw that. So we could all have a dagster.yaml with our file paths, then just set the home just before typing dagit
gotta say, dagster seems way simpler than flyte
o

owen

11/28/2022, 7:44 PM
glad to hear it! 😄
p

Peter Davidson

11/28/2022, 7:45 PM
just gotta get a working PoC and then convince the team we adopt it. (we need a framework)
o

owen

11/28/2022, 7:45 PM
let me know if you run into any roadblocks 🙂
🙏 1
p

Peter Davidson

11/30/2022, 8:16 AM
Making progress 🙂 I have somethign like this -> basically I want the base_dir that is passed to assets to be a calculated value. The job here works, and a typical use would be to open the launchpad in dagit, type in an id which then returns a path_to_folder as job output. How do you configure something similar for the assets? Code below doesn't work for the assets, only the get_fldr job So, launch dagit, type in an id, then materialize assets&resources which use the id as a config -> or rather, the output from the job as an input.
Copy code
from dagster import load_assets_from_package_module, repository, with_resources, op, job

@op(config_schema={'param_set_id': int})
def get_ws_root_from_param_set_id(context) -> str:
    folders = {1: '/fldr1', 2: '/fldr2'}
    return folders.get(context.op_config['param_set_id'])

@job
def get_fldr() -> str:
    get_ws_root_from_param_set_id()


@repository
def results_set():
    return [
        *with_resources(
            load_assets_from_package_module(assets)
            , resource_defs={
                "io_manager": local_pkl_io_manager.configured(
                    {"base_dir": get_ws_root_from_param_set_id}),
            }
            ,
        ),
        get_fldr
    ]
5 Views