19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
#dagster-support
Title
# dagster-support
j
jasono
04/04/2022, 12:39 AMHi, how do I change the base_dir of dagstermill output notebook? The source code indicates the base_dir can be defined in the init_context argument so I'm trying to create an init_context object reflecting the desired output base_dir. I'm stuck at creating this init_context object, especially this part.
init_context.instance.storage_directory()
c
claire
04/04/2022, 5:01 PMHi jasono. The dagstermill library contains a
get_contextbase_dir Copy code
import dagstermill as dm
from dagster import ModeDefinition, local_file_manager
context = dm.get_context(
mode_def=ModeDefinition(resource_defs={"file_manager": local_file_manager}),
run_config={"resources": {"file_manager": {"config": {"base_dir": ...}}}},
)j
jasono
04/04/2022, 10:34 PMThank you for your response. Unfortunately I’m getting this error. I seem to be putting
context Copy code
Dagster.check.ParameterCheckError: Param "init_context" is not a InitResourceContext. Got <dagstermill.context.DagstermillExecutionContext object at 0x00000013E96B7D90> which is type <class 'dagstermill.context.DagstermillExecutionContext'>.Here is my code
Copy code
context1 = dm.get_context(
mode_def = ModeDefinition(
resource_defs = {
"file_manager": local_file_manager}
),
run_config={
"resources": {"file_manager":
{"config": {"base_dir":"t:/data/output/me_recon/recon_8510R" }}
}
}
)
notebook_op_8510R = dm.define_dagstermill_op(
"recon_8510R",
script_relative_path("datapipeline/me_recon_supports/recon_8510R.ipynb"),
output_notebook_name="recon_8510R_output",
config_schema={
"text1": Field(
Int,
default_value=777,
is_required=False,
description="The number of clusters to find",
)},
)
@job(
resource_defs={
"output_notebook_io_manager": dm.local_output_notebook_io_manager(context1)
}
)
def run_8510R():
notebook_op_8510R()c
claire
04/04/2022, 11:09 PMHi jasono, got it. So in this case because you are providing config to a specific resource
output_notebook_io_manager Copy code
@job(
resource_defs={
"output_notebook_io_manager": dm.local_output_notebook_io_manager
},
config={
"resources": {"output_notebook_io_manager":
{"config": {"base_dir":"t:/data/output/me_recon/recon_8510R" }}
}
}
)
def run_8510R():
notebook_op_8510R()j
jasono
04/04/2022, 11:16 PMIt worked!! One more question if you don’t mind. I noticed the output notebook is placed in a folder within a randomly named zip file. Is there a way to make it just save a file (ideally with a custom name) without a folder or zipping?
c
claire
04/04/2022, 11:50 PMcc @yuhan
2 Views