Hello everyone, I have a python function for migr...
# ask-community
d
Hello everyone, I have a python function for migrating tables from a database to another. So I want to create a job calling the same migration function, but with a different argument value defining the schema name. For this I've used the following doc: https://docs.dagster.io/concepts/configuration/configured#-reusing-an-op-definition-with-configured
Copy code
migrate__1 = configured(migration_func,
                        name="migrate__1")({"db_source": "database_name",
                                            "schema_list": ["schema_name"]})
migrate__2 = configured(migration_func,
                        name="migrate__2")({"db_source": "database_name",
                                            "schema_list": ["schema_name"]})
@job
def job_db_migration():
    migrate__1()
    migrate__2()
It works just fine. However I cant access/set the variables "_db_source_" and "_schema_list_" from the launchpad. If I write the following structure, it doesn't change the execution of the job:
{
ops: {
migrate__1: {
config: {
db_source: "db_name",
schema_list: ["schema_name_new"]
}
}
}
}
What am I doing wrong? Thank you so much !
j
Hi @Davi the
configured
API sets the values for the configuration in code, so in your code snippet you are setting
db_source
to
database_name
. If you want to set values in the launchpad, instead you can define your job like this
Copy code
@job  
def job_db_migration():
    migrate.alias("migrate__1")
    migrate.alias("migrate__2")
d
Hello :) Am I able to set one of the arguments via code and let only one to set via launchpad? On the same line where I choose an alias, I could as well, somehow, set one of the arguments ? Thank you so much !
j
yeah you can use configured but only set one of the config values and then the other should still be configurable in the launchpad
Copy code
migrate__1 = configured(migration_func,
                        name="migrate__1")({"db_source": "database_1"})
migrate__2 = configured(migration_func,
                        name="migrate__2")({"db_source": "database_2"})
@job
def job_db_migration():
    migrate__1()
    migrate__2()
d
Thank you so much @jamie ! I really appreciate your help. :)
Hello @jamie, When I did as you said: configure some of the variables via configured() and the remaining ones via launchpad, the launchpad keeps telling me there is a missing variable even after I setting it. What am I doing wrong ? Thank you in advance !
I have my op migration function down below:
On this picture you can see that I must set the "schema name" and also a "execute" variable in order to trigger or not the migration. In the job, I pre-set the schema name via the configured() function, and let the "execute" variable to be set via launchpad.
Than I have something like this on the launchpad:
It seems everything is fine... But when I launch it, I find this error:
dagster._core.errors.DagsterInvalidConfigError: Error in config for solid 'apac_environment'
Error 1: Missing required config entry "execute" at path root:config. Sample config for missing entry: {'execute': True}
I hope you can give me a little help on this 🙂 @jamie
j
that's a bit odd because the error message is referencing an op called
apac_environment
. is there anything with that name anywhere in your code?
couple other questions: what version of dagster are you using? since the error message also says
solid
are you using the
@solid
decorator anywhere?
d
You are right ! I changed the names on the code to post it here, but schema_1 actually is called
apac_environment
, that's way on the error it uses the actual name on our production env. We have a bunch of different schemas on our migration process, but I called schema_1, schema_2, etc... to make it easier to understand here. :)
No, I don't have any @solid on my functions. My migrate function is only @op :

https://dagster.slack.com/files/U03UW371YPN/F041EM1886Q/cleanshot_2022-09-07_at_10.32.44.png

j
ok thanks for the clarification. we must have a straggling error message referring to solids somewhere. i'll look around for it. not sure how i missed this in the docs before, but to partially specify config you have to use the
@configured
decorator specifically, like this
Copy code
@configured(
    migration_func,
    config_schema={"schema": str}
)
def migrate__1(config):
    return {"db_source": "database_1", "schema": config["schema"]}

@configured(
    migration_func,
    config_schema={"schema": str}
)
def migrate__2(config):
    return {"db_source": "database_1", "schema": config["schema"]}
    
@job
def job_db_migration():
    migrate__1()
    migrate__2()
i tried this out myself and it worked for me. sorry i missed this before and it took a while to get the correct code snippet
d
No problem! The most important is to solve the problem. haha 🙂