19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hello Dagster,
Could you help on this issue?
I’m t...
# ask-communityw
Wonjae Lee
05/05/2023, 11:50 PMHello Dagster,
Could you help on this issue?
I’m trying to add sensor, job, ops using new python configurations, but got the error below:
Error loading repository location deciduouse_definitions.pydagster. core.errors.DagsterInvalidConfigError Error in config when building job ‘deciduous_ml_job’
Error 1: Missing required config entry “ops” at the root.
Sample config for missing entry:
{‘ops’: {‘create_raw_data’: {‘config’: {‘company_id’: ‘...’, ‘label_events_name’: ‘...’, ‘model_id’: ‘...’, ‘version_id’: ‘...’}},
‘download_model’: {‘config’: {‘company_id’: ‘...’, ‘gcs_bucket’: ‘...’, ‘model_id’: ‘...’, ‘model_type’: ‘...’, ‘version’: 0}},
‘export_evaluation’: {‘config’: {‘company_id’: ‘...’, ‘dataset’: ‘...’, ‘model_id’: ‘...’, ‘model_type’: ‘...’, ‘project’: ‘...’, ‘version’: 0}},
‘export_model’: {‘config’: {‘company_id’: ‘...’, ‘dataset’: ‘...’, ‘gcs_bucket’: ‘...’, ‘model_id’: ‘...’, ‘model_type’: ‘...’, ‘version’: 0}},
‘generate_model_version’: {‘config’: {‘company_id’: ‘...’, ‘model_id’: ‘...’, ‘model_type’: ‘...’}},
‘train_model’: {‘config’: {‘collection_name’: ‘...’, ‘company_id’: ‘...’, ‘database_name’: ‘...’, ‘dataset’: ‘...’, ‘model_id’: ‘...’, ‘model_type’: ‘...’, ‘project’: ‘...’, ‘version’: 0}}}}
Wonjae Lee
05/05/2023, 11:51 PMthe code snippets are:
run_config = RunConfig(ops={
“generate_model_version”: ModelVersionConfig(company_id=model[‘company_id’],
model_id=model[‘model_id’],
model_type=model[‘model_type’]),
“create_raw_data”: CreateRawDataConfig(label_events_name=model[‘target_event’],
company_id=model[‘company_id’],
model_id=model[‘model_id’],
version_id=model[‘version’]),
“train_model”: TrainModelConfig(project=“statsig-ingestions”,
dataset=“deciduous”,
company_id=model[‘company_id’],
model_id=model[‘model_id’],
model_type=model[‘model_type’],
version=model[‘version’],
database_name=“company”,
collection_name=“deciduous_model”),
“export_model”: ExportModelConfig(dataset=“deciduous”,
company_id=model[‘company_id’],
model_id=model[‘model_id’],
version=model[‘version’],
model_type=model[‘model_type’],
gcs_bucket=“deciduous_models”),
“export_evaluation”: CreateModelEvaluationConfig(
project=“statsig-ingestions”,
dataset=“deciduous”,
company_id=model[‘company_id’],
model_id=model[‘model_id’],
version=model[‘version’],
model_type=model[‘model_type’]),
“download_model”: DownloadModelConfig(gcs_bucket=“deciduous_models”,
company_id=model[‘company_id’],
model_id=model[‘model_id’],
version=model[‘version’],
model_type=model[‘model_type’])
})
yield RunRequest(
run_key=run_key,
run_config=run_config)
Wonjae Lee
05/05/2023, 11:51 PM@job(
name=“deciduous_ml_job”,
description=“Run end to end deciduous ML job from raw data to model export”,
resource_defs={
“bigquery_client”: bigquery_resource,
“gcs_client”: gcs_resource,
“mongo_client”: mongodb_client.configured({
‘atlast_cluster’: “console”
})
},
config={
‘resources’: {
“bigquery_client” : {
“config”: {
“project” : “statsig-ingestions”
}
},
“gcs_client” : {
“config”: {
“project” : “statsig-ingestions”
}
},
},
}
)
def generate_model():
generate_model_version()
create_raw_data()
clean_data()
train_data, features = create_features()
train_model(train_data=train_data, features=features)
export_model()
export_evaluation(train_data=train_data, features=features)
download_model()
3 Views