https://dagster.io/ logo
Powered by
Title
v

Vax Thurai

09/28/2021, 4:55 AM
is there a way to pass the scheduled date into the dbt_cli_run function (see 
event_date
) 
dbt_run_models = dbt_cli_run.configured(
    name="dbt_run_models",
    config_or_config_fn={
        "project-dir": PROJECT_DIR,
        "profiles-dir": PROFILES_DIR,
        "models": MODELS,
        "target": TARGET,
        "vars": '{event_date: ' + event_date +'}'
    }
)
o

owen

09/28/2021, 4:32 PM
hi @Vax Thurai are you using a dagster schedule to run this solid? instead of passing in a dictionary config, you can pass in a config function that looks like 
config_or_config_fn=lambda event_date: {
        "project-dir": PROJECT_DIR,
        "profiles-dir": PROFILES_DIR,
        "models": MODELS,
        "target": TARGET,
        "vars": '{event_date: ' + event_date +'}'
    }
this will make a new configuration schema for your dbt solid that has just a single field for the event_date. then, in your schedule code (assuming the schedule knows what the event_date should be), you can return a run config that looks like : 
{ "solids": {"dbt_run_models": {"config": event_date}}}
alternatively, you could sip the config_or_config_function bit entirely, keep the original dbt_cli_run configuration schema, and have your schedule return a config blob with all of the relevant info ("project-dir", "profiles-dir", etc.).
this configuration stuff can be a bit tricky to wrap your head around so i'm happy to answer any followups. also, if you'd like to avoid a lot of this configuration strangeness, you can alternatively use the dbt_cli_resource instead of the pre-built solids.
v

Vax Thurai

09/28/2021, 4:38 PM
yea im using the daily scheduler decoration currently. atm i have the date being returned to the config for one of my earlier solids. 
@daily_schedule( # The decorated schedule function should accept a datetime
    pipeline_name="csg_pipeline",
    start_date=datetime(2021, 1, 1),
    execution_time=time(0, 5),
    execution_timezone="UTC",
    mode=csg_preset.mode,
    tags_fn_for_date=lambda _: csg_preset.tags 
)

def csg_daily_schedule(date):
    return {"solids": {"get_entity_configurations": {"config": {"date": date.strftime("%Y-%m-%d")}}}}
can i amend my daily scheduler method to be somethign like this: 
def csg_daily_schedule(date):
    return {"solids": {
        "get_entity_configurations": {
            "config": {
                "date": date.strftime("%Y-%m-%d")
            }
        },
        "dbt_run_models": {
            "config": {
                "event_date": date.strftime("%Y-%m-%d")
            }
        }
    }
}
o

owen

09/28/2021, 4:40 PM
basically, except it would be more like: 
def csg_daily_schedule(date):
    return {"solids": {
        "get_entity_configurations": {
            "config": {
                "date": date.strftime("%Y-%m-%d")
            }
        },
        "dbt_run_models": {
            "config": date.strftime("%Y-%m-%d")
        }
    }
}
v

Vax Thurai

09/28/2021, 4:41 PM
okay let me give it a try and ill get back to you
@owen so in dagit this is the error i get for the dbt_run_model 
dagster.core.errors.DagsterConfigMappingFunctionError: The config mapping function on a `configured` SolidDefinition has thrown an unexpected error during its execution.

  File "/Library/Python/3.8/site-packages/dagster/grpc/impl.py", line 342, in get_external_execution_plan_snapshot
    create_execution_plan(
  File "/Library/Python/3.8/site-packages/dagster/core/execution/api.py", line 718, in create_execution_plan
    resolved_run_config = ResolvedRunConfig.build(pipeline_def, run_config, mode=mode)
  File "/Library/Python/3.8/site-packages/dagster/core/system_config/objects.py", line 220, in build
    solid_config_dict = composite_descent(
  File "/Library/Python/3.8/site-packages/dagster/core/system_config/composite_descent.py", line 79, in composite_descent
    return {
  File "/Library/Python/3.8/site-packages/dagster/core/system_config/composite_descent.py", line 79, in <dictcomp>
    return {
  File "/Library/Python/3.8/site-packages/dagster/core/system_config/composite_descent.py", line 113, in _composite_descent
    config_mapped_solid_config = solid.definition.apply_config_mapping(
  File "/Library/Python/3.8/site-packages/dagster/core/definitions/configurable.py", line 55, in apply_config_mapping
    self.config_schema.resolve_config(config)
  File "/Library/Python/3.8/site-packages/dagster/core/definitions/definition_config_schema.py", line 120, in resolve_config
    self._invoke_user_config_fn(processed_config),
  File "/Library/Python/3.8/site-packages/dagster/core/definitions/definition_config_schema.py", line 113, in _invoke_user_config_fn
    return {"config": self._config_fn(processed_config.get("config", {}))}
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Library/Python/3.8/site-packages/dagster/core/errors.py", line 192, in user_code_error_boundary
    raise error_cls(
the error logs isnt clear what the issue is
o

owen

09/28/2021, 6:16 PM
hmm, strange that it doesn't show the actual error. what's the exact config function that you're passing into configured() / the exact run config you're supplying?
v

Vax Thurai

09/28/2021, 6:19 PM
basically the same as what you had above 
dbt_run_models = dbt_cli_run.configured(
    name="dbt_run_models",
    config_or_config_fn=lambda event_date: {
        "project-dir": PROJECT_DIR,
        "profiles-dir": PROFILES_DIR,
        "models": MODELS,
        "target": TARGET,
        "vars": '{event_date: ' + event_date +'}'
    }
)
PROJECT_DIR
, 
PROFILES_DIR
, 
MODELS
, 
TARGET
are assigned globally while event_date is passed through the scheduler
o

owen

09/28/2021, 6:24 PM
do you mind sending a screenshot of the error in dagit? generally these errors should re-raise the original user error. also if you're running dagit on your local machine it's possible that your terminal output might contain more info on the error
I figure that it's probably something simple but it's hard to debug that sort of thing without the source error, but it's possible that the framework is somehow eating it (which would be a bug)
v

Vax Thurai

09/28/2021, 6:29 PM
Screen Shot 2021-09-28 at 2.26.16 PM
✅ 1
o

owen

09/28/2021, 6:31 PM
gotcha that definitely seems like a bug -- I can create a ticket for that part
any luck on the terminal output?
v

Vax Thurai

09/28/2021, 6:32 PM
no because it fails even before running
this error appears right when i click Launch Execution
o

owen

09/28/2021, 6:35 PM
gotcha I'll try to reproduce this real quick :blob_smile_sweat2:
hm I get a different error, but one quick thing to try would be 
config_or_config_fn=lambda event_date: {
        "project-dir": PROJECT_DIR,
        "profiles-dir": PROFILES_DIR,
        "models": MODELS,
        "target": TARGET,
        "vars": {"event_date" : event_date}
    }
( I noticed that "vars" is meant to be a dictionary, not a string)
v

Vax Thurai

09/28/2021, 6:40 PM
yea its a dictionary. when i was manually testing the dbt code i wrapped it around quotes. let me try it as you have it though and see
yea that was it
o

owen

09/28/2021, 6:44 PM
nice
v

Vax Thurai

09/28/2021, 6:45 PM
its now working fine (atleast with the manual dagit runs). hopefully this works when its running through scheduler
thanks for the help!
o

owen

09/28/2021, 6:46 PM
no problem!
2 Views
#dagster-supportJoin Slack
Powered by