https://dagster.io/ logo
Powered by
r

raaid

02/24/2021, 2:10 PM
Hello Dagster folks! My team is building out our first bigger piece of software that uses Dagster (loving it so far). I have a question that involves modes and resources vs. configs. We have defined modes for “prod”, “dev”, “local”, and “test”, and have defined resources to use in each of these. We are now adding a solid that needs to know what environment we’re in, so for the “prod” mode we want a “prod” environment, but for “dev” and “local” we want “dev” as the env. Is it antipattern to simply add a resource that is a string for “prod” vs “dev”? Is this a case of using configs instead? If it is, is there a common pattern for tying it in with our modes? Thanks!
Would presets be the right thing to use here?
d

dhume

02/24/2021, 3:18 PM
We have been handling this with presets and have at least one preset correspond for each mode
r

raaid

02/24/2021, 3:34 PM
awesome, that makes sense. Thank you 🙂
If you have any publicly available code as an example, that’d be appreciated btw! If not no worries, should be able to figure it out
d

dhume

02/24/2021, 3:36 PM
As it so happens I’m working on a blog post about this. Hopefully that will be up soon
But I can put together a gist in the meantime
So if you had two modes like 
local
and 
dev
you could set your modes like this
Copy code
local = ModeDefinition(
    name="local",
    resource_defs={
        "data_lake": mock_s3,
        "snowflake": mock_snowflake,
        "ge_data_context": ge_data_context,
    },
    description="Local mode of pipelines (No AWS, Dev/Local Resources)",
)

dev = ModeDefinition(
    name="dev",
    description="Dev mode of pipelines (Dev AWS, Dev Resources)",
    intermediate_storage_defs=s3_plus_default_intermediate_storage_defs,
    resource_defs={
        "s3": s3_resource,
        "data_lake": s3,
        "snowflake": mock_snowflake,
        "ge_data_context": ge_data_context,
    },
)
Then have different presets dev preset
Copy code
dev_run_config = {
    "resources": {
        "snowflake": {
            "config": {"file_path": file_relative_path(__file__, "data")}
        },
        "data_lake": {"config": {"bucket": DAGSTER_BUCKET_DEV}},
        "ge_data_context": {
            "config": {
                "ge_root_dir": file_relative_path(
                    __file__, "great_expectations"
                )
            }
        },
    },
    "solids": {
        "store_order_items": {
            "inputs": {"query": {"value": "xxx.csv"}}
        }
    },
    "intermediate_storage": {
        "s3": {
            "config": {
                "s3_bucket": DAGSTER_BUCKET_DEV,
                "s3_prefix": S3_PREFIX,
            }
        }
    },
}
local preset
Copy code
local_run_config = {
    "resources": {
        "snowflake": {
            "config": {"file_path": file_relative_path(__file__, "data")}
        },
        "ge_data_context": {
            "config": {
                "ge_root_dir": file_relative_path(
                    __file__, "great_expectations"
                )
            }
        },
    },
    "solids": {
        "store_order_items": {
            "inputs": {"query": {"value": "xxx.csv"}}
        }
    },
}
amaze 1
👍 1
r

raaid

02/24/2021, 4:01 PM
Amazing wow thank you so much for such a thorough example!!
👍 2
c

Charles Lariviere

03/01/2021, 8:17 PM
Hey @dhume, I’m just seeing this — this is super helpful! I’m curious on your 
mock_snowflake
resource for local/dev; are you simply wrapping a 
LIMIT
statement around queries, or are you completely abstracting from making requests to Snowflake? I’m currently working on setting up unit tests and would ideally like to fully abstract from making requests. 🤔
d

dhume

03/01/2021, 8:26 PM
For local, our 
mock_snowflake
resource just reads files that contain results equivalent to the query responses. So people can run tests and make changes all they want without actually pinging Snowflake. In dev we have an actual snowflake connection and query usually with a LIMIT or targeting a staging database and in prod we have the query no LIMIT
c

Charles Lariviere

03/01/2021, 8:29 PM
Awesome, that’s really helpful — thanks! 🙏
3 Views
Powered by