https://dagster.io/ logo
#ask-community
Title
# ask-community
a

Andres Crucetta

04/12/2023, 3:12 PM
dagster._core.errors.DagsterInvalidConfigError: Error in config for resources E Error 1: Post processing at path rootdb connconfig:user of original value {'env': 'SNOWFLAKE_USER'} failed: E dagster._config.errors.PostProcessingError: You have attempted to fetch the environment variable "SNOWFLAKE_USER" which is not set. In order for this execution to succeed it must be set in this environment. What's the best way to access resources? I'm conflicted on whether to use EnvVar or os.getenv(). I have .env set-up in my root directory but somehow it's unable to access it using the Pythonic config/resources. Code in thread.
Copy code
import pytest
from dagster import EnvVar, build_op_context

from orchestrator import DatabaseResource
from orchestrator.ops import load_sftp_to_snowflake, DataTransferConfig


@pytest.fixture
def data_config():
    """
    Op configuration parameters.
    """
    return DataTransferConfig(
        raw_table="RAW_CERPASS_MAC",
        azure_stage="STAGE_CERPASS_MAC",
        file_format="PLAIN_TEXT"
    )


@pytest.fixture
def db_resource():
    """
    Snowflake database resource.
    """
    return DatabaseResource(
        user=EnvVar("SNOWFLAKE_USER"),
        account=EnvVar("SNOWFLAKE_ACCOUNT"),
        password=EnvVar("SNOWFLAKE_PASSWORD"),
        warehouse=EnvVar("SNOWFLAKE_WAREHOUSE_NAME"),
    )


def test_load_sftp_to_snowflake(data_config: DataTransferConfig,
                                db_resource: DatabaseResource):
    """
    Test the copy into op.
    """
    context = build_op_context()
    result = load_sftp_to_snowflake(context, config=data_config, db_conn=db_resource)
    assert result is True
Copy code
defs = Definitions(
    sensors=all_sensors,
    schedules=[upload_data_cerpass_mac],
    jobs=BindResourcesToJobs([upload_data]),
    resources={
        "db_conn": DatabaseResource(
            account=EnvVar("SNOWFLAKE_ACCOUNT"),
            user=EnvVar("SNOWFLAKE_USER"),
            password=EnvVar("SNOWFLAKE_PASSWORD"),
            warehouse=EnvVar("SNOWFLAKE_WAREHOUSE_NAME"),
        )
    },
)
c

Charlie Bini

04/12/2023, 8:10 PM
yeah EnvVar isn't working for me either. It just literally plugs in the variable name. I think it's not implemented yet?
f

Frederik Löw

04/25/2023, 1:02 PM
It is the same for me.
19 Views