https://dagster.io/ logo
#ask-ai
Title
# ask-ai
v

Valdas Druskinis

03/13/2024, 9:49 AM
I need help with os.environ values when working with pytest. I have a few environ variables in init.py file, where they get picked up if pytest is executed. How would I mock them?
s

Scout

03/13/2024, 9:50 AM
When working with pytest and needing to mock environment variables that are accessed via
os.environ
, you can use the
monkeypatch
fixture provided by pytest. This fixture allows you to temporarily modify or add environment variables during your test functions, and it will ensure that the changes do not affect other tests or the system environment after the test finishes. Here's a step-by-step guide on how to use
monkeypatch
to mock environment variables: 1. Import the
monkeypatch
fixture in your test function by including it as an argument to the function. 2. Use the
monkeypatch.setenv
method to set the environment variable to the value you want for the test. 3. Write your test code that uses the environment variable. 4. Run your tests. The
monkeypatch
fixture will ensure that the environment variable is set for the duration of the test and then revert to its original state afterward. Here's an example of how you might use
monkeypatch
in a pytest test function:
Copy code
python def test_my_function(monkeypatch):     # Mock the environment variable     monkeypatch.setenv('MY_ENV_VAR', 'mock_value')      # Now, when your code accesses 'MY_ENV_VAR' via os.environ,     # it will see 'mock_value' instead of the actual value.     result = my_function_that_uses_env_var()      # Perform your test assertions     assert result == 'expected_result_based_on_mock_value'
In this example,
my_function_that_uses_env_var
would be the function you're testing that accesses the
MY_ENV_VAR
environment variable. The
monkeypatch.setenv
call ensures that within this test,
MY_ENV_VAR
will have the value
'mock_value'
. Remember that the mocking only applies to the test in which you call
monkeypatch.setenv
, and it will not affect other tests or the actual environment variables on your system.
v

Valdas Druskinis

03/13/2024, 11:30 AM
@Scout this is my case. And my pytest fails at os.environ["GOOGLE_APPLICATION_CREDENTIALS"]. HOw to avoid this?
s

Scout

03/13/2024, 11:30 AM
If your pytest is failing because it expects the
GOOGLE_APPLICATION_CREDENTIALS
environment variable to be set, you can use the
monkeypatch
fixture in pytest to mock this environment variable during your tests. Here's how you can do it: 1. Add the
monkeypatch
fixture to your test function's arguments. 2. Use
monkeypatch.setenv
to set the
GOOGLE_APPLICATION_CREDENTIALS
environment variable to a mock value that points to a valid credentials file for the duration of the test. 3. Ensure that you have a mock credentials file at the specified path, or your code is able to handle the mock path accordingly. Here's an example of how to mock the
GOOGLE_APPLICATION_CREDENTIALS
in a pytest test:
Copy code
python def test_my_google_function(monkeypatch, tmp_path):     # Create a temporary mock credentials file     mock_credentials = tmp_path / "mock_credentials.json"     mock_credentials.write_text('{"type": "service_account", "project_id": "my-mock-project"}')      # Set the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to the mock file     monkeypatch.setenv("GOOGLE_APPLICATION_CREDENTIALS", str(mock_credentials))      # Now, when your code accesses 'GOOGLE_APPLICATION_CREDENTIALS' via os.environ,     # it will see the path to the mock credentials file.     result = my_google_function()      # Perform your test assertions     assert result == 'expected_result'
In this example,
tmp_path
is a pytest fixture that provides a temporary directory unique to the test function. A mock credentials file is created in this directory, and the
GOOGLE_APPLICATION_CREDENTIALS
environment variable is set to point to this file. The
my_google_function
is the function you're testing that uses the
GOOGLE_APPLICATION_CREDENTIALS
environment variable. By using
monkeypatch.setenv
, the environment variable will only be set for the duration of the test, and it will not affect other tests or the actual environment on your system. Make sure that the mock credentials file you create is sufficient for your code to run without actually needing to authenticate with Google Cloud services.
v

Valdas Druskinis

03/13/2024, 11:30 AM
@Scout this is my case. # Deployment type deployment_name = os.getenv("DAGSTER_DEPLOYMENT", "prod") # Google credentials os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = os.getenv("GOOGLE_CREDS") defs = Definitions( assets=combined_assets, schedules=[bs_job_schedule], # asset_checks=qualitychecks_test, resources=resources[deployment_name], ). And my pytest fails at os.environ["GOOGLE_APPLICATION_CREDENTIALS"]. HOw to avoid this?
s

Scout

03/13/2024, 11:30 AM
In the case you've described, it looks like your code is setting the
GOOGLE_APPLICATION_CREDENTIALS
environment variable directly within the module's code. This can cause issues during testing because the
os.getenv("GOOGLE_CREDS")
call may return
None
if the
GOOGLE_CREDS
environment variable is not set, leading to a failure when trying to set
os.environ["GOOGLE_APPLICATION_CREDENTIALS"]
. To avoid this issue during testing with pytest, you can use the
monkeypatch
fixture to mock the
GOOGLE_CREDS
environment variable before the code that sets
GOOGLE_APPLICATION_CREDENTIALS
is executed. Here's how you can modify your test to handle this:
Copy code
python def test_my_dagster_deployment(monkeypatch, tmp_path):     # Create a temporary mock credentials file     mock_credentials = tmp_path / "mock_credentials.json"     mock_credentials.write_text('{"type": "service_account", "project_id": "my-mock-project"}')      # Mock the GOOGLE_CREDS environment variable to point to the mock file     monkeypatch.setenv("GOOGLE_CREDS", str(mock_credentials))      # Now, when your code accesses 'GOOGLE_CREDS' via os.getenv,     # it will receive the path to the mock credentials file.     # This will then set the GOOGLE_APPLICATION_CREDENTIALS environment variable correctly.     from my_module import defs  # Import here to ensure the environment variable is set first      # Perform your test logic and assertions     # ...
In this example, you create a mock credentials file and use
monkeypatch.setenv
to set the
GOOGLE_CREDS
environment variable to the path of this mock file. When your code runs, it will pick up this mock path and set the
GOOGLE_APPLICATION_CREDENTIALS
environment variable accordingly. Please note that you should import the module containing the
defs
variable inside the test function after setting the mock environment variable. This ensures that the environment variable is mocked before the code that depends on it is executed. By using this approach, you avoid the issue of
os.getenv("GOOGLE_CREDS")
returning
None
, and your tests should no longer fail due to the
GOOGLE_APPLICATION_CREDENTIALS
environment variable not being set.
v

Valdas Druskinis

03/13/2024, 11:33 AM
@Scout so lets say this is the test case that I would like to evaluate. How would a full example look like with monkeypatching? def test_generate_bs_assets_correct(): spec = {"table_name": "test_table", "destination": "test_dataset.test_table"} load_table_asset, save_data_bq_asset = generate_bs_assets(spec) assert ( load_table_asset.asset_and_check_keys_by_output_name["result"][0][0] == f"load_data_{spec['table_name']}" ) assert ( save_data_bq_asset.asset_and_check_keys_by_output_name["result"][0][0] == f"bs_save_data_bq_{spec['table_name']}" )
s

Scout

03/13/2024, 11:33 AM
To fully mock the environment variables in your test case using
monkeypatch
, you would need to do the following: 1. Use the
monkeypatch
fixture to set the environment variables that are expected by your code. 2. Import the code that uses these environment variables within the test function after setting the mocks to ensure they are applied correctly. 3. Run your assertions as normal. Here's how you could write your test case with `monkeypatch`:
Copy code
python
def test_generate_bs_assets_correct(monkeypatch, tmp_path):
    # Mock the environment variables that your code expects
    # For example, if your code requires 'DAGSTER_DEPLOYMENT' and 'GOOGLE_CREDS'
    monkeypatch.setenv("DAGSTER_DEPLOYMENT", "test")
    
    # Create a temporary mock credentials file for 'GOOGLE_CREDS'
    mock_credentials = tmp_path / "mock_credentials.json"
    mock_credentials.write_text('{"type": "service_account", "project_id": "my-mock-project"}')
    
    # Set the 'GOOGLE_CREDS' environment variable to point to the mock file
    monkeypatch.setenv("GOOGLE_CREDS", str(mock_credentials))

    # Import the module/function after setting the environment variables
    from my_module import generate_bs_assets

    # Call the function you want to test
    spec = {"table_name": "test_table", "destination": "test_dataset.test_table"}
    load_table_asset, save_data_bq_asset = generate_bs_assets(spec)

    # Perform your test assertions
    assert (
        load_table_asset.asset_and_check_keys_by_output_name["result"][0][0]
        == f"load_data_{spec['table_name']}"
    )
    assert (
        save_data_bq_asset.asset_and_check_keys_by_output_name["result"][0][0]
        == f"bs_save_data_bq_{spec['table_name']}"
    )
In this example,
monkeypatch.setenv
is used to set both
DAGSTER_DEPLOYMENT
and
GOOGLE_CREDS
environment variables before the
generate_bs_assets
function is imported and executed. This ensures that when the code within
generate_bs_assets
runs, it uses the mocked environment variables. Remember to replace
"my_module"
with the actual name of the module where
generate_bs_assets
is defined. Also, ensure that the
tmp_path
fixture is used to create a temporary directory for the mock credentials file, which avoids any side effects on the actual file system. By following this pattern, you can isolate your tests from external dependencies such as environment variables and ensure that they are repeatable and reliable. Asset Checks on DBT Assets button Error on Asset Dependency with Multiple ... button Asset checks: severity and blocking mate... button Asset checks | Dagster button Thank you for your feedback!
14 Views