https://dagster.io/ logo
#ask-community
Title
# ask-community
d

David Jayatillake

09/19/2022, 12:28 AM
When ingesting an Airflow DAG into Dagster, what’s the best practice for dealing with connections the Airflow DAG used to use? For example I was using Fivetran and dbt Cloud connections… is a connections.json expected to be somewhere when developing locally? What do you do with Dagster Cloud for the same purpose?
d

Dagster Jarred

09/19/2022, 2:13 AM
@Joe is the resident expert on Dagster <> Airflow, I'm sure he can get back to you tomorrow morning.
🙏 1
d

David Jayatillake

09/19/2022, 11:24 AM
I’m also just interested in how to deal with environment variables in general with Dagster. I installed the cli to see if I could deal with them here but even though the pip install worked fine it doesn’t run.
d

daniel

09/19/2022, 12:14 PM
Hi David, I can speak to the environment variables/secrets question. In Dagster Cloud Serverless you can include them in our GitHub action or CLI as described here: https://docs.dagster.io/dagster-cloud/deployment/serverless#adding-secrets - those would also then be included in any branch deployments. In open source Dagster the answer depends a bit on where your runs are being launched - for example if you're on Kubernetes you can specify a list of k8s secrets or key-value pairs for different jobs in our Helm chart, and in ECS there's an AWS secretsmanager integration. If you're just running Dagster locally for development, the runs are all happening in subprocesses in your Python environment so the easiest way may be to ensure that the environment variables you need are available in the environment where Dagit is running.
j

Joe

09/19/2022, 1:07 PM
what’s the best practice for dealing with connections the Airflow DAG used to use?
For envs/secrets I'd recommend following daniel's advice. You probably noticed but
airflow_operator_to_op
is only expecting instantiations of airflow connector classes and how you manage the state captured in those is up to you. Having a py file with the connections defined and envs/secrets for sensitive state, should work fairly well.
d

David Jayatillake

09/19/2022, 3:56 PM
The CLI is the most attractive way to me, but for some reason I can’t get it to run.
d

daniel

09/19/2022, 3:57 PM
When you say the CLI - which command are you referring to specifically?
d

David Jayatillake

09/19/2022, 3:57 PM
just any dagster-cloud command
d

daniel

09/19/2022, 3:57 PM
Got it - does it give you an error message?
d

David Jayatillake

09/19/2022, 3:58 PM
/Library/Developer/CommandLineTools/usr/bin/python3: No module named dagster_cloud.__main__; 'dagster_cloud' is a package and cannot be directly executed
when I ran
python3 -m dagster_cloud --version
/Library/Developer/CommandLineTools/usr/bin/python3: No module named dagster-cloud
d

daniel

09/19/2022, 3:59 PM
Ah try
Copy code
python3 -m dagster-cloud --version
(dash not underscore)
d

David Jayatillake

09/19/2022, 3:59 PM
yeah I tried that first
and got the second output above
d

daniel

09/19/2022, 4:00 PM
I see - and this was after running
pip install dagster-cloud
in your Python environment?
d

David Jayatillake

09/19/2022, 4:00 PM
when I tried the pip install it seemed to work fine
but I do get this note at the start:
Defaulting to user installation because normal site-packages is not writeable
d

daniel

09/19/2022, 4:01 PM
Hmmm what if you do
python3 -m pip install dagster-cloud
d

David Jayatillake

09/19/2022, 4:01 PM
that’s exactly what I ran
it output:
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: dagster-cloud in ./Library/Python/3.8/lib/python/site-packages (1.0.9)
Requirement already satisfied: questionary in ./Library/Python/3.8/lib/python/site-packages (from dagster-cloud) (1.10.0)
Requirement already satisfied: dagster-cloud-cli==1.0.9 in ./Library/Python/3.8/lib/python/site-packages (from dagster-cloud) (1.0.9)
Requirement already satisfied: requests in ./Library/Python/3.8/lib/python/site-packages (from dagster-cloud) (2.28.1)
Requirement already satisfied: typer[all] in ./Library/Python/3.8/lib/python/site-packages (from dagster-cloud) (0.6.1)
Requirement already satisfied: dagster==1.0.9 in ./Library/Python/3.8/lib/python/site-packages (from dagster-cloud) (1.0.9)
Requirement already satisfied: croniter>=0.3.34 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (1.3.7)
Requirement already satisfied: toposort>=1.0 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (1.7)
Requirement already satisfied: coloredlogs<=14.0,>=6.1 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (14.0)
Requirement already satisfied: sqlalchemy>=1.0 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (1.4.41)
Requirement already satisfied: watchdog>=0.8.3 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (2.1.9)
Requirement already satisfied: click>=5.0 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (8.1.3)
Requirement already satisfied: grpcio<1.48.1,>=1.32.0 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (1.47.0)
Requirement already satisfied: pytz in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (2022.2.1)
Requirement already satisfied: grpcio-health-checking<1.44.0,>=1.32.0 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (1.43.0)
Requirement already satisfied: alembic!=1.6.3,<1.7.0,>=1.2.1 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (1.6.5)
Requirement already satisfied: setuptools in /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages (from dagster==1.0.9->dagster-cloud) (49.2.1)
Requirement already satisfied: tabulate in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (0.8.10)
Requirement already satisfied: PyYAML>=5.1 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (6.0)
Requirement already satisfied: docstring-parser in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (0.15)
Requirement already satisfied: rx<2,>=1.6 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (1.6.1)
Requirement already satisfied: tqdm in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (4.64.1)
Requirement already satisfied: typing-compat in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (0.1.0)
Requirement already satisfied: protobuf<4,>=3.13.0 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (3.20.2)
Requirement already satisfied: pendulum in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (2.1.2)
Requirement already satisfied: Jinja2 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (3.1.2)
Requirement already satisfied: typing-extensions>=4.0.1 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (4.3.0)
Requirement already satisfied: python-dateutil in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (2.8.2)
Requirement already satisfied: packaging>=20.9 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (21.3)
Requirement already satisfied: prompt_toolkit<4.0,>=2.0 in ./Library/Python/3.8/lib/python/site-packages (from questionary->dagster-cloud) (3.0.31)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./Library/Python/3.8/lib/python/site-packages (from requests->dagster-cloud) (1.26.12)
Requirement already satisfied: idna<4,>=2.5 in ./Library/Python/3.8/lib/python/site-packages (from requests->dagster-cloud) (3.4)
Requirement already satisfied: charset-normalizer<3,>=2 in ./Library/Python/3.8/lib/python/site-packages (from requests->dagster-cloud) (2.1.1)
Requirement already satisfied: certifi>=2017.4.17 in ./Library/Python/3.8/lib/python/site-packages (from requests->dagster-cloud) (2022.9.14)
Requirement already satisfied: shellingham<2.0.0,>=1.3.0 in ./Library/Python/3.8/lib/python/site-packages (from typer[all]->dagster-cloud) (1.5.0)
Requirement already satisfied: colorama<0.5.0,>=0.4.3 in ./Library/Python/3.8/lib/python/site-packages (from typer[all]->dagster-cloud) (0.4.5)
Requirement already satisfied: rich<13.0.0,>=10.11.0 in ./Library/Python/3.8/lib/python/site-packages (from typer[all]->dagster-cloud) (12.5.1)
Requirement already satisfied: Mako in ./Library/Python/3.8/lib/python/site-packages (from alembic!=1.6.3,<1.7.0,>=1.2.1->dagster==1.0.9->dagster-cloud) (1.2.2)
Requirement already satisfied: python-editor>=0.3 in ./Library/Python/3.8/lib/python/site-packages (from alembic!=1.6.3,<1.7.0,>=1.2.1->dagster==1.0.9->dagster-cloud) (1.0.4)
Requirement already satisfied: humanfriendly>=7.1 in ./Library/Python/3.8/lib/python/site-packages (from coloredlogs<=14.0,>=6.1->dagster==1.0.9->dagster-cloud) (10.0)
Requirement already satisfied: six>=1.5.2 in /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages (from grpcio<1.48.1,>=1.32.0->dagster==1.0.9->dagster-cloud) (1.15.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./Library/Python/3.8/lib/python/site-packages (from packaging>=20.9->dagster==1.0.9->dagster-cloud) (3.0.9)
Requirement already satisfied: wcwidth in ./Library/Python/3.8/lib/python/site-packages (from prompt_toolkit<4.0,>=2.0->questionary->dagster-cloud) (0.2.5)
Requirement already satisfied: commonmark<0.10.0,>=0.9.0 in ./Library/Python/3.8/lib/python/site-packages (from rich<13.0.0,>=10.11.0->typer[all]->dagster-cloud) (0.9.1)
Requirement already satisfied: pygments<3.0.0,>=2.6.0 in ./Library/Python/3.8/lib/python/site-packages (from rich<13.0.0,>=10.11.0->typer[all]->dagster-cloud) (2.13.0)
Requirement already satisfied: MarkupSafe>=2.0 in ./Library/Python/3.8/lib/python/site-packages (from Jinja2->dagster==1.0.9->dagster-cloud) (2.1.1)
Requirement already satisfied: pytzdata>=2020.1 in ./Library/Python/3.8/lib/python/site-packages (from pendulum->dagster==1.0.9->dagster-cloud) (2020.1)
d

daniel

09/19/2022, 4:12 PM
Got it - is using a Python venv environment an option here? I think you have a couple of different installations of Python installed (Library/Developer/CommandLineTools/usr/bin/python3) and (Library/Python/3.8/lib/python) and they're getting each other confused. What i've done before to get that setup on my mac:
Copy code
brew install pyenv pyenv-virtualenv
Then add the following commands to your shell profile:
Copy code
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
and finally create and activate the virtualenv:
Copy code
pyenv install 3.9.11
pyenv virtualenv 3.9.11 dagster39
pyenv activate dagster39
Then there's an isolated environment where pip and python always point to the same place
👀 1
d

David Jayatillake

09/21/2022, 9:41 PM
Thanks Daniel this is working
🎉 1
Is there any way for dagster local to use secrets stored via dagster cloud CLI so that you have consistent secrets management? It would be cool if you could do something like {{dagster_cloud.this_environment_variable}} in your code.
d

daniel

09/22/2022, 9:34 PM
There isn't today, but totally agree - having a native secrets manager API that you can call within code is one of the next things we want to do
d

David Jayatillake

09/24/2022, 8:50 PM
Yeah one thing that worked just fine in Airflow was the connections manager
if you had a way to have something similar that spans all environments, Dagster would surpass Airflow in this regard
3 Views