https://dagster.io/ logo
Join Slack
Powered by
When ingesting an Airflow DAG into Dagster, what’s...
# ask-community
d
When ingesting an Airflow DAG into Dagster, what’s the best practice for dealing with connections the Airflow DAG used to use? For example I was using Fivetran and dbt Cloud connections… is a connections.json expected to be somewhere when developing locally? What do you do with Dagster Cloud for the same purpose?
d
@Joe is the resident expert on Dagster <> Airflow, I'm sure he can get back to you tomorrow morning.
🙏 1
d
I’m also just interested in how to deal with environment variables in general with Dagster. I installed the cli to see if I could deal with them here but even though the pip install worked fine it doesn’t run.
d
Hi David, I can speak to the environment variables/secrets question. In Dagster Cloud Serverless you can include them in our GitHub action or CLI as described here: https://docs.dagster.io/dagster-cloud/deployment/serverless#adding-secrets - those would also then be included in any branch deployments. In open source Dagster the answer depends a bit on where your runs are being launched - for example if you're on Kubernetes you can specify a list of k8s secrets or key-value pairs for different jobs in our Helm chart, and in ECS there's an AWS secretsmanager integration. If you're just running Dagster locally for development, the runs are all happening in subprocesses in your Python environment so the easiest way may be to ensure that the environment variables you need are available in the environment where Dagit is running.
j
what’s the best practice for dealing with connections the Airflow DAG used to use?
For envs/secrets I'd recommend following daniel's advice. You probably noticed but 
airflow_operator_to_op
is only expecting instantiations of airflow connector classes and how you manage the state captured in those is up to you. Having a py file with the connections defined and envs/secrets for sensitive state, should work fairly well.
d
The CLI is the most attractive way to me, but for some reason I can’t get it to run.
d
When you say the CLI - which command are you referring to specifically?
d
just any dagster-cloud command
d
Got it - does it give you an error message?
d
/Library/Developer/CommandLineTools/usr/bin/python3: No module named dagster_cloud.__main__; 'dagster_cloud' is a package and cannot be directly executed
when I ran 
python3 -m dagster_cloud --version
/Library/Developer/CommandLineTools/usr/bin/python3: No module named dagster-cloud
d
Ah try
Copy code
python3 -m dagster-cloud --version
(dash not underscore)
d
yeah I tried that first
and got the second output above
d
I see - and this was after running 
pip install dagster-cloud
in your Python environment?
d
when I tried the pip install it seemed to work fine
but I do get this note at the start: 
Defaulting to user installation because normal site-packages is not writeable
d
Hmmm what if you do 
python3 -m pip install dagster-cloud
d
that’s exactly what I ran
it output:
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: dagster-cloud in ./Library/Python/3.8/lib/python/site-packages (1.0.9)
Requirement already satisfied: questionary in ./Library/Python/3.8/lib/python/site-packages (from dagster-cloud) (1.10.0)
Requirement already satisfied: dagster-cloud-cli==1.0.9 in ./Library/Python/3.8/lib/python/site-packages (from dagster-cloud) (1.0.9)
Requirement already satisfied: requests in ./Library/Python/3.8/lib/python/site-packages (from dagster-cloud) (2.28.1)
Requirement already satisfied: typer[all] in ./Library/Python/3.8/lib/python/site-packages (from dagster-cloud) (0.6.1)
Requirement already satisfied: dagster==1.0.9 in ./Library/Python/3.8/lib/python/site-packages (from dagster-cloud) (1.0.9)
Requirement already satisfied: croniter>=0.3.34 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (1.3.7)
Requirement already satisfied: toposort>=1.0 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (1.7)
Requirement already satisfied: coloredlogs<=14.0,>=6.1 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (14.0)
Requirement already satisfied: sqlalchemy>=1.0 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (1.4.41)
Requirement already satisfied: watchdog>=0.8.3 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (2.1.9)
Requirement already satisfied: click>=5.0 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (8.1.3)
Requirement already satisfied: grpcio<1.48.1,>=1.32.0 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (1.47.0)
Requirement already satisfied: pytz in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (2022.2.1)
Requirement already satisfied: grpcio-health-checking<1.44.0,>=1.32.0 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (1.43.0)
Requirement already satisfied: alembic!=1.6.3,<1.7.0,>=1.2.1 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (1.6.5)
Requirement already satisfied: setuptools in /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages (from dagster==1.0.9->dagster-cloud) (49.2.1)
Requirement already satisfied: tabulate in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (0.8.10)
Requirement already satisfied: PyYAML>=5.1 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (6.0)
Requirement already satisfied: docstring-parser in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (0.15)
Requirement already satisfied: rx<2,>=1.6 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (1.6.1)
Requirement already satisfied: tqdm in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (4.64.1)
Requirement already satisfied: typing-compat in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (0.1.0)
Requirement already satisfied: protobuf<4,>=3.13.0 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (3.20.2)
Requirement already satisfied: pendulum in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (2.1.2)
Requirement already satisfied: Jinja2 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (3.1.2)
Requirement already satisfied: typing-extensions>=4.0.1 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (4.3.0)
Requirement already satisfied: python-dateutil in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (2.8.2)
Requirement already satisfied: packaging>=20.9 in ./Library/Python/3.8/lib/python/site-packages (from dagster==1.0.9->dagster-cloud) (21.3)
Requirement already satisfied: prompt_toolkit<4.0,>=2.0 in ./Library/Python/3.8/lib/python/site-packages (from questionary->dagster-cloud) (3.0.31)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./Library/Python/3.8/lib/python/site-packages (from requests->dagster-cloud) (1.26.12)
Requirement already satisfied: idna<4,>=2.5 in ./Library/Python/3.8/lib/python/site-packages (from requests->dagster-cloud) (3.4)
Requirement already satisfied: charset-normalizer<3,>=2 in ./Library/Python/3.8/lib/python/site-packages (from requests->dagster-cloud) (2.1.1)
Requirement already satisfied: certifi>=2017.4.17 in ./Library/Python/3.8/lib/python/site-packages (from requests->dagster-cloud) (2022.9.14)
Requirement already satisfied: shellingham<2.0.0,>=1.3.0 in ./Library/Python/3.8/lib/python/site-packages (from typer[all]->dagster-cloud) (1.5.0)
Requirement already satisfied: colorama<0.5.0,>=0.4.3 in ./Library/Python/3.8/lib/python/site-packages (from typer[all]->dagster-cloud) (0.4.5)
Requirement already satisfied: rich<13.0.0,>=10.11.0 in ./Library/Python/3.8/lib/python/site-packages (from typer[all]->dagster-cloud) (12.5.1)
Requirement already satisfied: Mako in ./Library/Python/3.8/lib/python/site-packages (from alembic!=1.6.3,<1.7.0,>=1.2.1->dagster==1.0.9->dagster-cloud) (1.2.2)
Requirement already satisfied: python-editor>=0.3 in ./Library/Python/3.8/lib/python/site-packages (from alembic!=1.6.3,<1.7.0,>=1.2.1->dagster==1.0.9->dagster-cloud) (1.0.4)
Requirement already satisfied: humanfriendly>=7.1 in ./Library/Python/3.8/lib/python/site-packages (from coloredlogs<=14.0,>=6.1->dagster==1.0.9->dagster-cloud) (10.0)
Requirement already satisfied: six>=1.5.2 in /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages (from grpcio<1.48.1,>=1.32.0->dagster==1.0.9->dagster-cloud) (1.15.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./Library/Python/3.8/lib/python/site-packages (from packaging>=20.9->dagster==1.0.9->dagster-cloud) (3.0.9)
Requirement already satisfied: wcwidth in ./Library/Python/3.8/lib/python/site-packages (from prompt_toolkit<4.0,>=2.0->questionary->dagster-cloud) (0.2.5)
Requirement already satisfied: commonmark<0.10.0,>=0.9.0 in ./Library/Python/3.8/lib/python/site-packages (from rich<13.0.0,>=10.11.0->typer[all]->dagster-cloud) (0.9.1)
Requirement already satisfied: pygments<3.0.0,>=2.6.0 in ./Library/Python/3.8/lib/python/site-packages (from rich<13.0.0,>=10.11.0->typer[all]->dagster-cloud) (2.13.0)
Requirement already satisfied: MarkupSafe>=2.0 in ./Library/Python/3.8/lib/python/site-packages (from Jinja2->dagster==1.0.9->dagster-cloud) (2.1.1)
Requirement already satisfied: pytzdata>=2020.1 in ./Library/Python/3.8/lib/python/site-packages (from pendulum->dagster==1.0.9->dagster-cloud) (2020.1)
d
Got it - is using a Python venv environment an option here? I think you have a couple of different installations of Python installed (Library/Developer/CommandLineTools/usr/bin/python3) and (Library/Python/3.8/lib/python) and they're getting each other confused. What i've done before to get that setup on my mac:
Copy code
brew install pyenv pyenv-virtualenv
Then add the following commands to your shell profile:
Copy code
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
and finally create and activate the virtualenv:
Copy code
pyenv install 3.9.11
pyenv virtualenv 3.9.11 dagster39
pyenv activate dagster39
Then there's an isolated environment where pip and python always point to the same place
👀 1
d
Thanks Daniel this is working
🎉 1
Is there any way for dagster local to use secrets stored via dagster cloud CLI so that you have consistent secrets management? It would be cool if you could do something like {{dagster_cloud.this_environment_variable}} in your code.
d
There isn't today, but totally agree - having a native secrets manager API that you can call within code is one of the next things we want to do
d
Yeah one thing that worked just fine in Airflow was the connections manager
if you had a way to have something similar that spans all environments, Dagster would surpass Airflow in this regard
3 Views
Open in Slack
PreviousNext
Powered by