https://dagster.io/ logo
Powered by
#random
Title
# random
k

Kyle Hamlin

03/03/2022, 3:39 PM
What is the best way to start porting Airflow DAGs from airflow to dagster? I’ve been trying to use this method without much success https://docs.dagster.io/_apidocs/libraries/dagster-airflow#dagster_airflow.make_dagster_repo_from_airflow_dags_path Getting errors like this:
Copy code
[2022-03-03 08:37:40,862] {api.py:539} INFO - Started Dagster code server for file data_eng/jobs/make_dagster_pipeline.py in process 87160
/Users/khamlin/.pyenv/versions/3.8.8/lib/python3.8/site-packages/dagster/core/workspace/context.py:556: UserWarning: Error loading repository location make_dagster_pipeline.py:dagster.core.errors.DagsterInvariantViolationError: No jobs, pipelines, graphs, asset collections, or repositories found in "make_dagster_pipeline".
a

alex

03/03/2022, 3:43 PM
are you running 
make_dagster_repo_from_airflow_dags_path
in 
make_dagster_pipeline.py
at module scope? are you specifying the attribute name the repository object should be found at within file in the cli args? the issue may be with our more lenient “auto discover” code, so the advice above may help you
k

Kyle Hamlin

03/03/2022, 3:53 PM
Yeah so I just auto generated a very basic workspace setup with one repo called data_eng and within that repo there is a job called 
make_dagster_pipeline.py
with the following code:
Copy code
from dagster_airflow.dagster_pipeline_factory import make_dagster_repo_from_airflow_dags_path

def make_repo_from_dir():
    return make_dagster_repo_from_airflow_dags_path(
        "/<path-to-airflow>/airflow/dags/", "airflow_dags"
    )
and basically I’ve been trying to run that job file with the dagit cli: 
dagit -f data_eng/jobs/make_dagster_pipeline.py
So do you mean the target repository or the repo the job is running from?
Not sure what the relation is here but that -n flag doesn’t exist anymore from the docs
Copy code
dagit -f path/to/make_dagster_repo.py -n make_repo_from_dir
a

alex

03/03/2022, 4:02 PM
dagit -f path/to/make_dagster_repo.py -a make_repo_from_dir
k

Kyle Hamlin

03/03/2022, 4:05 PM
got it so that runs but nothing really happens 😅
nvm just had the wrong path
a

alex

03/03/2022, 4:09 PM
whew
k

Kyle Hamlin

03/03/2022, 4:10 PM
loading dags now… all the modue importing is failing on the airflow side but its finding dags
a

alex

03/03/2022, 4:12 PM
all the modue importing is failing
if its due to relative imports you may need to change the way you are loading in to dagit. Check 
dagit --help
, consider module loading or setting working directory
k

Kyle Hamlin

03/03/2022, 4:17 PM
yeah it definitely the relative imports
Powered by