Melle Minderhoud
03/11/2021, 4:29 PM'''
The airflow DAG scaffold for airflow_tests.airflow.hello_cereal_pipeline
Note that this docstring must contain the strings "airflow" and "DAG" for
Airflow to properly detect it as a DAG
See: <http://bit.ly/307VMum>
'''
import datetime
import yaml
from dagster_airflow.factory import make_airflow_dag_containerized
################################################################################
# #
# # This environment is auto-generated from your configs and/or presets
# #
################################################################################
ENVIRONMENT = '''
intermediate_storage:
filesystem:
config:
base_dir: /tmp/dagster-airflow/hello_cereal_pipeline
'''
################################################################################
# #
# # NOTE: these arguments should be edited for your environment
# #
################################################################################
DEFAULT_ARGS = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime.datetime(2021, 3, 10),
'email': ['<mailto:airflow@example.com|airflow@example.com>'],
'email_on_failure': False,
'email_on_retry': False,
'auto_remove': False
}
dag, tasks = make_airflow_dag_containerized(
# NOTE: you must ensure that airflow_tests.airflow is
# installed or available on sys.path, otherwise, this import will fail.
module_name='airflow_test.airflow',
pipeline_name='hello_cereal_pipeline',
image = 'dagster-docker-test-cereal',
run_config=yaml.safe_load(ENVIRONMENT),
dag_kwargs={'default_args': DEFAULT_ARGS, 'max_active_runs': 1}
)
And the pipeline is defined as:
from dagster import pipeline, solid
@solid
def hello_cereal(context):
print('Hallo')
print("finished")
<http://context.log.info|context.log.info>('finished')
@pipeline
def hello_cereal_pipeline():
hello_cereal()
ps. The docs are refering to 'dagster_graphql', but I assume it should be dagster-graphql?yuhan
03/11/2021, 7:37 PMdagster-graphql
- on a side note, dagster_graphql
also works because pip would convert underscores to dashes.dagster-graphql
may not be required anymore.