AirFlow : A 5 step guide

Image for post
Image for post
Image for post
Image for post
5 Steps for airflow
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
from datetime import timedeltaimport airflowfrom airflow import DAGfrom airflow.operators.bash_operator import BashOperator# These args will get passed on to each operator# You can override them on a per-task basis during operator initializationdefault_args = {'owner': 'airflow','depends_on_past': False,'start_date': airflow.utils.dates.days_ago(2),'email': ['airflow@example.com'],'email_on_failure': False,'email_on_retry': False,'retries': 1,'retry_delay': timedelta(minutes=5),}dag = DAG('tutorial',default_args=default_args,description='A simple tutorial DAG',schedule_interval=timedelta(days=1),)# t1, t2 and t3 are examples of tasks created by instantiating operatorst1 = BashOperator(task_id='print_date',bash_command='date',dag=dag,)t1.doc_md = """\#### Task DocumentationYou can document your task using the attributes `doc_md` (markdown),`doc` (plain text), `doc_rst`, `doc_json`, `doc_yaml` which getsrendered in the UI's Task Instance Details page.![img](http://montcs.bloomu.edu/~bobmon/Semesters/2012-01/491/import%20soul.png)"""dag.doc_md = __doc__t2 = BashOperator(task_id='sleep',depends_on_past=False,bash_command='sleep 5',dag=dag,)templated_command = """{% for i in range(5) %}echo "{{ ds }}"echo "{{ macros.ds_add(ds, 7)}}"echo "{{ params.my_param }}"{% endfor %}"""t3 = BashOperator(task_id='templated',depends_on_past=False,bash_command=templated_command,params={'my_param': 'Parameter I passed in'},dag=dag,)t1 >> [t2, t3]

Turn On the chain by toggling the on button

It will display the tasks in graph view

Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store