General Info

$AIRFLOW_HOME is the path to your airflow installation folder.

Links to go through, before working on Airflow

• Getting started tutorial
  → Going from start to finish on creating a dag.
• UI / Screenshots
  → Look and feel of Airflow.
• Concepts
  → Get an idea of what airflow is used for and things you can do with it.
• Scheduling and Triggers
  → How scheduling and triggering works in Airflow.
• View logs of task
  → How to view the output of your dags.

Basic Airflow tasks:

Index

• Start up airflow after pc shutdown
• Change the update interval
• Add another table to existing dag run
• Cancel a dag run
• Startup a dag schedule that has been paused
• Add new dag to airflow

Startup airflow after pc shutdown

1. Open 2 Ubuntu applications (The 3de is for demonstration purposes)
2. Startup Postgresql in one of the Ubuntu applications
3. Run airflow scheduler
4. Run airflow webserver

Change the update interval

1. Open the dag that you want to update in your prefered editor. ( $AIRFLOW_HOME/dags )
2. Go to the scheduled_interval parameter where the DAG class is instantiated and set to the new interval.
3. Save the file.

Links: Timedelta, Scheduling and Triggers

Add another table to existing dag run

1. Open the dag that you want to update in your prefered editor.
2. Make sure the new table config and SQL have been added to the database-pipeline folder.
3. Go to the tables variable and add the new table to the list.
4. Save the file.

Cancel a dag run

1. Open the airflow web-app.
2. Click on the DAG's tab.
3. Click on the desired dag that you want to cancel a job on.
4. Click on the circle/dag run.
5. Click clear.
6. Then ok.

Startup / Shutdown a dag schedule.

1. Open the airflow web-app.
2. Click on the DAG's tab.
3. Click on the on/off switch to toggle between starting and stopping the scheduling.

Add new dag to airflow

1. Copy the new dag to the airflow dag folder. ( $AIRFLOW_HOME/dags )
2. Run the list_dags command to make sure that airflow is picking up your new dag.