In Apache Airflow, directed acyclic graphs (DAGs) are typically scheduled and executed automatically according to their schedule_interval. However, there might be scenarios where you want to manually trigger a DAG run, such as for testing or due to an ad hoc requirement. While you can do this via the Airflow web interface, this article shows you how to achieve the same result directly from your console or command prompt, providing more flexibility and allowing for automation opportunities.
Prerequisites:
- Airflow Installation: Airflow should be properly installed and set up on your machine.
- DAG Definition: You should have a defined DAG that you wish to trigger. We will use a simple DAG that adds two numbers as an example.
Steps to Trigger an Airflow DAG from the Command Line:
1. Define your DAG:
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from datetime import datetime, timedelta
def add_numbers(number1, number2):
result = number1 + number2
print(f"The sum of {number1} and {number2} is {result}")
default_args = {
'owner': 'airflow',
'start_date': datetime(2023, 7, 20),
'retries': 1,
'retry_delay': timedelta(minutes=1),
}
dag = DAG('add_numbers_dag', default_args=default_args, schedule_interval=timedelta(days=1))
t1 = PythonOperator(
task_id='add_numbers_task',
python_callable=add_numbers,
op_kwargs={'number1': 7, 'number2': 3},
dag=dag)This DAG, named add_numbers_dag, has a single task add_numbers_task that adds the numbers 7 and 3
2. Trigger the DAG Run:
Airflow provides a command-line interface (CLI) that allows you to interact with it using various commands. To trigger a DAG run, use the airflow dags trigger command followed by the DAG id.
Open your command prompt or console, navigate to your Airflow directory (the one that contains the airflow.cfg file), and run the following command:
airflow dags trigger add_numbers_dag
You should see a message indicating that the DAG run has been created.
3. Monitor the DAG Run:
You can monitor the progress of your DAG run in the Airflow web UI. Navigate to the Graph View or Tree View of your DAG and you should see the newly triggered run and its status. Additionally, the output of the task (i.e., the result of the addition) can be viewed in the task logs.
Triggering an Airflow DAG from the console can be a powerful tool, allowing for more flexibility than relying solely on the schedule_interval or the web UI. By integrating this capability into shell scripts or other automation tools, you can create more dynamic and responsive workflows.
Read more on Airflow here : Airflow
