Apache Airflow is a highly versatile platform for managing complex workflows, including the ability to trigger scripts on remote servers. However, it’s equally important to obtain the results or response from these scripts for analysis, debugging, or just for the sake of logging. This article explores a step-by-step approach to get the response from a remote Python or Shell script back to Airflow logs.
Prerequisites:
- Airflow and SSH Setups: You should have Airflow and SSH connections properly set up, as detailed in the previous article.
- Understanding of Bash Command Execution: Basic understanding of how bash commands execute and return outputs is necessary. The standard output (stdout) of a bash command contains the output of a script, which is what we aim to capture.
Steps to Capture Script Results:
1. Modifying the Airflow Task:
Airflow uses the concept of operators to perform tasks. The BashOperator and PythonOperator are commonly used for executing bash commands and Python scripts respectively. However, to capture the output of a command or script, we need to utilize the SSHOperator.
Replace your existing BashOperator with the SSHOperator:
from airflow.providers.ssh.operators.ssh import SSHOperator
t1 = SSHOperator(
ssh_conn_id='ssh_connection',
task_id='run_remote_script',
command='python /mnt/frehers/jobs/daily/viwership.py',
dag=dag
)
The SSHOperator connects to the remote server using the ssh_conn_id and runs the command. The output of the command is captured in Airflow’s task instance log.
2. Handling Python Scripts Output:
For Python scripts, ensure your script uses print() statements to output results. This output is automatically sent to stdout, which is then captured by Airflow and placed in the logs.
# Python script
result = perform_complex_task()
print(result)
3. Handling Shell Script Output:
For shell scripts, stdout is automatically captured. Make sure your script isn’t suppressing or redirecting the output.
#!/bin/bash
# Shell script
result=$(command_to_execute)
echo $result
4. Viewing the Output in Airflow:
Once the task is executed, the output can be seen in the Airflow logs:
a. In the Airflow Web UI, navigate to Browse > Task Instances.
b. Click on the task instance for the DAG and task you just ran.
c. Click on the “Log” button to view the logs for that task. The output of your remote script will be displayed in these logs.
Hope you should now see the output of your remotely executed scripts within the Airflow logs, allowing you to monitor and troubleshoot your workflows more effectively.
