How To Run Bash Script File In Airflow

I have a bash script that creates a file (if it does not exist) that I want to run in Airflow, but when I try it fails. How do I do this? #!/bin/bash #create_file.sh file=filename

Solution 1:

From the tutorial this is OK:

t2 = BashOperator(
    task_id='sleep',
    bash_command='sleep 5',
    retries=3,
    dag=dag)

But you're passing a multi-line command to it

create_command = """
 ./scripts/create_file.sh
"""

should be

create_command = "./scripts/create_file.sh "

Moreover, you also have to make sure that you are in the correct directory to avoid cryptic errors. Do it like this for example:

create_command = "./scripts/create_file.sh "ifos.path.exists(create_command):
   t1 = BashOperator(
        task_id= 'create_file',
        bash_command=create_command,
        dag=dag
   )
else:
    raise Exception("Cannot locate {}".format(create_command))

Solution 2:

From the documentation:

t2 = BashOperator(
    task_id='bash_example',
    # "scripts" folder is under "/usr/local/airflow/dags"bash_command="scripts/test.sh",
    dag=dag)