How To Connect Airflow To Minio S3

I'm trying to run docker containers with airflow and minio and connect airflow tasks to buckets defined in minio. I'm using the new versions - airflow 2.1.3 and the newest minio im

Solution 1:

I hope this might help someone. So what I did is that I created a user and through the UI logged in with that user. Afterwards, through the UI I created a service account for that user which generated a secret key and access key.

The part of the docker-compose file that I changed looks like this now:

locals3_init:image:minio/mcdepends_on:-locals3entrypoint:>
      /bin/sh -c "
      while ! /usr/bin/mc config host add locals3 http://locals3:9000 user password; do echo 'MinIO not up and running yet...' && sleep 1; done;
      echo 'Added mc host config.';
      /usr/bin/mc admin user add locals3 airflow airflow_secret;
      echo 'Added user airflow.';
      /usr/bin/mc admin policy set locals3 readwrite user=airflow;
      /usr/bin/mc mb locals3/data;
      /usr/bin/mc alias set locals3 http://locals3 9RTK1ISXS13J85I4U6JS 4z+akfubnu+XZuoCXhqGwrtq+jgK2AYcrgGH5zsQ --api s3v4;
      exit 0;
      "

Perhaps some the mc commands could be cleaned up.

Afterwards I added a connection to the airflow. I added the secret key in the login field, the secret access key in the password field. For the connection type I chose S3.

Now, adding the name of the container (locals3 in my case) in the host field and port in the port field DOES NOT WORK. I added both the host and port through the extras:

{"host":"http://locals3:9000"}

Afterwards I was able to connect.

I'm not sure whtether the connection would work if I added a service account through the root user or used the root credentials because I did not test this yet.

EDIT:

Tested with root user credentials and it works. So the problem seems to be in the way host and port were defined.

EDIT2:

Comparing the two connection strings:

1. added host and port as extra value:

   s3://user:password@?host=http%3A%2F%2Flocals3%3A9000

2. added host and port through fields:

   s3://user:password@locals3:9000

The only explanation I can find why the first one works, and the second doesn't is because the characters in the second are not url formatted.