Skip to content

Commit e229f35

Browse files
potiukuranusjr
potiuk
and
uranusjr
authored
Use Pip 21.* to install airflow officially (#15513)
* Use Pip 21.* to install airflow officially The PIP 20.2.4 was so far the only officially supported installation mechanism for Airflow as there were some problems with conflicting dependencies (which were ignored by previous versio of PIP). This change attempts to solve this by removing a [gcp] extra from `apache-beam` which turns out to be the major source of the problem - as it contains requirements to the old version of google client libraries (but apparently only used for tests). The "apache-beam" provider migh however need the [gcp] extra for other components so in order to not break the backwards compatibility, another approach is used. Instead of adding [gcp] as extra in the apache-beam extra, the apache.beam provider's [google] extra is extended with 'apache-beam[gcp]' additional requirement so that whenever the provider is installed, the apache-beam with [gcp] extra is installed as well. * Update airflow/providers/apache/beam/CHANGELOG.rst Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * Update airflow/providers/apache/beam/CHANGELOG.rst Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * Update airflow/providers/google/CHANGELOG.rst Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * Update airflow/providers/google/CHANGELOG.rst Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
1 parent 6573846 commit e229f35

32 files changed

Lines changed: 172 additions & 173 deletions

‎CONTRIBUTING.rst‎

Lines changed: 2 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -553,15 +553,7 @@ Airflow dependencies
553553

554554
.. note::
555555

556-
On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
557-
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
558-
depend on your choice of extras. In order to install Airflow you might need to either downgrade
559-
pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
560-
you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.
561-
562-
While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
563-
set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
564-
Due to those constraints, only ``pip`` installation is currently officially supported.
556+
Only ``pip`` installation is currently officially supported.
565557

566558
While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
567559
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
@@ -788,15 +780,7 @@ Pinned constraint files
788780

789781
.. note::
790782

791-
On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
792-
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
793-
depend on your choice of extras. In order to install Airflow you might need to either downgrade
794-
pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
795-
you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.
796-
797-
While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
798-
set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
799-
Due to those constraints, only ``pip`` installation is currently officially supported.
783+
Only ``pip`` installation is officially supported.
800784

801785
While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
802786
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as

‎CONTRIBUTORS_QUICK_START.rst‎

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -167,15 +167,7 @@ Setup Airflow with Breeze and PyCharm
167167

168168
.. note::
169169

170-
On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
171-
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
172-
depend on your choice of extras. In order to install Airflow you might need to either downgrade
173-
pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
174-
you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.
175-
176-
While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
177-
set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
178-
Due to those constraints, only ``pip`` installation is currently officially supported.
170+
Only ``pip`` installation is currently officially supported.
179171

180172
While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
181173
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as

‎Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ ARG AIRFLOW_GID="50000"
4444

4545
ARG PYTHON_BASE_IMAGE="python:3.6-slim-buster"
4646

47-
ARG AIRFLOW_PIP_VERSION=20.2.4
47+
ARG AIRFLOW_PIP_VERSION=21.1
4848

4949
# By default PIP has progress bar but you can disable it.
5050
ARG PIP_PROGRESS_BAR="on"

‎Dockerfile.ci‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -212,7 +212,7 @@ ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="true"
212212
# By default in the image, we are installing all providers when installing from sources
213213
ARG INSTALL_PROVIDERS_FROM_SOURCES="true"
214214
ARG INSTALL_FROM_PYPI="true"
215-
ARG AIRFLOW_PIP_VERSION=20.2.4
215+
ARG AIRFLOW_PIP_VERSION=21.1
216216
# Setup PIP
217217
# By default PIP install run without cache to make image smaller
218218
ARG PIP_NO_CACHE_DIR="true"

‎IMAGES.rst‎

Lines changed: 2 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -172,15 +172,7 @@ This will build the image using command similar to:
172172
173173
.. note::
174174

175-
On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
176-
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
177-
depend on your choice of extras. In order to install Airflow you might need to either downgrade
178-
pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
179-
you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.
180-
181-
While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
182-
set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
183-
Due to those constraints, only ``pip`` installation is currently officially supported.
175+
Only ``pip`` installation is currently officially supported.
184176

185177
While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
186178
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
@@ -632,7 +624,7 @@ The following build arguments (``--build-arg`` in docker build command) can be u
632624
| ``ADDITIONAL_RUNTIME_APT_ENV`` | | Additional env variables defined |
633625
| | | when installing runtime deps |
634626
+------------------------------------------+------------------------------------------+------------------------------------------+
635-
| ``AIRFLOW_PIP_VERSION`` | ``20.2.4`` | PIP version used. |
627+
| ``AIRFLOW_PIP_VERSION`` | ``21.1`` | PIP version used. |
636628
+------------------------------------------+------------------------------------------+------------------------------------------+
637629
| ``PIP_PROGRESS_BAR`` | ``on`` | Progress bar for PIP installation |
638630
+------------------------------------------+------------------------------------------+------------------------------------------+

‎INSTALL‎

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -28,14 +28,6 @@ java -jar apache-rat.jar -E ./.rat-excludes -d .
2828
python3 -m venv PATH_TO_YOUR_VENV
2929
source PATH_TO_YOUR_VENV/bin/activate
3030

31-
NOTE!!
32-
33-
On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
34-
does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice
35-
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
36-
``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
37-
``--use-deprecated legacy-resolver`` to your pip install command.
38-
3931
# [required] building and installing by pip (preferred)
4032
pip install .
4133

‎LOCAL_VIRTUALENV.rst‎

Lines changed: 2 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -63,15 +63,7 @@ Extra Packages
6363

6464
.. note::
6565

66-
On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
67-
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
68-
depend on your choice of extras. In order to install Airflow you might need to either downgrade
69-
pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
70-
you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.
71-
72-
While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
73-
set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
74-
Due to those constraints, only ``pip`` installation is currently officially supported.
66+
Only ``pip`` installation is currently officially supported.
7567

7668
While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
7769
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
@@ -137,15 +129,7 @@ To create and initialize the local virtualenv:
137129

138130
.. note::
139131

140-
On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
141-
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
142-
depend on your choice of extras. In order to install Airflow you might need to either downgrade
143-
pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
144-
you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.
145-
146-
While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
147-
set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
148-
Due to those constraints, only ``pip`` installation is currently officially supported.
132+
Only ``pip`` installation is currently officially supported.
149133

150134
While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
151135
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as

‎README.md‎

Lines changed: 1 addition & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -149,14 +149,7 @@ correct Airflow tag/version/branch and Python versions in the URL.
149149

150150
NOTE!!!
151151

152-
On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
153-
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
154-
depend on your choice of extras. In order to install Airflow reliably, you might need to either downgrade
155-
pip to version 20.2.4 `pip install --upgrade pip==20.2.4` or, in case you use Pip 20.3,
156-
you might need to add option] `--use-deprecated legacy-resolver` to your pip install command.
157-
While `pip 20.3.3` solved most of the `teething` problems of 20.3, this note will remain here until we
158-
set `pip 20.3` as official version in our CI pipeline where we are testing the installation as well.
159-
Due to those constraints, only `pip` installation is currently officially supported.
152+
Only `pip` installation is currently officially supported.
160153

161154
While they are some successes with using other tools like [poetry](https://python-poetry.org) or
162155
[pip-tools](https://pypi.org/project/pip-tools), they do not share the same workflow as

‎UPDATING.md‎

Lines changed: 0 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1795,13 +1795,6 @@ you should use `pip install apache-airflow[apache.atlas]`.
17951795

17961796
NOTE!
17971797

1798-
On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
1799-
does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice
1800-
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
1801-
`pip install --upgrade pip==20.2.4` or, in case you use Pip 20.3, you need to add option
1802-
`--use-deprecated legacy-resolver` to your pip install command.
1803-
1804-
18051798
If you want to install integration for Microsoft Azure, then instead of
18061799

18071800
```

‎airflow/provider.yaml.schema.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -191,6 +191,10 @@
191191
"items": {
192192
"type": "string"
193193
}
194+
},
195+
"additional-extras": {
196+
"type": "object",
197+
"description": "Additional extras that the provider should have"
194198
}
195199
},
196200
"additionalProperties": false,

0 commit comments

Comments
 (0)