Skip to content

Commit 92585ca

Browse files
potiukpotiuk
authored
Added automated release notes generation for backport operators (#8807)
We have now mechanism to keep release notes updated for the backport operators in an automated way. It really nicely generates all the necessary information: * summary of requirements for each backport package * list of dependencies (including extras to install them) when package depends on other providers packages * table of new hooks/operators/sensors/protocols/secrets * table of moved hooks/operators/sensors/protocols/secrets with information where they were moved from * changelog of all the changes to the provider package (this will be automatically updated with incremental changelog whenever we decide to release separate packages. The system is fully automated - we will be able to produce release notes automatically (per-package) whenever we decide to release new version of the package in the future.
1 parent f82ad45 commit 92585ca

155 files changed

Lines changed: 9938 additions & 923 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.pre-commit-config.yaml‎

Lines changed: 6 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,8 @@ repos:
6969
- --fuzzy-match-generates-todo
7070
- id: insert-license
7171
name: Add license for all JINJA template files
72-
files: ^airflow/www/templates/.*\.html$|^docs/templates/.*\.html$|^airflow/contrib/plugins/metastore_browser/templates/.*\.html$ # yamllint disable-line rule:line-length
72+
files: "^airflow/www/templates/.*\\.html$|^docs/templates/.*\\.html$|^airflow/contrib/plugins/\
73+
metastore_browser/templates/.*\\.html$|.*\\.jinja2"
7374
exclude: ^\.github/.*$|^airflow/_vendor/.*$
7475
args:
7576
- --comment-style
@@ -120,7 +121,7 @@ repos:
120121
- id: insert-license
121122
name: Add license for all md files
122123
files: \.md$
123-
exclude: ^\.github/.*$|^airflow/_vendor/.*$
124+
exclude: ^\.github/.*$|^airflow/_vendor/.*|PROVIDERS_CHANGES.*\.md
124125
args:
125126
- --comment-style
126127
- "<!--|| -->"
@@ -132,16 +133,10 @@ repos:
132133
hooks:
133134
- id: doctoc
134135
name: Add TOC for md files
135-
files: ^README\.md$|^CONTRIBUTING\.md$|^UPDATING.md$|^dev/README\.md$
136+
files: ^README\.md$|^CONTRIBUTING\.md$|^UPDATING.md$|^dev/README\.md$|^dev/BACKPORT_PACKAGES.md$
136137
args:
137138
- "--maxlevel"
138139
- "2"
139-
- repo: https://github.com/thlorenz/doctoc.git
140-
rev: v1.4.0
141-
hooks:
142-
- id: doctoc
143-
name: Add TOC for backport readme files
144-
files: BACKPORT_README\.md$
145140
- repo: meta
146141
hooks:
147142
- id: check-hooks-apply
@@ -277,7 +272,8 @@ repos:
277272
^airflow/contrib/.*\.py$
278273
- id: provide-create-sessions
279274
language: pygrep
280-
name: To avoid import cycles make sure provide_session and create_session are imported from airflow.utils.session # yamllint disable-line rule:line-length
275+
name: To avoid import cycles make sure provide_session and create_session are imported from
276+
airflow.utils.session
281277
entry: "from airflow\\.utils\\.db import.* (provide_session|create_session)"
282278
files: \.py$
283279
pass_filenames: true

‎.rat-excludes‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,7 @@ rat-results.txt
7575
apache-airflow-.*\+source.tar.gz.*
7676
apache-airflow-.*\+bin.tar.gz.*
7777
PULL_REQUEST_TEMPLATE.md
78+
PROVIDERS_CHANGES*.md
7879

7980
# vendored modules
8081
_vendor/*

‎BREEZE.rst‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -655,6 +655,8 @@ This is the current syntax for `./breeze <./breeze>`_:
655655
cleanup-image Cleans up the container image created
656656
exec Execs into running breeze container in new terminal
657657
generate-requirements Generates pinned requirements for pip dependencies
658+
generate-backport-readme Generates backport packages readme files
659+
prepare-backport-packages Prepares backport packages
658660
initialize-local-virtualenv Initializes local virtualenv
659661
setup-autocomplete Sets up autocomplete for breeze
660662
stop Stops the docker-compose environment
@@ -870,6 +872,84 @@ This is the current syntax for `./breeze <./breeze>`_:
870872
####################################################################################################
871873
872874
875+
Detailed usage for command: generate-backport-readme
876+
877+
breeze [FLAGS] generate-backport-readme -- <EXTRA_ARGS>
878+
879+
Prepares README.md files for backport packages. You can provide (after --) optional version
880+
in the form of YYYY.MM.DD, optionally followed by the list of packages to generate readme for.
881+
If the first parameter is not formatted as a date, then today is used as version.
882+
If no packages are specified, readme for all packages are generated.
883+
If no date is specified, current date + 3 days is used (allowing for PMC votes to pass).
884+
885+
Examples:
886+
887+
'breeze generate-backport-readme' or
888+
'breeze generate-backport-readme -- 2020.05.10' or
889+
'breeze generate-backport-readme -- 2020.05.10 https google amazon'
890+
891+
General form:
892+
893+
'breeze generate-backport-readme -- YYYY.MM.DD <PACKAGE_ID> ...'
894+
895+
* YYYY.MM.DD - is the CALVER version of the package to prepare. Note that this date
896+
cannot be earlier than the already released version (the script will fail if it
897+
will be). It can be set in the future anticipating the future release date.
898+
899+
* <PACKAGE_ID> is usually directory in the airflow/providers folder (for example
900+
'google' but in several cases, it might be one level deeper separated with
901+
'.' for example 'apache.hive'
902+
903+
Flags:
904+
905+
-v, --verbose
906+
Show verbose information about executed commands (enabled by default for running test).
907+
Note that you can further increase verbosity and see all the commands executed by breeze
908+
by running 'export VERBOSE_COMMANDS="true"' before running breeze.
909+
910+
911+
####################################################################################################
912+
913+
914+
Detailed usage for command: prepare-backport-packages
915+
916+
breeze [FLAGS] prepare-backport-packages -- <EXTRA_ARGS>
917+
918+
Builds backport packages. You can provide (after --) optional list of packages to prepare.
919+
If no packages are specified, readme for all packages are generated. You can specify optional
920+
--version-suffix flag to generate rc candidates for the packages.
921+
922+
Make sure to set the right version in './backport_packages/setup_backport_packages.py'
923+
924+
Examples:
925+
926+
'breeze prepare-backport-packages' or
927+
'breeze prepare-backport-packages -- google' or
928+
'breeze prepare-backport-packages --version-suffix rc1 -- http google amazon'
929+
930+
General form:
931+
932+
'breeze prepare-backport-packages -- <PACKAGE_ID> ...'
933+
934+
* <PACKAGE_ID> is usually directory in the airflow/providers folder (for example
935+
'google'), but in several cases, it might be one level deeper separated with '.'
936+
for example 'apache.hive'
937+
938+
Flags:
939+
940+
-S, --version-suffix
941+
Adds optional suffix to the generated backport package version. It can be used to generate
942+
rc1/rc2 ... versions of the packages.
943+
944+
-v, --verbose
945+
Show verbose information about executed commands (enabled by default for running test).
946+
Note that you can further increase verbosity and see all the commands executed by breeze
947+
by running 'export VERBOSE_COMMANDS="true"' before running breeze.
948+
949+
950+
####################################################################################################
951+
952+
873953
Detailed usage for command: initialize-local-virtualenv
874954
875955
breeze [FLAGS] initialize-local-virtualenv -- <EXTRA_ARGS>
@@ -1340,6 +1420,13 @@ This is the current syntax for `./breeze <./breeze>`_:
13401420
-H, --dockerhub-repo
13411421
DockerHub repository used to pull, push, build images. Default: airflow.
13421422
1423+
****************************************************************************************************
1424+
Flags for generation of the backport packages
1425+
1426+
-S, --version-suffix
1427+
Adds optional suffix to the generated backport package version. It can be used to generate
1428+
rc1/rc2 ... versions of the packages.
1429+
13431430
****************************************************************************************************
13441431
Increase verbosity of the scripts
13451432

‎CONTRIBUTING.rst‎

Lines changed: 5 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -313,14 +313,15 @@ This is the full list of those extras:
313313

314314
.. START EXTRAS HERE
315315
316-
all, all_dbs, amazon, apache.atlas, apache.cassandra, apache.druid, apache.hdfs, apache.hive,
317-
apache.pinot, apache.webhdfs, async, atlas, aws, azure, cassandra, celery, cgroups, cloudant,
318-
cncf.kubernetes, dask, databricks, datadog, devel, devel_ci, devel_hadoop, doc, docker, druid,
316+
all_dbs, amazon, apache.atlas, apache_beam, apache.cassandra, apache.druid, apache.hdfs,
317+
apache.hive, apache.pinot, apache.webhdfs, async, atlas, aws, azure, cassandra, celery, cgroups,
318+
cloudant, cncf.kubernetes, dask, databricks, datadog, devel, devel_hadoop, doc, docker, druid,
319319
elasticsearch, exasol, facebook, gcp, gcp_api, github_enterprise, google, google_auth, grpc,
320320
hashicorp, hdfs, hive, jdbc, jira, kerberos, kubernetes, ldap, microsoft.azure, microsoft.mssql,
321321
microsoft.winrm, mongo, mssql, mysql, odbc, oracle, pagerduty, papermill, password, pinot, postgres,
322322
presto, qds, rabbitmq, redis, salesforce, samba, segment, sendgrid, sentry, singularity, slack,
323-
snowflake, ssh, statsd, tableau, vertica, virtualenv, webhdfs, winrm, yandexcloud
323+
snowflake, spark, ssh, statsd, tableau, vertica, virtualenv, webhdfs, winrm, yandexcloud, all,
324+
devel_ci
324325

325326
.. END EXTRAS HERE
326327
@@ -981,52 +982,3 @@ Resources & Links
981982
- `Airflow’s official documentation <http://airflow.apache.org/>`__
982983

983984
- `More resources and links to Airflow related content on the Wiki <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Links>`__
984-
985-
Preparing backport packages
986-
===========================
987-
988-
As part of preparation to Airflow 2.0 we decided to prepare backport of providers package that will be
989-
possible to install in the Airflow 1.10.*, Python 3.6+ environment.
990-
Some of those packages will be soon (after testing) officially released via PyPi, but you can build and
991-
prepare such packages on your own easily.
992-
993-
* The setuptools.py script only works in python3.6+. This is also our minimally supported python
994-
version to use the packages in.
995-
996-
* Make sure you have ``setuptools`` and ``wheel`` installed in your python environment. The easiest way
997-
to do it is to run ``pip install setuptools wheel``
998-
999-
* Run the following command:
1000-
1001-
.. code-block:: bash
1002-
1003-
./scripts/ci/ci_prepare_packages.sh
1004-
1005-
* Usually you only build some of the providers package. The ``providers`` directory is separated into
1006-
separate providers. You can see the list of all available providers by running
1007-
``./scripts/ci/ci_prepare_packages.sh --help``. You can build the backport package
1008-
by running ``./scripts/ci/ci_prepare_packages.sh <PROVIDER_NAME>``. Note that there
1009-
might be (and are) dependencies between some packages that might prevent subset of the packages
1010-
to be used without installing the packages they depend on. This will be solved soon by
1011-
adding cross-dependencies between packages.
1012-
1013-
* This creates a wheel package in your ``dist`` folder with a name similar to:
1014-
``apache_airflow_backport_providers-0.0.1-py2.py3-none-any.whl``
1015-
1016-
* You can install this package with ``pip install <PACKAGE_FILE>``
1017-
1018-
1019-
* You can also build sdist (source distribution packages) by running
1020-
``python setup.py <PROVIDER_NAME> sdist`` but this is only needed in case of distribution of the packages.
1021-
1022-
Each package has description generated from the the general ``backport_packages/README.md`` file with the
1023-
following replacements:
1024-
1025-
* ``{{ PACKAGE_NAME }}`` is replaced with the name of the package
1026-
(``apache-airflow-backport-providers-<NAME>``)
1027-
* ``{{ PACKAGE_DEPENDENCIES }}`` is replaced with list of optional dependencies for the package
1028-
* ``{{ PACKAGE_BACKPORT_README }}`` is replaced with the content of ``BACKPORT_README.md`` file in the
1029-
package folder if it exists.
1030-
1031-
Note that those are unofficial packages yet - they are not yet released in PyPi, but you might use them to
1032-
test the master versions of operators/hooks/sensors in Airflow 1.10.* environment with Python3.6+

‎INSTALL‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -44,14 +44,15 @@ pip install . --constraint requirements/requirements-python3.7.txt
4444
# You can also install Airflow with extras specified. The list of available extras:
4545
# START EXTRAS HERE
4646

47-
all, all_dbs, amazon, apache.atlas, apache.cassandra, apache.druid, apache.hdfs, apache.hive,
48-
apache.pinot, apache.webhdfs, async, atlas, aws, azure, cassandra, celery, cgroups, cloudant,
49-
cncf.kubernetes, dask, databricks, datadog, devel, devel_ci, devel_hadoop, doc, docker, druid,
47+
all_dbs, amazon, apache.atlas, apache_beam, apache.cassandra, apache.druid, apache.hdfs,
48+
apache.hive, apache.pinot, apache.webhdfs, async, atlas, aws, azure, cassandra, celery, cgroups,
49+
cloudant, cncf.kubernetes, dask, databricks, datadog, devel, devel_hadoop, doc, docker, druid,
5050
elasticsearch, exasol, facebook, gcp, gcp_api, github_enterprise, google, google_auth, grpc,
5151
hashicorp, hdfs, hive, jdbc, jira, kerberos, kubernetes, ldap, microsoft.azure, microsoft.mssql,
5252
microsoft.winrm, mongo, mssql, mysql, odbc, oracle, pagerduty, papermill, password, pinot, postgres,
5353
presto, qds, rabbitmq, redis, salesforce, samba, segment, sendgrid, sentry, singularity, slack,
54-
snowflake, ssh, statsd, tableau, vertica, virtualenv, webhdfs, winrm, yandexcloud
54+
snowflake, spark, ssh, statsd, tableau, vertica, virtualenv, webhdfs, winrm, yandexcloud, all,
55+
devel_ci
5556

5657
# END EXTRAS HERE
5758

‎TESTING.rst‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -547,12 +547,12 @@ Preparing backport packages for System Tests for Airflow 1.10.* series
547547
----------------------------------------------------------------------
548548

549549
To run system tests with old Airflow version you need to prepare backport packages. This
550-
can be done by running ``./scripts/ci/ci_prepare_packages.sh <PACKAGES TO BUILD>``. For
550+
can be done by running ``./breeze prepare-backport-packages -- <PACKAGES TO BUILD>``. For
551551
example the below command will build google postgres and mysql packages:
552552

553553
.. code-block:: bash
554554
555-
./scripts/ci/ci_prepare_packages.sh google postgres mysql
555+
./breeze prepare-backport-packages -- google postgres mysql
556556
557557
Those packages will be prepared in ./dist folder. This folder is mapped to /dist folder
558558
when you enter Breeze, so it is easy to automate installing those packages for testing.
@@ -614,7 +614,7 @@ Here is the typical session that you need to do to run system tests:
614614

615615
.. code-block:: bash
616616
617-
./scripts/ci/ci_prepare_packages.sh google postgres mysql
617+
./breeze prepare-backport-packages -- google postgres mysql
618618
619619
2. Enter breeze with installing Airflow 1.10.*, forwarding credentials and installing
620620
backported packages (you need an appropriate line in ``./files/airflow-breeze-config/variables.env``)
@@ -686,7 +686,7 @@ The typical session then looks as follows:
686686

687687
.. code-block:: bash
688688
689-
./scripts/ci/ci_prepare_packages.sh google postgres mysql
689+
./breeze prepare-backport-packages -- google postgres mysql
690690
691691
2. Enter breeze with installing Airflow 1.10.*, forwarding credentials and installing
692692
backported packages (you need an appropriate line in ``./files/airflow-breeze-config/variables.env``)
@@ -716,7 +716,7 @@ In the host:
716716

717717
.. code-block:: bash
718718
719-
./scripts/ci/ci_prepare_packages.sh google
719+
./breeze prepare-backport-packages -- google
720720
721721
In the container:
722722

‎airflow/contrib/operators/bigquery_operator.py‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,13 @@
1919

2020
import warnings
2121

22-
from airflow.providers.google.cloud.operators.bigquery import BigQueryExecuteQueryOperator
22+
# pylint: disable=unused-import
23+
from airflow.providers.google.cloud.operators.bigquery import ( # noqa; noqa; noqa; noqa; noqa
24+
BigQueryCreateEmptyDatasetOperator, BigQueryCreateEmptyTableOperator, BigQueryCreateExternalTableOperator,
25+
BigQueryDeleteDatasetOperator, BigQueryExecuteQueryOperator, BigQueryGetDatasetOperator,
26+
BigQueryGetDatasetTablesOperator, BigQueryPatchDatasetOperator, BigQueryUpdateDatasetOperator,
27+
BigQueryUpsertTableOperator,
28+
)
2329

2430
warnings.warn(
2531
"This module is deprecated. Please use `airflow.providers.google.cloud.operators.bigquery`.",

‎airflow/jobs/base_job.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -271,6 +271,7 @@ def reset_state_for_orphaned_tasks(self, filter_by_dag_run=None, session=None):
271271
TI.dag_id == DR.dag_id,
272272
TI.execution_date == DR.execution_date))
273273
.filter(
274+
# pylint: disable=comparison-with-callable
274275
DR.state == State.RUNNING,
275276
DR.run_id.notlike(f"{DagRunType.BACKFILL_JOB.value}__%"),
276277
TI.state.in_(resettable_states))).all()

‎airflow/jobs/scheduler_job.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1091,6 +1091,7 @@ def _change_state_for_tis_without_dagrun(self,
10911091
.filter(models.TaskInstance.dag_id.in_(simple_dag_bag.dag_ids)) \
10921092
.filter(models.TaskInstance.state.in_(old_states)) \
10931093
.filter(or_(
1094+
# pylint: disable=comparison-with-callable
10941095
models.DagRun.state != State.RUNNING,
10951096
models.DagRun.state.is_(None))) # pylint: disable=no-member
10961097
# We need to do this for mysql as well because it can cause deadlocks

‎airflow/models/serialized_dag.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,7 @@ def remove_dag(cls, dag_id: str, session=None):
140140
:param dag_id: dag_id to be deleted
141141
:param session: ORM Session
142142
"""
143+
# pylint: disable=no-member
143144
session.execute(cls.__table__.delete().where(cls.dag_id == dag_id))
144145

145146
@classmethod
@@ -158,6 +159,7 @@ def remove_stale_dags(cls, expiration_date, session=None):
158159
"scheduler since %s from %s table ", expiration_date, cls.__tablename__)
159160

160161
session.execute(
162+
# pylint: disable=no-member
161163
cls.__table__.delete().where(cls.last_updated < expiration_date)
162164
)
163165

0 commit comments

Comments
 (0)