Skip to content

Commit f6bd817

Browse files
potiukpotiuk
authored
Introduce 'transfers' packages (#9320)
* Consistent naming of transfer operators Transfer operators have consistent names and are grouped in the 'transfer' packages. * fixup! Consistent naming of transfer operators * Introduces 'transfers' packages. Closes #9161 and #8620 * fixup! Introduces 'transfers' packages. * fixup! fixup! Introduces 'transfers' packages. * fixup! fixup! fixup! Introduces 'transfers' packages.
1 parent c78e2a5 commit f6bd817

343 files changed

Lines changed: 2612 additions & 1544 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.pre-commit-config.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -264,6 +264,8 @@ metastore_browser/templates/.*\\.html$|.*\\.jinja2"
264264
(?x)
265265
^airflow/providers/apache/cassandra/hooks/cassandra.py$|
266266
^airflow/providers/apache/hive/operators/hive_stats.py$|
267+
^airflow/providers/apache/hive/PROVIDERS_CHANGES_*|
268+
^airflow/providers/apache/hive/README.md$|
267269
^tests/providers/apache/cassandra/hooks/test_cassandra.py
268270
- id: consistent-pylint
269271
language: pygrep

‎CONTRIBUTING.rst‎

Lines changed: 66 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -542,8 +542,71 @@ We support the following types of tests:
542542

543543
For details on running different types of Airflow tests, see `TESTING.rst <TESTING.rst>`_.
544544

545+
546+
Naming Conventions for provider packages
547+
========================================
548+
549+
In Airflow 2.0 we standardized and enforced naming for provider packages, modules and classes.
550+
those rules (introduced as AIP-21) were not only introduced but enforced using automated checks
551+
that verify if the naming conventions are followed. Here is a brief summary of the rules, for
552+
detailed discussion you can go to [AIP-21 Changes in import paths](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths)
553+
554+
The rules are as follows:
555+
556+
* Provider packages are all placed in 'airflow.providers'
557+
558+
* Providers are usually direct sub-packages of the 'airflow.providers' package but in some cases they can be
559+
further split into sub-packages (for example 'apache' package has 'cassandra', 'druid' ... providers ) out
560+
of which several different provider packages are produced (apache.cassandra, apache.druid). This is
561+
case when the providers are connected under common umbrella but very loosely coupled on the code level.
562+
563+
* In some cases the package can have sub-packages but they are all delivered as single provider
564+
package (for example 'google' package contains 'ads', 'cloud' etc. sub-packages). This is in case
565+
the providers are connected under common umbrella and they are also tightly coupled on the code level.
566+
567+
* Typical structure of provider package:
568+
* example_dags -> example DAGs are stored here (used for documentation and System Tests)
569+
* hooks -> hooks are stored here
570+
* operators -> operators are stored here
571+
* sensors -> sensors are stored here
572+
* secrets -> secret backends are stored here
573+
* transfers -> transfer operators are stored here
574+
575+
* Module names do not contain word "hooks" , "operators" etc. The right type comes from
576+
the package. For example 'hooks.datastore' module contains DataStore hook and 'operators.datastore'
577+
contains DataStore operators.
578+
579+
* Class names contain 'Operator', 'Hook', 'Sensor' - for example DataStoreHook, DataStoreExportOperator
580+
581+
* Operator name usually follows the convention: <Subject><Action><Entity>Operator
582+
(BigQueryExecuteQueryOperator) is a good example
583+
584+
* Transfer Operators are those that actively push data from one service/provider and send it to another
585+
service (might be for the same or another provider). This usually involves two hooks. The convention
586+
for those <Source>To<Destination>Operator. They are not named *TransferOperator nor *Transfer.
587+
588+
* Operators that use external service to perform transfer (for example CloudDataTransferService operators
589+
are not placed in "transfers" package and do not have to follow the naming convention for
590+
transfer operators.
591+
592+
* It is often debatable where to put transfer operators but we agreed to the following criteria:
593+
594+
* We use "maintainability" of the operators as the main criteria - so the transfer operator
595+
should be kept at the provider which has highest "interest" in the transfer operator
596+
597+
* For Cloud Providers or Service providers that usually means that the transfer operators
598+
should land at the "target" side of the transfer
599+
600+
* Secret Backend name follows the convention: <SecretEngine>Backend.
601+
602+
* Tests are grouped in parallel packages under "tests.providers" top level package. Module name is usually
603+
"test_<object_to_test>.py',
604+
605+
* System tests (not yet fully automated but allowing to run e2e testing of partucular provider) are
606+
named with _system.py suffix.
607+
545608
Metadata Database Updates
546-
==============================
609+
=========================
547610

548611
When developing features, you may need to persist information to the metadata
549612
database. Airflow has `Alembic <https://github.com/sqlalchemy/alembic>`__ built-in
@@ -623,7 +686,7 @@ could get a reproducible build. See the `Yarn docs
623686

624687

625688
Generate Bundled Files with yarn
626-
----------------------------------
689+
--------------------------------
627690

628691
To parse and generate bundled files for Airflow, run either of the following
629692
commands:
@@ -910,6 +973,7 @@ You can join the channels via links at the `Airflow Community page <https://airf
910973
* The deprecated `JIRA issues <https://issues.apache.org/jira/projects/AIRFLOW/issues/AIRFLOW-4470?filter=allopenissues>`_ for:
911974
* checking out old but still valuable issues that are not on Github yet
912975
* mentioning the JIRA issue number in the title of the related PR you would like to open on Github
976+
913977
**IMPORTANT**
914978
We don't create new issues on JIRA anymore. The reason we still look at JIRA issues is that there are valuable tickets inside of it. However, each new PR should be created on `Github issues <https://github.com/apache/airflow/issues>`_ as stated in `Contribution Workflow Example <https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example>`_
915979

‎UPDATING.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -912,7 +912,7 @@ The following table shows changes in import paths.
912912
|airflow.contrib.operators.dataproc_operator.DataprocWorkflowTemplateInstantiateOperator |airflow.providers.google.cloud.operators.dataproc.DataprocInstantiateWorkflowTemplateOperator |
913913
|airflow.contrib.operators.datastore_export_operator.DatastoreExportOperator |airflow.providers.google.cloud.operators.datastore.DatastoreExportOperator |
914914
|airflow.contrib.operators.datastore_import_operator.DatastoreImportOperator |airflow.providers.google.cloud.operators.datastore.DatastoreImportOperator |
915-
|airflow.contrib.operators.file_to_gcs.FileToGoogleCloudStorageOperator |airflow.providers.google.cloud.operators.local_to_gcs.FileToGoogleCloudStorageOperator |
915+
|airflow.contrib.operators.file_to_gcs.FileToGoogleCloudStorageOperator |airflow.providers.google.cloud.transfers.local_to_gcs.FileToGoogleCloudStorageOperator |
916916
|airflow.contrib.operators.gcp_bigtable_operator.BigtableClusterUpdateOperator |airflow.providers.google.cloud.operators.bigtable.BigtableUpdateClusterOperator |
917917
|airflow.contrib.operators.gcp_bigtable_operator.BigtableInstanceCreateOperator |airflow.providers.google.cloud.operators.bigtable.BigtableCreateInstanceOperator |
918918
|airflow.contrib.operators.gcp_bigtable_operator.BigtableInstanceDeleteOperator |airflow.providers.google.cloud.operators.bigtable.BigtableDeleteInstanceOperator |
@@ -1006,7 +1006,7 @@ The following table shows changes in import paths.
10061006
|airflow.contrib.operators.gcs_acl_operator.GoogleCloudStorageBucketCreateAclEntryOperator |airflow.providers.google.cloud.operators.gcs.GCSBucketCreateAclEntryOperator |
10071007
|airflow.contrib.operators.gcs_acl_operator.GoogleCloudStorageObjectCreateAclEntryOperator |airflow.providers.google.cloud.operators.gcs.GCSObjectCreateAclEntryOperator |
10081008
|airflow.contrib.operators.gcs_delete_operator.GoogleCloudStorageDeleteOperator |airflow.providers.google.cloud.operators.gcs.GCSDeleteObjectsOperator |
1009-
|airflow.contrib.operators.gcs_download_operator.GoogleCloudStorageDownloadOperator |airflow.providers.google.cloud.operators.gcs.GCSToLocalOperator |
1009+
|airflow.contrib.operators.gcs_download_operator.GoogleCloudStorageDownloadOperator |airflow.providers.google.cloud.operators.gcs.GCSToLocalFilesystemOperator |
10101010
|airflow.contrib.operators.gcs_list_operator.GoogleCloudStorageListOperator |airflow.providers.google.cloud.operators.gcs.GCSListObjectsOperator |
10111011
|airflow.contrib.operators.gcs_operator.GoogleCloudStorageCreateBucketOperator |airflow.providers.google.cloud.operators.gcs.GCSCreateBucketOperator |
10121012
|airflow.contrib.operators.gcs_to_bq.GoogleCloudStorageToBigQueryOperator |airflow.operators.gcs_to_bq.GoogleCloudStorageToBigQueryOperator |

‎airflow/contrib/operators/adls_to_gcs.py‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -15,28 +15,28 @@
1515
# KIND, either express or implied. See the License for the
1616
# specific language governing permissions and limitations
1717
# under the License.
18-
"""This module is deprecated. Please use `airflow.providers.google.cloud.operators.adls_to_gcs`."""
18+
"""This module is deprecated. Please use `airflow.providers.google.cloud.transfers.adls_to_gcs`."""
1919

2020
import warnings
2121

22-
from airflow.providers.google.cloud.operators.adls_to_gcs import ADLSToGCSOperator
22+
from airflow.providers.google.cloud.transfers.adls_to_gcs import ADLSToGCSOperator
2323

2424
warnings.warn(
25-
"This module is deprecated. Please use `airflow.providers.google.cloud.operators.adls_to_gcs`.",
25+
"This module is deprecated. Please use `airflow.providers.google.cloud.transfers.adls_to_gcs`.",
2626
DeprecationWarning, stacklevel=2
2727
)
2828

2929

3030
class AdlsToGoogleCloudStorageOperator(ADLSToGCSOperator):
3131
"""
3232
This class is deprecated.
33-
Please use `airflow.providers.google.cloud.operators.adls_to_gcs.ADLSToGCSOperator`.
33+
Please use `airflow.providers.google.cloud.transfers.adls_to_gcs.ADLSToGCSOperator`.
3434
"""
3535

3636
def __init__(self, *args, **kwargs):
3737
warnings.warn(
3838
"""This class is deprecated.
39-
Please use `airflow.providers.google.cloud.operators.adls_to_gcs.ADLSToGCSOperator`.""",
39+
Please use `airflow.providers.google.cloud.transfers.adls_to_gcs.ADLSToGCSOperator`.""",
4040
DeprecationWarning, stacklevel=2
4141
)
4242
super().__init__(*args, **kwargs)

‎airflow/contrib/operators/bigquery_to_bigquery.py‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,14 +15,14 @@
1515
# KIND, either express or implied. See the License for the
1616
# specific language governing permissions and limitations
1717
# under the License.
18-
"""This module is deprecated. Please use `airflow.providers.google.cloud.operators.bigquery_to_bigquery`."""
18+
"""This module is deprecated. Please use `airflow.providers.google.cloud.transfers.bigquery_to_bigquery`."""
1919

2020
import warnings
2121

2222
# pylint: disable=unused-import
23-
from airflow.providers.google.cloud.operators.bigquery_to_bigquery import BigQueryToBigQueryOperator # noqa
23+
from airflow.providers.google.cloud.transfers.bigquery_to_bigquery import BigQueryToBigQueryOperator # noqa
2424

2525
warnings.warn(
26-
"This module is deprecated. Please use `airflow.providers.google.cloud.operators.bigquery_to_bigquery`.",
26+
"This module is deprecated. Please use `airflow.providers.google.cloud.transfers.bigquery_to_bigquery`.",
2727
DeprecationWarning, stacklevel=2
2828
)

‎airflow/contrib/operators/bigquery_to_gcs.py‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -15,28 +15,28 @@
1515
# KIND, either express or implied. See the License for the
1616
# specific language governing permissions and limitations
1717
# under the License.
18-
"""This module is deprecated. Please use `airflow.providers.google.cloud.operators.bigquery_to_gcs`."""
18+
"""This module is deprecated. Please use `airflow.providers.google.cloud.transfers.bigquery_to_gcs`."""
1919

2020
import warnings
2121

22-
from airflow.providers.google.cloud.operators.bigquery_to_gcs import BigQueryToGCSOperator
22+
from airflow.providers.google.cloud.transfers.bigquery_to_gcs import BigQueryToGCSOperator
2323

2424
warnings.warn(
25-
"This module is deprecated. Please use `airflow.providers.google.cloud.operators.bigquery_to_gcs`.",
25+
"This module is deprecated. Please use `airflow.providers.google.cloud.transfers.bigquery_to_gcs`.",
2626
DeprecationWarning, stacklevel=2
2727
)
2828

2929

3030
class BigQueryToCloudStorageOperator(BigQueryToGCSOperator):
3131
"""
3232
This class is deprecated.
33-
Please use `airflow.providers.google.cloud.operators.bigquery_to_gcs.BigQueryToGCSOperator`.
33+
Please use `airflow.providers.google.cloud.transfers.bigquery_to_gcs.BigQueryToGCSOperator`.
3434
"""
3535

3636
def __init__(self, *args, **kwargs):
3737
warnings.warn(
3838
"""This class is deprecated.
39-
Please use `airflow.providers.google.cloud.operators.bigquery_to_gcs.BigQueryToGCSOperator`.""",
39+
Please use `airflow.providers.google.cloud.transfers.bigquery_to_gcs.BigQueryToGCSOperator`.""",
4040
DeprecationWarning, stacklevel=2
4141
)
4242
super().__init__(*args, **kwargs)

‎airflow/contrib/operators/bigquery_to_mysql_operator.py‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,14 +15,14 @@
1515
# KIND, either express or implied. See the License for the
1616
# specific language governing permissions and limitations
1717
# under the License.
18-
"""This module is deprecated. Please use `airflow.providers.google.cloud.operators.bigquery_to_mysql`."""
18+
"""This module is deprecated. Please use `airflow.providers.google.cloud.transfers.bigquery_to_mysql`."""
1919

2020
import warnings
2121

2222
# pylint: disable=unused-import
23-
from airflow.providers.google.cloud.operators.bigquery_to_mysql import BigQueryToMySqlOperator # noqa
23+
from airflow.providers.google.cloud.transfers.bigquery_to_mysql import BigQueryToMySqlOperator # noqa
2424

2525
warnings.warn(
26-
"This module is deprecated. Please use `airflow.providers.google.cloud.operators.bigquery_to_mysql`.",
26+
"This module is deprecated. Please use `airflow.providers.google.cloud.transfers.bigquery_to_mysql`.",
2727
DeprecationWarning, stacklevel=2
2828
)

‎airflow/contrib/operators/cassandra_to_gcs.py‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -16,29 +16,29 @@
1616
# specific language governing permissions and limitations
1717
# under the License.
1818
"""
19-
This module is deprecated. Please use `airflow.providers.google.cloud.operators.cassandra_to_gcs`.
19+
This module is deprecated. Please use `airflow.providers.google.cloud.transfers.cassandra_to_gcs`.
2020
"""
2121

2222
import warnings
2323

24-
from airflow.providers.google.cloud.operators.cassandra_to_gcs import CassandraToGCSOperator
24+
from airflow.providers.google.cloud.transfers.cassandra_to_gcs import CassandraToGCSOperator
2525

2626
warnings.warn(
27-
"This module is deprecated. Please use `airflow.providers.google.cloud.operators.cassandra_to_gcs`.",
27+
"This module is deprecated. Please use `airflow.providers.google.cloud.transfers.cassandra_to_gcs`.",
2828
DeprecationWarning, stacklevel=2
2929
)
3030

3131

3232
class CassandraToGoogleCloudStorageOperator(CassandraToGCSOperator):
3333
"""
3434
This class is deprecated.
35-
Please use `airflow.providers.google.cloud.operators.cassandra_to_gcs.CassandraToGCSOperator`.
35+
Please use `airflow.providers.google.cloud.transfers.cassandra_to_gcs.CassandraToGCSOperator`.
3636
"""
3737

3838
def __init__(self, *args, **kwargs):
3939
warnings.warn(
4040
"""This class is deprecated.
41-
Please use `airflow.providers.google.cloud.operators.cassandra_to_gcs.CassandraToGCSOperator`.""",
41+
Please use `airflow.providers.google.cloud.transfers.cassandra_to_gcs.CassandraToGCSOperator`.""",
4242
DeprecationWarning, stacklevel=2
4343
)
4444
super().__init__(*args, **kwargs)

‎airflow/contrib/operators/dynamodb_to_s3.py‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,14 +15,14 @@
1515
# KIND, either express or implied. See the License for the
1616
# specific language governing permissions and limitations
1717
# under the License.
18-
"""This module is deprecated. Please use `airflow.providers.amazon.aws.operators.dynamodb_to_s3`."""
18+
"""This module is deprecated. Please use `airflow.providers.amazon.aws.transfers.dynamodb_to_s3`."""
1919

2020
import warnings
2121

2222
# pylint: disable=unused-import
23-
from airflow.providers.amazon.aws.operators.dynamodb_to_s3 import DynamoDBToS3Operator # noqa
23+
from airflow.providers.amazon.aws.transfers.dynamodb_to_s3 import DynamoDBToS3Operator # noqa
2424

2525
warnings.warn(
26-
"This module is deprecated. Please use `airflow.providers.amazon.aws.operators.dynamodb_to_s3`.",
26+
"This module is deprecated. Please use `airflow.providers.amazon.aws.transfers.dynamodb_to_s3`.",
2727
DeprecationWarning, stacklevel=2
2828
)

‎airflow/contrib/operators/file_to_gcs.py‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -16,30 +16,30 @@
1616
# specific language governing permissions and limitations
1717
# under the License.
1818
"""
19-
This module is deprecated. Please use `airflow.providers.google.cloud.operators.local_to_gcs`.
19+
This module is deprecated. Please use `airflow.providers.google.cloud.transfers.local_to_gcs`.
2020
"""
2121

2222
import warnings
2323

24-
from airflow.providers.google.cloud.operators.local_to_gcs import LocalFilesystemToGCSOperator
24+
from airflow.providers.google.cloud.transfers.local_to_gcs import LocalFilesystemToGCSOperator
2525

2626
warnings.warn(
27-
"This module is deprecated. Please use `airflow.providers.google.cloud.operators.local_to_gcs`,",
27+
"This module is deprecated. Please use `airflow.providers.google.cloud.transfers.local_to_gcs`,",
2828
DeprecationWarning, stacklevel=2
2929
)
3030

3131

3232
class FileToGoogleCloudStorageOperator(LocalFilesystemToGCSOperator):
3333
"""
3434
This class is deprecated.
35-
Please use `airflow.providers.google.cloud.operators.local_to_gcs.LocalFilesystemToGCSOperator`.
35+
Please use `airflow.providers.google.cloud.transfers.local_to_gcs.LocalFilesystemToGCSOperator`.
3636
"""
3737

3838
def __init__(self, *args, **kwargs):
3939
warnings.warn(
4040
"""This class is deprecated.
4141
Please use
42-
`airflow.providers.google.cloud.operators.local_to_gcs.LocalFilesystemToGCSOperator`.""",
42+
`airflow.providers.google.cloud.transfers.local_to_gcs.LocalFilesystemToGCSOperator`.""",
4343
DeprecationWarning, stacklevel=2
4444
)
4545
super().__init__(*args, **kwargs)

0 commit comments

Comments
 (0)