Skip to content

Commit 3d4bfdc

Browse files
potiukpotiuk
authored
Add missing __init__.py files for some test packages (#18142)
Lacl of the __init__.py caused failures in some specific cases - especially when new providers have been added. This PR adds missing ``__init__.py`` files and modifies pre-commit check which was only implemented for main files and example_dags. It checks if those files are present and adds them if missing.
1 parent 0df31cd commit 3d4bfdc

18 files changed

Lines changed: 229 additions & 21 deletions

File tree

‎.pre-commit-config.yaml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -508,16 +508,16 @@ repos:
508508
# Keep dependency versions in sync w/ airflow/www/package.json
509509
additional_dependencies: ['stylelint@13.3.1', 'stylelint-config-standard@20.0.0']
510510
- id: providers-init-file
511-
name: Provider init file
511+
name: Provider init file is missing
512512
pass_filenames: false
513513
always_run: true
514514
entry: ./scripts/ci/pre_commit/pre_commit_check_providers_init.sh
515515
language: system
516-
- id: providers-example-dags-init-file
517-
name: Provider example_dags init file
516+
- id: providers-subpackages-init-file
517+
name: Provider subpackage init files are there
518518
pass_filenames: false
519519
always_run: true
520-
entry: ./scripts/ci/pre_commit/pre_commit_check_providers_example_dag_init.py
520+
entry: ./scripts/ci/pre_commit/pre_commit_check_providers_subpackages_all_have_init.py
521521
language: python
522522
require_serial: true
523523
- id: provider-yamls

‎BREEZE.rst‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2182,11 +2182,11 @@ This is the current syntax for `./breeze <./breeze>`_:
21822182
insert-license isort json-schema language-matters lint-dockerfile lint-openapi
21832183
markdownlint mermaid mixed-line-ending mypy mypy-helm no-providers-in-core-examples
21842184
no-relative-imports pre-commit-descriptions pre-commit-hook-names pretty-format-json
2185-
provide-create-sessions providers-changelogs providers-example-dags-init-file
2186-
providers-init-file provider-yamls pydevd pydocstyle python-no-log-warn pyupgrade
2187-
restrict-start_date rst-backticks setup-order setup-extra-packages shellcheck
2188-
sort-in-the-wild sort-spelling-wordlist stylelint trailing-whitespace ui-lint
2189-
update-breeze-file update-extras update-local-yml-file update-setup-cfg-file
2185+
provide-create-sessions providers-changelogs providers-init-file
2186+
providers-subpackages-init-file provider-yamls pydevd pydocstyle python-no-log-warn
2187+
pyupgrade restrict-start_date rst-backticks setup-order setup-extra-packages
2188+
shellcheck sort-in-the-wild sort-spelling-wordlist stylelint trailing-whitespace
2189+
ui-lint update-breeze-file update-extras update-local-yml-file update-setup-cfg-file
21902190
verify-db-migrations-documented version-sync www-lint yamllint yesqa
21912191
21922192
You can pass extra arguments including options to the pre-commit framework as

‎STATIC_CODE_CHECKS.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -224,7 +224,7 @@ require Breeze Docker images to be installed locally.
224224
------------------------------------ ---------------------------------------------------------------- ------------
225225
``providers-changelogs`` Updates documentation for providers changelogs
226226
------------------------------------ ---------------------------------------------------------------- ------------
227-
``providers-example-dags-init-file`` Check that providers' example_dags __init__.py file is present
227+
``providers-subpackages-init-file`` Check that providers' subpackages __init__.py files are there
228228
------------------------------------ ---------------------------------------------------------------- ------------
229229
``providers-init-file`` Check that provider's __init__.py file is removed
230230
------------------------------------ ---------------------------------------------------------------- ------------

‎airflow/providers/google/config_templates/__init__.py‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.

‎breeze-complete‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -123,8 +123,8 @@ pre-commit-hook-names
123123
pretty-format-json
124124
provide-create-sessions
125125
providers-changelogs
126-
providers-example-dags-init-file
127126
providers-init-file
127+
providers-subpackages-init-file
128128
provider-yamls
129129
pydevd
130130
pydocstyle

scripts/ci/pre_commit/pre_commit_check_providers_example_dag_init.py renamed to scripts/ci/pre_commit/pre_commit_check_providers_subpackages_all_have_init.py

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -24,29 +24,29 @@
2424
ROOT_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir, os.pardir, os.pardir))
2525

2626

27-
def check_example_dags_dir_init_file(example_dags_dirs: List[str]) -> None:
27+
def check_dir_init_file(provider_files: List[str]) -> None:
2828
missing_init_dirs = []
29-
for example_dags_dir in example_dags_dirs:
30-
if not os.path.exists(example_dags_dir + "__init__.py"):
31-
missing_init_dirs.append(example_dags_dir)
29+
for dags_file in provider_files:
30+
if os.path.isdir(dags_file) and not os.path.exists(os.path.join(dags_file, "__init__.py")):
31+
missing_init_dirs.append(dags_file)
3232

3333
if missing_init_dirs:
3434
with open(os.path.join(ROOT_DIR, "license-templates/LICENSE.txt")) as license:
3535
license_txt = license.readlines()
3636
prefixed_licensed_txt = [f"# {line}" if line != "\n" else "#\n" for line in license_txt]
3737

3838
for missing_init_dir in missing_init_dirs:
39-
with open(missing_init_dir + "__init__.py", "w") as init_file:
39+
with open(os.path.join(missing_init_dir, "__init__.py"), "w") as init_file:
4040
init_file.write("".join(prefixed_licensed_txt))
4141

42-
print("No __init__.py file was found in the following provider example_dags directories:")
42+
print("No __init__.py file was found in the following provider directories:")
4343
print("\n".join(missing_init_dirs))
4444
print("\nThe missing __init__.py files have been created. Please add these new files to a commit.")
4545
sys.exit(1)
4646

4747

4848
if __name__ == "__main__":
49-
all_provider_example_dags_dirs = sorted(
50-
glob(f"{ROOT_DIR}/airflow/providers/**/example_dags/", recursive=True)
51-
)
52-
check_example_dags_dir_init_file(all_provider_example_dags_dirs)
49+
all_provider_subpackage_dirs = sorted(glob(f"{ROOT_DIR}/airflow/providers/**/*", recursive=True))
50+
check_dir_init_file(all_provider_subpackage_dirs)
51+
all_test_provider_subpackage_dirs = sorted(glob(f"{ROOT_DIR}/tests/providers/**/*", recursive=True))
52+
check_dir_init_file(all_test_provider_subpackage_dirs)

‎tests/providers/amazon/aws/config_templates/__init__.py‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.

‎tests/providers/amazon/aws/infrastructure/__init__.py‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.

‎tests/providers/amazon/aws/infrastructure/example_s3_to_redshift/__init__.py‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.

‎tests/providers/amazon/aws/secrets/__init__.py‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.

0 commit comments

Comments
 (0)