Add Python 3.12 support

Finally after a number of dependency upgrades we seem to be able to
upgrade to Python 3.12 (pending universal_pathlib 0.2.0 conversion)

Several providers are excluded from being installed and wait for
Python 3.12, but it should not block Airlfow's general 3.12 support.

Author: dirrao

Dockerfile

@@ -1371,7 +1371,7 @@ ARG USE_CONSTRAINTS_FOR_CONTEXT_PACKAGES="false"
 
 # By changing the epoch we can force reinstalling Airflow and pip all dependencies
 # It can also be overwritten manually by setting the AIRFLOW_CI_BUILD_EPOCH environment variable.
-ARG AIRFLOW_CI_BUILD_EPOCH="10"
+ARG AIRFLOW_CI_BUILD_EPOCH="11"
 ENV AIRFLOW_CI_BUILD_EPOCH=${AIRFLOW_CI_BUILD_EPOCH}
 
 # In case of Production build image segment we want to pre-install main version of airflow

Dockerfile.ci

@@ -954,6 +954,17 @@ function check_download_pendulum() {
     pip check
 }
 
+function uninstall_cassandra_for_python_3_12() {
+    if [[ ${PYTHON_MAJOR_MINOR_VERSION} != "3.12" ]]; then
+        return
+    fi
+    echo
+    echo "${COLOR_BLUE}Uninstalling cassandra in case it was installed via cache${COLOR_RESET}"
+    echo
+    pip uninstall --root-user-action ignore -y "cassandra-driver" || true
+    pip check
+}
+
 function check_run_tests() {
     if [[ ${RUN_TESTS=} != "true" ]]; then
         return
@@ -985,6 +996,7 @@ check_boto_upgrade
 check_pydantic
 check_download_sqlalchemy
 check_download_pendulum
+uninstall_cassandra_for_python_3_12
 check_run_tests "${@}"
 
 exec /bin/bash "${@}"
@@ -1012,7 +1024,7 @@ ARG PYTHON_BASE_IMAGE
 ARG AIRFLOW_IMAGE_REPOSITORY="https://github.com/apache/airflow"
 
 # By increasing this number we can do force build of all dependencies
-ARG DEPENDENCIES_EPOCH_NUMBER="10"
+ARG DEPENDENCIES_EPOCH_NUMBER="11"
 
 # Make sure noninteractive debian install is used and language variables set
 ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE} \

README.md

@@ -100,7 +100,7 @@ Apache Airflow is tested with:
 
 |             | Main version (dev)           | Stable version (2.8.1) |
 |-------------|------------------------------|------------------------|
-| Python      | 3.8, 3.9, 3.10, 3.11         | 3.8, 3.9, 3.10, 3.11   |
+| Python      | 3.8, 3.9, 3.10, 3.11, 3.12   | 3.8, 3.9, 3.10, 3.11   |
 | Platform    | AMD64/ARM64(\*)              | AMD64/ARM64(\*)        |
 | Kubernetes  | 1.25, 1.26, 1.27, 1.28, 1.29 | 1.25, 1.26, 1.27, 1.28 |
 | PostgreSQL  | 12, 13, 14, 15, 16           | 12, 13, 14, 15, 16     |

airflow/__init__.py

@@ -39,7 +39,18 @@
 # configuration is therefore initted early here, simply by importing it.
 from airflow import configuration, settings
 
-__all__ = ["__version__", "DAG", "PY36", "PY37", "PY38", "PY39", "PY310", "PY311", "XComArg"]
+__all__ = [
+    "__version__",
+    "DAG",
+    "PY36",
+    "PY37",
+    "PY38",
+    "PY39",
+    "PY310",
+    "PY311",
+    "PY312",
+    "XComArg",
+]
 
 # Make `airflow` a namespace package, supporting installing
 # airflow.providers.* in different locations (i.e. one in site, and one in user
@@ -61,6 +72,7 @@
 PY39 = sys.version_info >= (3, 9)
 PY310 = sys.version_info >= (3, 10)
 PY311 = sys.version_info >= (3, 11)
+PY312 = sys.version_info >= (3, 12)
 
 # Things to lazy import in form {local_name: ('target_module', 'target_name', 'deprecated')}
 __lazy_imports: dict[str, tuple[str, str, bool]] = {

airflow/providers/apache/cassandra/provider.yaml

@@ -46,6 +46,26 @@ dependencies:
   - apache-airflow>=2.6.0
   - cassandra-driver>=3.13.0
 
+# Cassandra provider is not yet compatible with Python 3.12
+# The main issue is that python cassandra driver by default uses asyncore which has been deprecated since
+# Python 3.6 and removed in Python 3.12 (https://docs.python.org/3.11/library/asyncore.html)
+#
+# Currently the "wheel" package 3.29.0 distributed for manylinux platform is build without libev support (
+# which could be a viable asyncore replacement), and cassandra driver works in Python 3.12 if it is built
+# with libev support (using sdist, having gcc, libev4 and libev-dev installed). But it would be too much to
+# expect our users to build the driver from sources to use it with Python 3.12, so we are waiting for the
+# next release of cassandra-driver which will have libev support in the wheel package.
+# The issue is tracked here https://datastax-oss.atlassian.net/browse/PYTHON-1378
+#
+# Another option to get cassandra drive back is to have asyncio support in the driver instead of asyncore:
+# The issue is tracked here: https://datastax-oss.atlassian.net/browse/PYTHON-1375 and is scheduled
+# to be fixed in cassandra-driver 3.30.0.
+#
+# All Cassandra tests are automatically skipped if cassandra package is not present, so once you remove the
+# exclusion, they will start running for Python 3.12.
+#
+excluded-python-versions: ['3.12']
+
 integrations:
   - integration-name: Apache Cassandra
     external-doc-url: https://cassandra.apache.org/

contributing-docs/07_local_virtualenv.rst

@@ -37,7 +37,7 @@ Required Software Packages
 Use system-level package managers like yum, apt-get for Linux, or
 Homebrew for macOS to install required software packages:
 
-* Python (One of: 3.8, 3.9, 3.10, 3.11)
+* Python (One of: 3.8, 3.9, 3.10, 3.11, 3.12)
 * MySQL 5.7+
 * libxml
 * helm (only for helm chart tests)
@@ -186,6 +186,8 @@ This is what it shows currently:
 +-------------+---------+----------+---------------------------------------------------------------+
 | airflow-311 | virtual | devel    | Environment with Python 3.11                                  |
 +-------------+---------+----------+---------------------------------------------------------------+
+| airflow-312 | virtual | devel    | Environment with Python 3.12                                  |
++-------------+---------+----------+---------------------------------------------------------------+
 
 The default env (if you have not used one explicitly) is ``default`` and it is a Python 3.8
 virtualenv for maximum compatibility with ``devel`` extra installed - this devel extra contains the minimum set

dev/README_RELEASE_AIRFLOW.md

@@ -819,7 +819,7 @@ the older branches, you should set the "skip" field to true.
 ## Verify production images
 
 ```shell script
-for PYTHON in 3.8 3.9 3.10 3.11
+for PYTHON in 3.8 3.9 3.10 3.11 3.12
 do
     docker pull apache/airflow:${VERSION}-python${PYTHON}
     breeze prod-image verify --image-name apache/airflow:${VERSION}-python${PYTHON}

dev/breeze/README.md

@@ -66,6 +66,6 @@ PLEASE DO NOT MODIFY THE HASH BELOW! IT IS AUTOMATICALLY UPDATED BY PRE-COMMIT.
 
 ---------------------------------------------------------------------------------------------------------
 
-Package config hash: e2db123fd25e40b515520fb9f6ed32a601fe85e6a22b9845343bc5c81e5785b472527ed3bf6d07521c512d2222f99877a1ce1911ac504a3a6af7b7866aa674cc
+Package config hash: 9fb8d5c48b1bb2376ac01daf20896fe6a4d7d187cae6db7f629ac65c62e540f7be1dd3c4af17f9b871dc1b163304da38640494267b0b37da278a574ba3274783
 
 ---------------------------------------------------------------------------------------------------------

dev/breeze/doc/ci/02_images.md

@@ -571,7 +571,7 @@ percent-encoded when you access them via UI (/ = %2F)
 
 - \<BRANCH\> might be either "main" or "v2-\*-test"
 - \<X.Y\> - Python version (Major + Minor).Should be one of \["3.8",
-  "3.9", "3.10", "3.11"\].
+  "3.9", "3.10", "3.11", "3.12" \].
 - \<COMMIT_SHA\> - full-length SHA of commit either from the tip of the
   branch (for pushes/schedule) or commit from the tip of the branch used
   for the PR.