PEP: 605
Title: A rolling feature release stream for CPython
Version: $Revision$
Last-Modified: $Date$
Author: Steve Dower <steve.dower@python.org>, Nick Coghlan <ncoghlan@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-605-a-rolling-feature-release-stream-for-cpython/2418
Status: Rejected
Type: Informational
Content-Type: text/x-rst
Created: 20-Sep-2019
Python-Version: 3.9
Post-History: 01-Oct-2019, 06-Oct-2019, 20-Oct-2019
Rejection Notice
================
This PEP was rejected in favour of :pep:`602`. The potential alpha/beta alternation
was deemed too confusing and the two year cadence between releases deemed too
long.
Abstract
========
For a long time, CPython's nominal feature release cadence has been "every 18-24
months", and in recent years, has been pretty consistently on the "18 month"
end of that window. :pep:`607` provides some common background on the problems
that arise due to that choice of cadence, as well as some of the risks that
need to be accounted for when proposing to change it.
The proposal in this PEP aims to allow CPython's user base to self-select into
two distinct but overlapping groups:
* users of stable feature releases (and their associated maintenance release
streams) that are published every 24 months; and
* early adopters of a new rolling release stream that takes the place of the
previous CPython pre-release process
As part of this proposal, the usage guidance given for beta releases would
become "suitable for production use only in environments with sufficiently
robust compatibility testing and operational monitoring capabilities", rather
than the currently unqualified "not for production use".
Similarly, the guidance given for alpha releases would be amended to state
"intended for library compatibility testing and the creation of ABI compatible
binary artifacts", rather than simply saying "not for production use".
The PEP authors believe these outcomes can be achieved by amending CPython's
pre-release management process as described in the Proposal section below.
This PEP also proposes that the frequency of X.Y.0 releases be adjusted to
begin each new release series in August every two years (starting in 2021,
around two years after the release of Python 3.8.0).
Example Future Release Schedules
================================
Under this proposal, Python 3.9.0a1 would be released in December 2019, two
months after the Python 3.8.0 baseline feature release in October 2019.
Assuming no further breaking changes were made to the full CPython ABI, the
3.9.0b2 release would then follow 2 months later in February 2020, continuing
through to 3.9.0b9 in April 2021.
Any time a breaking change to the full CPython ABI was introduced, the first
pre-release that included it would be marked as an alpha release.
3.9.0rc1 would be published in June 2021, 3.9.0rc2 in July 2021, and then
the full release published as 3.9.0 in August 2021.
The cycle would start over again in October 2021, with the publication
of 3.10.0a1 (4 months after the creation of the 3.9.x maintenance branch).
The exact schedule of maintenance releases would be up to the release team,
but assuming maintenance releases of 3.9.x were also to occur every other month
(offset from the 3.10.0 beta releases), the overall release timeline
would look like:
* 2019-12: 3.9.0a1
* 2020-02: 3.9.0b2
* ... beta (or alpha) releases every other month
* 2021-04: 3.9.0b9
* 2021-06: 3.9.0rc1 (feature freeze, ABI freeze, pyc format freeze)
* 2021-07: 3.9.0rc2
* 2021-08: 3.9.0
* 2021-09: 3.9.1, 3.8.x (final 3.8.x binary maintenance release)
* 2021-10: 3.10.0a1
* 2021-11: 3.9.2
* 2021-12: 3.10.0b2
* ... beta (or alpha) and maintenance releases continue in alternate months
* 2023-04: 3.10.0b10
* 2023-05: 3.9.11
* 2023-06: 3.10.0rc1 (feature freeze, ABI freeze, pyc format freeze)
* 2023-07: 3.10.0rc2, 3.9.12
* 2023-08: 3.10.0
* 2023-09: 3.10.1, 3.9.13 (final 3.9.x binary maintenance release)
* 2023-10: 3.11.0a1
* 2023-12: 3.11.0b2
* ... etc
If we assume two additional pre-releases were made that introduced breaking
changes to the full CPython ABI in the 3.9.0a5 and 3.9.0a7 releases, then the
overall calendar would look like:
.. figure:: pep-0605-example-release-calendar.png
:align: center
:width: 100%
Figure 1. Impact of the pre-release process changes on the calendar.
There are always two or three active maintenance branches in this model,
which preserves the status quo in that respect. The major difference is that
we would start encouraging publishers to provide pre-built binaries for the
pre-freeze rolling releases in addition to providing them for the stable
maintenance branches.
.. figure:: pep-0605-overlapping-support-matrix.png
:align: center
:width: 50%
Figure 2. Testing matrix in the 18-month cadence vs. the 24-month
Package publishers targeting the full CPython ABI that choose to provide
pre-built binaries for the rolling pre-freeze releases would at least need
to build new wheel archives following the 3.9.0a1 release. Whether they needed
to publish updated binaries after subsequent alpha releases (e.g. 3.9.0a5 or
3.9.0a7 releases in the example timeline) would depend on whether or not they
were actually affected by the ABI breaks in those later releases.
As with the status quo, all package publishers wishing to provide pre-built
binaries for the final release will need to build new wheel archives following
the ABI freeze date. Unlike the status quo, this date will be clearly marked
by the publication of the first release candidate, and it will occur early
enough to give publishers a couple of months to get ready for the final release.
Example Future Release Announcements
====================================
If this PEP is accepted, the primary channels used to communicate the updated
pre-release management process to end users would be the Python 3.9 What's New
document, and the release announcements themselves.
This section provides initial drafts of text that could be used for those
purposes.
Suggested "What's New in Python 3.9" Entry
------------------------------------------
The following subsection would be added to the Python 3.9 What's New document,
and then linked from each of the Python 3.9 alpha and beta announcements.
PEP 605: Changes to the pre-release management process
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
As detailed in :pep:`605`, the pre-release management process has been updated to
produce a rolling series of beta releases that are considered suitable for
production use in environments with sufficiently robust integration testing and
operational monitoring capabilities.
Under this new rolling model, the alpha and beta releases are intermingled as
part of a combined "pre-freeze" period, with alpha releases indicating breaks
in the full CPython ABI that may require recompilation of extension modules or
embedding applications, and beta releases indicating full binary compatibility
with the immediately preceding pre-release.
Unlike previous releases, publishing pre-built binaries for 3.9.0 alpha and beta
releases is actively encouraged, as a new pre-release ABI flag ("p") is now
set when building and loading extension modules prior to the full CPython ABI
freeze, ensuring that all such pre-freeze extension module builds will be
ignored by post-freeze interpreter builds.
The full CPython ABI will be frozen, and the pre-release flag dropped from the
ABI flags, in 3.9.0rc1, which is expected to occur 2 months prior to the final
3.9.0 release (refer to the release schedule in :pep:`596` for exact target dates).
For application developers, migrating to the rolling release stream provides
the opportunity to be actively involved in the design and development of
enhancements to the standard library and reference interpreter prior to the
next stable release. It also provides the opportunity to benefit from
interpreter performance enhancements up to a year or more before they become
available in a stable release.
For library developers that publish pre-built wheel archives, opting in to
supporting the 3.9.x rolling release stream in addition to the 3.8 stable
release series requires no specific action if the project is already publishing
either pure Python wheels (tagged as ``py3-none-any``), or builds against the
stable C ABI (tagged as ``cp38-abi3-<platform>``, or the equivalent from an
earlier CPython 3.x release). These same wheel archives will also be usable on
the subsequent 3.9 stable release series.
For library developers that publish pre-built wheel archives that are built
against the full CPython ABI, the binaries for the 3.9 stable release series
will need to be built after the full CPython ABI freeze (i.e. using 3.9.0rc1 or
later).
Developers of these libraries may also opt in to supporting the rolling release
stream by building against the 3.9.0a1 release (or a subsequent beta release)
and publishing the result as normal.
In the ideal case, binaries built this way will continue working all the way
through to the last pre-freeze release. However, if the project is affected by
a change in the full CPython C ABI during the pre-freeze period, then it will
be necessary to publish a maintenance update that rebuilds the affected binaries
against the alpha release that changed the relevant interface. In these cases,
a corresponding ``Python-Requires`` entry should be added to the project
metadata. For example, if a project is affected by an ABI change introduced in
``3.9.0a5``, then the ``Python-Requires`` entry to add would be::
Python-Requires: >= "3.9.0b6"; python_version == "3.9" and full_python_version != "3.9.0a5"
(This additional metadata ensures that the updated version won't be installed on
earlier pre-releases in the 3.9 series that offer an older variant of the ABI)
As for application developers, library developers that choose to support the
rolling release stream will have the opportunity to provide feedback on new and
updated API designs *before* they're locked down for multiple years in a stable
release (or before they're included as a provisional API in a stable release
series).
Example announcement text for the 3.9.0a1 release
-------------------------------------------------
This is the first preview release of Python 3.9. As an alpha release, it is
intended for library and application compatibility testing and the creation of
ABI compatible binary artifacts. It is not recommended for use in production
environments.
Changes to the pre-release management process
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
CPython has switched to a new pre-release management process that is designed
to produce a rolling series of beta releases that are considered suitable for
production use in environments with sufficiently robust integration testing and
operational monitoring capabilities. Refer to the Python 3.9 What's New
document (hyperlinked to relevant section) for details.
Major new features of the 3.9 series, compared to 3.8
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Many new features for Python 3.9 are still being planned and written. Among the
major new features and changes already implemented:
* ...
* (Hey, fellow core developer or user of the rolling release stream, if a
feature you find important is missing from this list, let <the release
manager> know.)
The next pre-release of Python 3.9 is expected to be 3.8.0b2, currently scheduled for 2020-02-02.
Example announcement text for the 3.9.0b2 release
-------------------------------------------------
This is the second preview release of Python 3.9. As a beta release, it is
fully binary compatible with the preceding 3.9.0a1 release. It is recommended
for production use only in environments with sufficiently robust integration
testing and operational monitoring capabilities.
(Remainder as per 3.9.0a1 announcement, with updates for implemented changes
and the next expected release being 3.9.0b3)
Example announcement text for 3.9.0a5 (a mid-stream alpha release)
------------------------------------------------------------------
This is the fifth preview release of Python 3.9. As an alpha release, it is
NOT fully binary compatible with the preceding 3.9.0b4 release. This release is
intended for library and application compatibility testing and the creation of
ABI compatible binary artifacts. It is not recommended for use in production
environments.
Breaking changes in the full CPython ABI between 3.9.0b4 and 3.9.0a5
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* new field ``ob_example`` added to the ``PyObject`` struct
* provisional field ``tp_example`` removed from the ``PyTypeObject`` struct
Projects that are supporting the rolling release stream and require a rebuild
to restore binary compatibility should add the following metadata to their
updated release::
Python-Requires: >= "3.9.0b6"; python_version == "3.9" and full_python_version != "3.9.0a5"
(Remainder as per 3.9.0a1 announcement, with updates for implemented changes
and the next expected release being 3.9.0b6)
Example announcement text for 3.9.0rc1
--------------------------------------
This is the first release candidate for Python 3.9. As a release candidate,
this release is now feature complete, the full CPython ABI is now frozen, and
the pre-release marker has been removed from the ABI compatibility flags. It is
recommended for production use only in environments with sufficiently robust
integration testing and operational monitoring capabilities.
Preparation for the final 3.9.0 release
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
With the full CPython ABI now frozen, library developers targeting that ABI are
encouraged to build and publish binaries for the stable 3.9.x series.
Application developers that have not been testing against the rolling release
stream are encouraged to test their applications against the release candidate
and report any compatibility regressions not already mentioned in the Porting
Guide (hyperlinked to relevant What's New section).
A second release candidate is planned for 2021-07-02, and then the final 3.9.0
release is planned for 2021-08-02.
Major new features of the 3.9 series, compared to 3.8
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Some of the major new features and changes in this release:
* ...
* (Hey, fellow core developer or user of the rolling release stream, if a
feature you find important is missing from this list, let <the release
manager> know.)
Motivation
==========
The current CPython pre-release and release management processes were developed
in an era where automated continuous integration and operational monitoring
systems were still relatively immature. Since that time, many organisations
have adopted deployment models that allow them to incorporate new CPython
feature releases without adding substantially more risk than they incur for any
other code change. Newer deployment models, such as lightweight task specific
application containers, also make it easier to combine an application with a
language runtime in a CI pipeline, and then keep them together until the entire
container image is later replaced by an updated one.
In light of those changes in the wider environment, :pep:`602` has proposed
reducing the feature delivery latency for the Python standard library and
CPython reference interpreter by increasing the frequency of CPython feature
releases from every 18-24 months to instead occur every 12 months.
Unfortunately, for many organisations, the cost of adopting a new Python release
doesn't automatically scale down with a reduced number of changes in the release,
as the primary costs aren't associated with resolving any discovered issues;
the primary costs are associated with the *search* for issues. This search may
involve manual testing of software systems, human review of written materials,
and other activities where the time required scales with the size of the
existing system, rather than with the number of changes between the versions of
Python.
For third party library developers, costs are primarily associated with the
*number* of distinct Python versions in widespread usage. This currently tends
to be influenced by a combination of which releases are still actively
maintained by python-dev, and which releases are the latest versions offered
by particular redistributors (with the Debian, Ubuntu LTS, and RHEL/CentOS
system Python versions being particularly popular development targets). In
addition to the basic CI cost of testing against more Python versions, having
more variants in widespread use can make it more difficult to determine when a
fault report is an actual error in the project, or an issue in the reporting
user's environment.
:pep:`602` proposes that affected organisations and projects simply switch to
adopting every second or third CPython release, rather than attempting to adopt
every release, but that creates its own set of new problems to be resolved, both
practical (e.g. deprecations would need to cover more than one release if we're
expecting users to routinely skip releases) and cultural (e.g. with a larger
number of versions in active use, there is a much higher chance that open source
library maintainers will receive bug reports that only occur on Python versions
that they're not using themselves).
:pep:`598` was an initial attempt by one of the authors of this PEP to propose
an alternative scheme to reduce feature delivery latency by adopting a
semantic versioning style policy that allowed for the incremental delivery of
backwards compatible features within a release series, until that series
reached feature complete status. That variant still had the undesirable
consequence of imposing visible changes on end users that are happy enough
with the current release management model.
This PEP takes the view that both :pep:`598` and :pep:`602` share a common flaw: they
are attempting to satisfy the needs of two quite distinct audiences within the
constraints of a single release model, which results in conflicting design
requirements, and the need for awkward trade-offs between those conflicting
requirements. The proposal in this PEP aims to avoid that flaw by proposing the
creation of two *distinct* production-ready release streams, with the existing
release stream being largely left alone, while the new release stream is
tailored towards the audience that would most benefit from a reduction in
feature delivery latency.
Aims of this Proposal
=====================
The proposal in this PEP arises from making the following key assumptions:
* the vast majority of Python's users aren't actively clamouring for new
language and runtime level features, and instead only upgrade when either
the version they were previously using is no longer supported, when their
Python provider switches to offering them a newer version by default, or when
enough changes that are of interest to them have accumulated to make a
compelling case for upgrading
* for many users of new releases, much of the work that arises when adopting a
new release doesn't arise from compatibility issues at the language level, but
from compatibility issues at the component installation level (i.e. filename
and installation path changes)
* that there's a subset of Python's user base that would be willing to run
production-ready pre-releases (akin to Windows Insider or Debian testing
builds) for at least some of their use cases
The core of the proposal in this PEP is changing the CPython pre-release process
to produce a rolling stream of incremental feature releases at a regular
cadence, and to ensure that most of those builds offer a sufficient level of
stability as to be suitable for use in appropriately managed production systems.
By adopting this approach, the proposal aims to provide an improved outcome
for almost all Python users and contributors:
* for users of the new incremental feature release stream, targeting the
pre-release phase allows for even lower feature delivery latency than the
annual cadence proposed in :pep:`602`;
* for core developers working on new features, increased frequency and adoption
of pre-releases should improve pre-release feedback cycles;
* for users of the established release stream, the increased adoption and
improved feedback cycles during the pre-release period should result in
increased feature maturity at the time of its first X.Y.0 release, as well
as higher levels of ecosystem readiness;
* for Python library maintainers, the rolling stream of pre-releases will
hopefully provide more opportunities to identify and resolve design issues
before they make it into a full stable release than is offered by the current
pre-release management process; and
* for developers of alternative Python implementations, the rolling stream of
pre-releases may provide an additional incentive for extension module authors
to migrate from the full CPython ABI to the Python stable ABI, which would
also serve to make more of the ecosystem compatible with implementations that
don't emulate the full CPython C API.
That said, it is acknowledged that not all the outcomes of this proposal will be
beneficial for all members of the wider Python ecosystem:
* for Python library maintainers, both this PEP and :pep:`602` would likely
result in user pressure to support the faster release cadence. While this PEP
attempts to mitigate that by clearly marking which pre-releases include
potentially breaking changes to the full CPython C ABI, and :pep:`602` attempts
to mitigate it by keeping the minimum time between full releases at
12 months, it isn't possible to eliminate this downside completely;
* for third party extension module maintainers, both this PEP and :pep:`602` would
likely result in user pressure to start supporting the stable ABI in order to
provide wheel archives that work on the new version as soon as it is
available. Whether that's a net negative or not will depend on how the request
is presented to them (it could be a positive if the request comes in the form
of a courteous contribution to their project from a developer interested in
supporting the rolling pre-freeze releases);
* for some users of the established release stream that rely on the
availability of pre-built wheel archives, switching to adopting a new release
every 12 months may be an acceptable rate increase, while moving consistently
to the 24 month end of the historical 18-24 month cadence would be an
undesirable rate reduction relative to the 18-month cycle used for recent
releases. Whether this proposal would be a net negative for these users will
depend on whether or not we're able to persuade library maintainers that
it's worth their while to support the upcoming stable release throughout its
pre-freeze period, rather than waiting until its API and ABI have been
frozen.
Proposal
========
The majority of the proposed changes in this PEP only affect the handling of
pre-release versions. The one change affecting full release versions is a
suggested change to their cadence.
Two year cadence for stable releases
------------------------------------
With the rolling pre-freeze releases available to users that are looking to
use leading edge versions of the reference interpreter and standard library,
this PEP proposes that the frequency of X.Y.0 releases be adjusted to publish
a new stable release in August every two years (starting in 2021,
around two years after the release of Python 3.8.0).
This change is arguably orthogonal to the proposed changes to the handling of
the pre-freeze release period, but the connection is that without those
pre-release management changes, the downsides of a two-year full release cadence
would probably outweigh the upsides, whereas the opposite is true for a
12-month release cadence (i.e. with the pre-release management changes proposed
in this PEP in place, the downsides of a 12-month full release cadence would
outweigh the upsides).
Merging of the alpha and beta phases into a "pre-freeze" phase
--------------------------------------------------------------
Rather than continuing the status quo where the pre-release alpha and beta
phases are distinct and sequential, this PEP proposes that they instead be
combined into a single "pre-freeze" phase with a monotonically increasing serial
number on the releases.
Rather than denoting distinct phases, the "alpha" and "beta" names would
instead indicate whether or not the release contains breaking changes to the
full CPython C ABI:
* "alpha" releases would be "ABI breaking" releases where extension modules
built against the full CPython ABI in the preceding pre-release are not
necessarily going to load correctly
* "beta" releases would be "binary compatible" releases, where extension modules
built against the full CPython ABI in the preceding pre-release are expected
to load correctly, as long as those modules abide by the following additional
criteria:
* the module must not be using any provisional or private C APIs (either from
the previous stable release series, or the in development pre-release series)
that were removed in this beta release, or were changed in an ABI incompatible
way
* the module must not be using any C APIs that were deprecated in the previous
stable release series, and removed in this beta release
Pre-freeze phase duration and cadence
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Rather than being released monthly for a period of a few months while preparing
a new X.Y.0 release, pre-freeze releases would instead be consistently published
every two months.
The only time this would not be the case is during the two month release
candidate period for an upcoming X.Y.0 release (see the release candidate
section below for more details). This means two otherwise scheduled releases
would be skipped (one corresponding with the first release candidate date, one
with the final release date).
The pre-freeze phase would typically be expected to start 2 months after the
preceding stable X.Y.0 release.
The first pre-freeze release for any new release series will always be X.Y.0a1
(as there is no preceding release with the same ABI version markers to judge
binary compatibility against).
Pre-freeze releases would gain an additional flag in their C ABI compatibility
markers to avoid binary compatibility issues with the eventual stable release.
Release policy for beta releases
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This PEP proposes that the policy for beta releases be set as follows:
* as with current beta releases, the stable BuildBot fleet is expected to be
green prior to preparation and publication of the beta release
* as with current beta releases, the release manager is expected to review
open release blocker issues prior to preparation and publication of the beta
release
* as with current beta releases, any additions to the ``abi3`` stable C ABI would
be expected to become a permanent part of that ABI unless and until that
stable ABI version is retired completely (Note: there are no current plans
to increment the stable ABI version)
* unlike current beta releases, beta releases under this PEP would *not* be
considered feature complete for the next X.Y.0 release
* unlike current beta releases, all APIs added since the last CPython feature
release (other than additions to the stable C ABI) would be considered
provisional
* unlike current beta releases, beta releases under this PEP would be prepared
and published from the master development branch
* unlike current alpha or beta releases, beta releases under this PEP would be
required to be fully ABI compatible with the immediately preceding pre-release
in the series (excluding any changes to provisional APIs, or the removal of
APIs that were deprecated in the previous release series)
Release policy for alpha releases
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This PEP proposes that the policy for alpha releases be set as follows:
* as with current alpha releases, the stable BuildBot fleet is expected to be
green prior to preparation and publication of the alpha release
* as with current alpha releases, the release manager is expected to review
open release blocker issues prior to preparation and publication of the beta
release
* unlike current alpha release, the release manager would be expected to
target a similar level of stability to the current beta releases, even
for the alpha releases
Under this PEP, an alpha release would be published whenever it isn't possible
to publish a release that satisfies the criteria for a beta release, and
allowing some additional time before making the release won't resolve the issue.
It is expected that the full CPython API changing in a way that breaks ABI
compatibility (for example, a field may have been added to or removed from a
public struct definition) will be the most likely reason for publishing
additional alpha releases beyond the initial compatibility tag defining
X.Y.0a1 release, but the decision for any particular release rests with the
release manager.
Release candidate policy, phase duration, and cadence
-----------------------------------------------------
Given the proposed changes to the alpha and beta release phases, the release
candidate phase would see the following related adjustments:
* Feature freeze, ABI freeze, pyc file format freeze, and maintenance branch
creation would all correspond with the creation of X.Y.0rc1 (currently these
occur across a mixture of X.Y.0b1, the last beta release, and X.Y.0rc1)
* The X.Y.0 release candidate period would be extended from 3 weeks to 2 months
* There would normally be two release candidates issued a month apart, but
additional candidates may be published at the release manager's discretion
* The final X.Y.0 release would occur between 1 and 4 weeks after the final
release candidate (depending if additional release candidates were needed
after the second)
* If the final X.Y.0 release is delayed beyond the August target date, the
subsequent release series is not affected, and will still be scheduled for
August (now slightly less than two years later).
In addition to allowing more time for end user feedback on the release
candidate, this adjusted policy also provides additional time for maintainers
of Python projects to build and publish pre-built wheel archives for the new
stable release series, significantly improving the initial user experience of
the X.Y.0 release.
Changes to management of the CPython stable C ABI
-------------------------------------------------
The CPython stable ABI [5]_ makes the commitment that binary extension modules
built against any particular CPython release will continue to work on future
CPython releases that support the same stable ABI version (this version is
currently ``abi3``).
Under the proposed rolling pre-freeze release model, this commitment would be
extended to also apply to the beta releases: once an intentional addition to the
``abi3`` stable ABI for the upcoming Python version has been shipped in a beta
release, then it will not be removed from future releases for as long as the
``abi3`` stable ABI remains supported.
Two main mechanisms will be available for obtaining community feedback on
additions to the stable ABI:
* the preferred mechanism will be to add new APIs to the full CPython API first,
and only promote them to the stable ABI after they have been included in at
least one published beta release and received relevant user feedback
* for APIs where that approach is unavailable for some reason (e.g. some API
additions may serve no useful purpose when the full CPython API is available),
then developers may request that the release manager mark the next release
as an alpha release (even in the absence of an ABI break in the full CPython
API), and attempt to obtain further feedback that way
As a slight readability and usability improvement, this PEP also proposes the
introduction of aliases for each major stable ABI version::
#define Py_LIMITED_API_3_3 0x03030000
#define Py_LIMITED_API_3_4 0x03040000
#define Py_LIMITED_API_3_5 0x03050000
#define Py_LIMITED_API_3_6 0x03060000
#define Py_LIMITED_API_3_7 0x03070000
#define Py_LIMITED_API_3_8 0x03080000
#define Py_LIMITED_API_3_9 0x03090000
// etc...
These would be used both in extension module code to set the target ABI
version::
#define Py_LIMITED_API Py_LIMITED_API_3_8
And also in the CPython interpreter implementation to check which symbols should
be made available::
#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= Py_LIMITED_API_3_9
// A Python 3.9+ addition to the stable ABI would appear here
#endif
The documentation for the rolling pre-freeze releases and the stable C ABI would
make it clear that extension modules built against the stable ABI in a later
pre-freeze release may not load correctly on an earlier pre-freeze release.
The documentation for alpha releases and the stable C ABI would make it clear
that even extension modules built against the stable ABI in an alpha release
may not load correctly on the next release if two alpha releases are published
in a row (this situation would ideally be rare).
Changes to management of the full CPython ABI
---------------------------------------------
This PEP proposes two changes to the management of the full CPython ABI.
An explicit commit and NEWS file convention to mark ABI breaking changes
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The proposal in this PEP requires that release managers be able to appropriately
mark a pre-freeze release as either an alpha or a beta release based on whether
or not it includes an ABI breaking change.
To assist in that process, core developers would be requested to include a
"(CPython ABI break)" marker at the beginning of all NEWS file snippets for
changes that introduce a breaking change in the full CPython C ABI.
The "CPython" marker is included to make it clear that these annotations relate
to the full CPython ABI, not the stable ABI.
For commit messages, the shorter marker "(ABI break)" would be placed at the
start of the summary line for the commit.
The pre-merge bots will be updated to ensure that if the ABI break marker
appears in one of the two locations, it appears in both of them.
If the marker is inadvertently omitted from the initial commit message and NEWS
entry, then the commit message marker should be included in the subsequent
commit that adds the marker to the NEWS entry.
In addition to being useful for release managers, these markers should also be
useful for developers investigating unexpected segfaults when testing against
the affected release.
Explicitly marking builds against the pre-freeze ABI
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The full CPython ABI has long operated under a policy where binary
compatibility only applies within a release series after the ABI has been
declared frozen, and only source compatibility applies between different
release series.
This policy means that extension modules built against CPython pre-releases
prior to the ABI freeze for that release series may not actually load correctly
on the final release.
This is due to the fact that the extension module may be relying on provisional
or previously deprecated interfaces that were changed or removed in a later
alpha or beta release, or it may be due to public structures used by the
extension module changing size due to the addition of new fields.
Historically, adoption of alpha and beta releases has been low enough that this
hasn't really been a problem in practice. However, this PEP proposes to actively
encourage widespread operational use of beta releases, which makes it desirable
to ensure that users of those releases won't inadvertently publish binary
extension modules that cause segfaults for users running the release candidates
and final releases.
To that end, this PEP proposes amending the extension module ``SOABI`` marker
on non-Windows systems to include a new "p" flag for CPython pre-releases, and
only switch back to omitting that flag once the ABI for that particular X.Y.0
version has been frozen on entry to the release candidate stage.
With this change, alpha and beta releases of 3.9.0 would get an SOABI tag of
``cpython-39p``, while all release candidates and final builds (for both 3.9.0
and later 3.9.x releases) would get an unqualified SOABI tag of ``cpython-39``
Debug builds would still add the "d" to the end of the tag, giving
``cpython-39pd`` for debug builds of pre-releases.
On Windows systems, the suffix for tagged ``pyd`` files in pre-release builds
would include "p" as a pre-release marker immediately after the version number,
giving markers like "cp39p-win_amd64".
A proposed reference implementation for this change is available at [4]_ (Note:
at time of writing, that implementation had not yet been tested on Windows).
Updating Python-Requires for projects affected by full C ABI changes
--------------------------------------------------------------------
When a project first opts in to providing pre-built binary wheels for the
rolling pre-freeze release series, they don't need to do anything special: they
would add the rolling release series to their build and test matrices and
publish binary archives that are flagged as being compatible with that release
series, just as they would if providing pre-built binary wheels after the
full CPython ABI freeze for that release series.
However, if the project is affected by a CPython ABI compatibility break in the
rolling release stream, then they will need to issue a version update that
includes both the new binary build, and a new environment constrained
``Python-Requires`` marker.
For example, if a project supporting the rolling release stream was affected by
a CPython ABI compatibility break in the 3.9.0a5 release, then they would add
the following metadata entry on the version that published the updated binary
build::
Python-Requires: >= "3.9.0b6"; python_version == "3.9" and full_python_version != "3.9.0a5"
What this does is add an additional compatibility constraint as part of the
published packages, so Python 3.9.0 beta versions prior to 3.9.0b6 won't
consider the updated package as a candidate for installation, and the only
alpha release that will consider the package is 3.9.0a5 itself.
Caveats and Limitations
=======================
Actual release dates may be scheduled up to a month earlier or later at
the discretion of the release manager, based on release team availability, and
the timing of other events (e.g. PyCon US, or the annual core developer
sprints). However, as one goal of the proposal is to provide a consistent
release cadence, adjustments should ideally be rare.
Within a release series, the exact frequency of maintenance releases would
still be up to the release manager and the binary release team; this PEP
only proposes an expected cadence for pre-releases and X.Y.0 releases.
However, for the sake of the example timelines, the PEP assumes maintenance
releases every other month, allowing them to alternate months with the rolling
pre-freeze releases.
The release manager and Steering Council would also retain the power to amend
various details of the proposal in this PEP. Possible amendments include (but
are not limited to):
* changing the timing for creation of the maintenance branch. If a major change
that would require a new alpha release is landed relatively late in the
pre-release process, the release manager could potentially choose to branch
off from a point prior to that major change. For example, it may make sense to
do this if the next scheduled release was intended to be the final beta
release or the first release candidate.
* the criteria for declaring an alpha release could potentially be expanded to
include all changes that require a "Porting" entry in the What's New document
* rather than declaring alpha releases on an as-needed basis, the release
manager could declare some dates as alpha releases in advance, and ask core
developers to time their higher risk changes accordingly.
The intent of the concrete proposal in the PEP is to provide a clear
illustrative example for reviewers to consider, not to constrain our ability
to adjust specific details based on practical experience with the process.
Design Discussion
=================
Why rolling pre-freeze releases over simply doing more frequent X.Y.0 releases?
-------------------------------------------------------------------------------
For large parts of Python's user base, *availability* of new CPython feature
releases isn't the limiting factor on their adoption of those new releases
(this effect is visible in such metrics as PyPI download metadata).
As such, any proposal based on speeding up full feature releases needs to strike
a balance between meeting the needs of users who would be adopting each release
as it became available, and those that would now be in a position of adopting
every 2nd, 3rd, or 4th release, rather than being able to migrate to almost
every release at some point within its lifecycle.
This proposal aims to approach the problem from a different angle by defining a
*new* production-ready release stream that is more specifically tailored to the
interests of operating environments that are able to consume new releases as
fast as the CPython core team is prepared to produce them.
Is it necessary to keep the "alpha" and "beta" naming scheme?
-------------------------------------------------------------
Using the "a" and "b" initials for the proposed rolling releases is a design
constraint imposed by some of the pragmatic aspects of the way CPython version
numbers are published.
Specifically, alpha releases, beta releases, and release candidates are reported
in some places using the strings "a", "b", and "c" respectively, while in others
they're reported using the hex digits ``0xA``, ``0xB``, and ``0xC``. We want to
preserve that, while also ensuring that any ``Python-Requires`` constraints
are expressed against the beta releases rather than the alpha releases (since
the latter may not enforce the ``abi3`` stability requirements if two alpha
releases occur in succession).
However, there isn't anything forcing us to say that the "a" stands for "alpha"
or the "b" stands for "beta".
That means that if we wanted to increase adoption amongst folks that were
only being put off by the "beta" label, then it may make sense to emphasise
the "\*A\*BI breaking" and "\*B\*inary compatible" names over the "alpha"
and "beta" names, giving:
* 3.9.0a1: ABI breaking pre-freeze release
* 3.9.0b2: binary compatible pre-freeze release
* 3.9.0rc1: release candidate
* 3.9.0: final release
This iteration of the PEP doesn't go that far, as limiting initial adoption
of the rolling pre-freeze releases to folks that are comfortable with the
"beta" label is likely to be a good thing, as it is the early adopters of these
releases that are going to encounter any unexpected consequences that occur
at the level of the wider Python ecosystem, and we're going to need them to
be willing to take an active part in getting those issues resolved.
Moving away from the "beta" naming would then become an option to keep in mind
for the future, assuming the resulting experience is sufficiently positive that
we decide the approach is worth continuing.
Why rolling pre-freeze releases rather than alternating between stable and unstable release series?
---------------------------------------------------------------------------------------------------
Rather than using the beta period for rolling releases, another option would be
to alternate between traditional stable releases (for 3.8.x, 3.10.x, etc), and
release series that used the new rolling release cadence (for 3.9.x, 3.11.x,
etc).
This idea suffers from the same core problem as :pep:`598` and :pep:`602`: it imposes
changes on end users that are happy with the status quo without offering them
any clear compensating benefit.
It's also affected by one of the main concerns raised against :pep:`598`: at least
some core developers and end users strongly prefer that no particular semantics
be assigned to the *value* of any of the numbers in a release version. These
community members instead prefer that all the semantic significance be
associated with the *position* within the release number that is changing.
By contrast, the rolling pre-freeze release proposal aims to address that concern
by ensuring that the proposed changes in policy all revolve around whether a
particular release is an alpha release, beta release, release candidate, or
final release.
Why not use Calendar Versioning for the rolling release stream?
---------------------------------------------------------------
Steve Dower's initial write-up of this proposal [1]_ suggested the use of
calendar versioning for the rolling release stream (so the first rolling
pre-release after Python 3.8.0 would have been Python 2019.12 rather than
3.9.0b1).
Paul Moore pointed out [2]_ two major practical problems with that proposal:
* it isn't going to be clear to users of the calendar-based versions where they
stand in relation to the traditionally numbered versions
* it breaks ``Python-Requires`` metadata processing in packaging tools with
no clear way of fixing it reliably (since all calendar versions would appear
as newer than any standard version)
This PEP aims to address both of those problems by using the established beta
version numbers for the rolling releases.
As an example, consider the following question: "Does Python 2021.12 include
all the new features released in Python 3.9.0?". With calendar versioning on
the rolling releases, that's impossible to answer without consulting a release
calendar to see when 3.9.0rc1 was branched off from the rolling release series.
By contrast, the equivalent question for rolling pre-freeze releases is
straightforward to answer: "Does Python 3.10.0b2 include all the new features
released in Python 3.9.0?". Just from formulating the question, the answer is
clearly "Yes, unless they were provisional features that got removed".
The beta numbering approach also avoids other questions raised by the calendar
versioning concept, such as how ``sys.version_info``, ``PY_VERSION_HEX``,
``site-packages`` directory naming, and installed Python binary and extension
module naming would work.
How would users of the rolling pre-freeze releases detect API changes?
----------------------------------------------------------------------
When adding new features, core developers would be strongly encouraged to
support feature detection and graceful fallback to alternative approaches via
mechanisms that don't rely on either ``sys.version_info`` or runtime code object
introspection.
In most cases, a simple ``hasattr`` check on the affected module will serve this
purpose, but when it doesn't, alternative approaches would be considered as part
of the feature addition. Prior art in this area includes the
``pickle.HIGHEST_PROTOCOL`` attribute, the ``hashlib.algorithms_available`` set,
and the various ``os.supports_*`` sets that the ``os`` module already offers for
platform dependent capability detection.
It would also be possible to add features that need to be explicitly enabled
via a ``__future__`` import when first included in the rolling pre-freeze releases,
even if that feature flag was subsequently enabled by default before its first
appearance in an X.Y.0 release candidate.
The rationale behind these approaches is that explicit detection/enabling like
this would make it straightforward for users of the rolling pre-freeze release
stream to notice when we remove or change provisional features
(e.g. ``from __future__`` imports break on compile if the feature flag no
longer exists), or to safely fall back on previous functionality.
The interpreter's rich attribute lookup machinery means we can also choose to
add warnings for provisional or deprecated imports and attributes that we don't
have any practical way to add for checks against the value of
``sys.version_info``.
Why add a new pre-freeze ABI flag to force rebuilds after X.Y.0rc1?
-------------------------------------------------------------------
The core development team currently actively *discourage* the creation of
public pre-built binaries for an X.Y series prior to the ABI freeze date.
The reason we do that is to avoid the risk of painful debugging sessions
on the stable X.Y.0 release that get traced back to "Oh, our dependency
'superfast-binary-operation' was affected by a CPython ABI break in
X.Y.0a3, but the project hasn't published a new build since then".
With the proposed pre-freeze ABI flag in place, this aspect of the
release adoption process continues on essentially unchanged from the
status quo: a new CPython X.Y release series hits ABI freeze -> package
maintainers publish new binary extension modules for that release
series -> end users only get segfaults due to actual bugs, not just
builds against an incompatible ABI.
The primary goal of the new pre-freeze ABI flag is then to improve
the user experience of the rolling pre-freeze releases themselves, by
allowing pre-built binary archives to be published for those releases
without risking the problems that currently cause us to actively
discourage the publication of binary artifacts prior to ABI freeze.
In the ideal case, package maintainers will only need to publish
one pre-freeze binary build at X.Y.0a1, and then a post-freeze
build after X.Y.0rc1. The only situations that should *require*
a rebuild in the meantime are those where the project was
actually affected by a CPython ABI break in an intervening alpha
release.
As a concrete example, consider the scenario where we end up having three
releases that include ABI breaks: X.Y.0a1, X.Y.0a5, X.Y.0a7. The X.Y.0a7 ABI is
then the ABI that carries through all the subsequent beta releases and into
X.Y.0rc1. (This is the scenario illustrated in figure 1)
Forcing everyone to rebuild the world every time there’s an alpha release in
the rolling release stream would almost certainly lead to publishers deciding
supporting the rolling releases was more trouble than it was worth, so we want
to allow modules built against X.Y.0a1 to be loaded against X.Y.0a7, as they’re
*probably* going to be compatible (there are very few projects that use every
C API that CPython publishes, and most ABI breaks affect a single specific API).
Once we publish X.Y.0rc1 though, we want to ensure that any binaries that were
built against X.Y.0a1 and X.Y.0a4 are completely removed from the end user
experience. It would be nice to be able to keep the builds against X.Y.0a7 and
any subsequent beta releases (since it turned out those actually were built
against the post-freeze ABI, even if we didn’t know that at the time), but
losing them isn’t any *worse* than the status quo.
This means that the pre-freeze flag is “the simplest thing that could possibly
work” to solve this problem - it’s just a new ABI flag, and we already have
the tools available to deal with ABI flags (both in the interpreter and in
package publication and installation tools).
Since the ABI flags have changed relative to the pre-releases, projects don't
even need to publish a new release: they can upload new wheel archives to their
existing releases, just as they can today.
A cleverer scheme that was able to retroactively accept everything built
against the last alpha or subsequent beta releases would likely be possible,
but it isn't considered *necessary* for adoption of this PEP, as even if we
initially start out with a simple pre-release ABI flag, it would still be
possible to devise a more sophisticated approach in the future.
Why allow additional alpha releases after X.Y.0a1?
--------------------------------------------------
In an ideal world, all breaking changes to the full CPython ABI would land in
X.Y.0a1 alongside the filesystem layout changes, and the ABI for the release
series would remain stable after that.
However, recent history doesn't suggest that we'd be able to actually make that
commitment and stick to it, so the PEP assumes that ABI changes will be made
progressively throughout the pre-freeze period, and the full lockdown will occur
only with the creation of the X.Y.z maintenance branch when preparing X.Y.0rc1.
Implications for CPython core development
-----------------------------------------
The major change for CPython core development is the need to keep the master
branch more consistently release ready.
While the main requirement for that would be to keep the stable BuildBot fleet
green, there would also be encouragement to keep the development version of
the documentation up to date for the benefit of users of the rolling pre-freeze
releases. This will include providing draft What's New entries for changes as
they are implemented, although the initial versions may be relatively sparse,
and then expanded based on feedback from beta release users.
For core developers working on the CPython C API, there would also be a new
requirement to consistently mark ABI breaking changes in their NEWS file
snippets.
On the specific topic of the stable ABI, most API designs will be able to go
through a process where they're first introduced as a provisional part of the
full CPython API (allowing changes between pre-freeze releases), and only
promoted to the stable ABI once developers are confident that the interface
is genuinely stable.
It's only in rare cases where an API serves no useful purpose outside the
stable ABI that it may make sense to publish an alpha release containing a
provisional stable ABI addition rather than iterating on the design in the
provisional CPython API instead.
Implications for Python library development
-------------------------------------------
If this PEP is successful in its aims, then supporting the rolling pre-freeze
release stream shouldn't be subtantially more painful for library authors than
supporting the stable releases.
For publishers of pure Python packages, this would be a matter of publishing
"py3" tagged wheel archives, and potentially adding the rolling pre-freeze
release stream to their test matrix if that option is available to them.
For publishers of binary extension modules, the preferred option would be to
target the stable C ABI (if feasible), and thus enjoy an experience similar to
that of pure Python packages, where a single pre-built wheel archive is able to
cover multiple versions of Python, including the rolling pre-freeze release
stream.
This option isn't going to be viable for all libraries, and the desired outcome
for those authors is that they be able to support the rolling releases by
building and publishing one additional wheel archive, built against the initial
X.Y.0a1 release. The subsequent build against X.Y.0rc1 or later is then the same
build that would have been needed if only supporting the final stable release.
Additional wheel builds beyond those two should then only be needed if that
particular library is directly affected by an ABI break in any other alpha
release that occurs between those two points.
Having a rolling pre-freeze release stream available may also make it more feasible
for more CI providers to offer a "CPython beta release" testing option. At the
moment, this feature is only available from CI providers that are willing and
able to put the necessary time and effort into creating, testing, and publishing
their own builds from the CPython master branch (e.g. [6]_).
Implications for the proposed Scientific Python ecosystem support period
------------------------------------------------------------------------
Based on discussions at SciPy 2019, NEP (NumPy Enhancement Proposal) 29 has
been drafted [3]_ to propose a common convention across the Scientific Python
ecosystem for dropping support for older Python versions.
While the exact formulation of that policy is still being discussed, the draft
proposal (as of October 20, 2019) recommends that projects support any Python
feature release published within the last 42 months, with a minimum of
supporting the latest 2 Python feature releases.
For an 18-month feature release cadence, that works out to always supporting at
least the two most recent feature releases, and then dropping support for all
X.Y.Z releases around 6 months after X.(Y+2).0 is released. This means there is
a 6-month period roughly every other year where the three most recent feature
releases are supported.
For a 12-month release cadence, it would work out to always supporting at
least the three most recent feature releases, and then dropping support for all
X.Y.Z releases around 6 months after X.(Y+3).0 is released. This means that
for half of each year, the four most recent feature releases would be supported,
with the other half of each year hopefully being used to get ready for that
year's feature release.
For a 24-month release cadence, the second clause takes priority over the first,
and the recommended Python version support period increases to 48 months from
the initial X.Y.0 release in order to consistently support the two most recent
CPython feature releases. For projects that also support the rolling release
stream, the number of supported feature releases would increase to three.
Release cycle alignment for core development sprints
----------------------------------------------------
With the proposal in this PEP, it is expected that the focus of core
development sprints would shift slightly based on the current location
in the two-year cycle.
In release years, the timing of PyCon US is suitable for new contributors to
work on bug fixes and smaller features before the first release candidate goes
out, while the Language Summit and core developer discussions can focus on
plans for the next release series.
The pre-alpha core development sprint in release years will provide an
opportunity to incorporate feedback received on the previous release, either
as part of the next maintenance release (for bug fixes and feedback on
provisional APIs), or as part of the first alpha release of the next release
series (for feedback received on stable APIs).
Those initial alpha releases would also be the preferred target for ABI breaking
changes to the full CPython ABI (while changes later in the release cycle
would still be permitted as described in this PEP, landing them in the X.Y.0a1
release means that they won't trigger any additional work for publishers of
pre-built binary packages).
The Steering Council elections for the next release cycle are also likely to
occur around the same time as the pre-alpha development sprints.
In non-release years, the focus for both events would just be on the upcoming
maintenance and pre-freeze releases. These less intense years would hopefully
provide an opportunity to tackle various process changes and infrastructure
upgrades without impacting the release candidate preparation process.
Release cycle alignment for prominent Linux distributions
---------------------------------------------------------
Some rolling release Linux distributions (e.g. Arch, Gentoo) may be in a
position to consume the new rolling pre-freeze releases proposed in this PEP,
but it is expected that most distributions would continue to use the established
releases.
The specific dates for final releases proposed in this PEP are chosen to align
with the feature freeze schedules for the annual October releases of the Ubuntu
and Fedora Linux distributions.
For both Fedora and Ubuntu, it means that the release candidate phase aligns
with the development period for a distro release, which is the ideal time for
them to test a new version and provide feedback on potential regressions and
compatibility concerns.
For Ubuntu, this also means that their April LTS releases will have benefited
from a full short-term release cycle using the new system Python version, while
still having that CPython release be open to upstream bug fixes for most of the
time until the next Ubuntu LTS release.
The one Linux release cycle alignment that is likely to be consistently poor
with the specific proposal in this PEP is with Debian, as that has been released
in the first half of odd-numbered years since 2005 (roughly 12 months offset
from Ubuntu LTS releases).
With the annual release proposal in :pep:`602`, both Debian and Ubuntu LTS would
consistently get a system Python version that is around 6 months old, but
would also consistently select different Python versions from each other.
With a two-year cadence, and CPython releases in the latter half of the year,
they're likely to select the same version as each other, but one of them will
be choosing a CPython release that is more than 18 months behind the latest beta
releases by the time the Linux distribution ships.
If that situation does occur, and is deemed undesirable (but not sufficiently
undesirable for *Debian* to choose to adjust their release timing), then that's
where the additional complexity of the "incremental feature release" proposal
in :pep:`598` may prove worthwhile.
(Moving CPython releases to the same half of the year as the Debian and Ubuntu
LTS releases would potentially help mitigate the problem, but also creates
new problems where a slip in the CPython release schedule could directly affect
the release schedule for a Linux distribution, or else result in a distribution
shipping a Python version that is *more* than 18 months old)
Implications for simple deployment environments
-----------------------------------------------
For the purposes of this PEP, a "simple" deployment environment is any use case
where it is straightforward to ensure that all target environments are updated
to a new Python release at the same time (or at least in advance of the rollout
of new higher level application versions), and any pre-release testing that
occurs need only target a single Python micro version.
The simplest such case would be scripting for personal use, where the testing
and target environments are the exact same environment.
Similarly simple environments would be containerised web services, where the
same Python container is used in the CI pipeline as is used on deployment, and
any application that bundles its own Python runtime, rather than relying on a
pre-existing Python deployment on the target system.
For these use cases, there is a straightforward mechanism to minimise the
impact of this PEP: continue using the stable releases, and ignore the rolling
pre-freeze releases.
To actually adopt the rolling pre-freeze releases in these environments, the
main challenge will be handling the potential for extension module segfaults
when the next pre-freeze release is an alpha release rather than a beta
release, indicating that the CPython ABI may have changed in an incompatible
way.
If all extension modules in use target the stable ABI, then there's no problem,
and everything will work just as smoothly as it does on the stable releases.
Alternatively, "rebuild and recache all extension modules" could become a
standard activity undertaken as part of updating to an alpha release.
Finally, it would also be reasonable to just not worry about it until something
actually breaks, and then handle it like any other library compatibility issue
found in a new alpha or beta release.
Aside from extension module ABI compatibility, the other main point of additional
complexity when using the rolling pre-freeze releases would be "roll-back"
compatibility for independently versioned features, such as pickle and SQLite,
where use of new or provisional features in the beta stream may create files
that are not readable by the stable release. Applications that use these
kinds of features and also require the ability to reliably roll-back to a
previous stable CPython release would, as today, be advised to avoid adopting
pre-release versions.
Implications for complex deployment environments
------------------------------------------------
For the purposes of this PEP, "complex" deployment environments are use cases
which don't meet the "simple deployment" criteria above. They may involve
multiple distinct versions of Python, use of a personalised build of Python,
or "gatekeepers" who are required to approve use of a new version prior to
deployment.
For example, organisations that install Python on their users' machines as part
of a standard operating environment fall into this category, as do those that
provide a standard build environment. Distributions such as conda-forge or
WinPython that provide collections of consistently built and verified packages
are impacted in similar ways.
These organisations tend to either prefer high stability (for example, all of
those who are happily using the system Python in a stable Linux distribution
like Debian, RHEL/CentOS, or Ubuntu LTS as their preferred Python environment)
or fast turnaround (for example, those who regularly contribute toward the
latest CPython pre-releases).
In some cases, both usage models may exist within the same organisation for
different purposes, such as:
* using a stable Python environment for mission critical systems, but allowing
data scientists to use the latest available version for ad hoc data analysis
* a hardware manufacturer deploying a stable Python version as part of their
production firmware, but using the latest available version in the development
and execution of their automated integration tests
Under any release model, each new release of Python generates work for these
organisations. This work may involve legal, security or technical reviews of
Python itself, assessment and verification of impactful changes, reapplication
of patches, recompilation and testing of third-party dependencies, and
only then deployment.
Organisations that can take updates quickly should be able to make use of the
more frequent beta releases. While each update will still require similar
investigative work to what they require today, the volume of work required per
release should be reduced as each release will be more similar to the previous
than it is under the present model. One advantage of the proposed
release-every-2-months model is that organisations can choose their own adoption
cadence from adopting every beta release, to adopting one per quarter, or one
every 6 months, or one every year. Beyond that, it would likely make more sense
to continue using the stable releases instead.
For organisations with stricter evaluations or a preference for stability, the
longer release cycle for stable releases will reduce the annual effort required
to update, the longer release candidate period will allow more time to do
internal testing before the X.Y.0 release, and the greater use by others
during the beta period will provide more confidence in the initial releases.
Meanwhile, the organisation can confidently upgrade through maintenance
releases for a longer time without fear of breaking changes.
Acknowledgements
================
Thanks to Łukasz Langa for creating :pep:`602` and prompting this discussion of
possible improvements to the CPython release cadence, and to Kyle Stanley
and h-vetinari for constructive feedback on the initial draft of this PEP.
References
==========
.. [1] Steve Dower's initial "Fast and Stable releases" proposal
(https://discuss.python.org/t/pep-602-annual-release-cycle-for-python/2296/20)
.. [2] Paul Moore's initial comments on Steve's proposal
(https://discuss.python.org/t/pep-602-annual-release-cycle-for-python/2296/37)
.. [3] NEP 29 proposes a common policy for dropping support of old Python versions
(https://numpy.org/neps/nep-0029-deprecation_policy.html)
.. [4] Example implementation for a pre-release SOABI flag
(https://github.com/ncoghlan/cpython/pull/3)
.. [5] CPython stable ABI documentation
(https://docs.python.org/3/c-api/stable.html)
.. [6] Travis CI nightly CPython builds
(https://docs.travis-ci.com/user/languages/python/#nightly-build-support)
Copyright
=========
This document is placed in the public domain or under the CC0-1.0-Universal
license, whichever is more permissive.