PEP: 507 Title: Migrate CPython to Git and GitLab Version: $Revision$ Last-Modified: $Date$ Author: Barry Warsaw <barry@python.org> Status: Rejected Type: Process Content-Type: text/x-rst Created: 30-Sep-2015 Post-History: Resolution: https://mail.python.org/pipermail/core-workflow/2016-January/000345.html Abstract ======== This PEP proposes migrating the repository hosting of CPython and the supporting repositories to Git. Further, it proposes adopting a hosted GitLab instance as the primary way of handling merge requests, code reviews, and code hosting. It is similar in intent to :pep:`481` but proposes an open source alternative to GitHub and omits the proposal to run Phabricator. As with :pep:`481`, this particular PEP is offered as an alternative to :pep:`474` and :pep:`462`. Rationale ========= CPython is an open source project which relies on a number of volunteers donating their time. As with any healthy, vibrant open source project, it relies on attracting new volunteers as well as retaining existing developers. Given that volunteer time is the most scarce resource, providing a process that maximizes the efficiency of contributors and reduces the friction for contributions, is of vital importance for the long-term health of the project. The current tool chain of the CPython project is a custom and unique combination of tools. This has two critical implications: * The unique nature of the tool chain means that contributors must remember or relearn, the process, workflow, and tools whenever they contribute to CPython, without the advantage of leveraging long-term memory and familiarity they retain by working with other projects in the FLOSS ecosystem. The knowledge they gain in working with CPython is unlikely to be applicable to other projects. * The burden on the Python/PSF infrastructure team is much greater in order to continue to maintain custom tools, improve them over time, fix bugs, address security issues, and more generally adapt to new standards in online software development with global collaboration. These limitations act as a barrier to contribution both for highly engaged contributors (e.g. core Python developers) and especially for more casual "drive-by" contributors, who care more about getting their bug fix than learning a new suite of tools and workflows. By proposing the adoption of both a different version control system and a modern, well-maintained hosting solution, this PEP addresses these limitations. It aims to enable a modern, well-understood process that will carry CPython development for many years. Version Control System ---------------------- Currently the CPython and supporting repositories use Mercurial. As a modern distributed version control system, it has served us well since the migration from Subversion. However, when evaluating the VCS we must consider the capabilities of the VCS itself as well as the network effect and mindshare of the community around that VCS. There are really only two real options for this, Mercurial and Git. The technical capabilities of the two systems are largely equivalent, therefore this PEP instead focuses on their social aspects. It is not possible to get exact numbers for the number of projects or people which are using a particular VCS, however we can infer this by looking at several sources of information for what VCS projects are using. The Open Hub (previously Ohloh) statistics [#openhub-stats]_ show that 37% of the repositories indexed by The Open Hub are using Git (second only to Subversion which has 48%) while Mercurial has just 2%, beating only Bazaar which has 1%. This has Git being just over 18 times as popular as Mercurial on The Open Hub. Another source of information on VCS popularity is PyPI itself. This source is more targeted at the Python community itself since it represents projects developed for Python. Unfortunately PyPI does not have a standard location for representing this information, so this requires manual processing. If we limit our search to the top 100 projects on PyPI (ordered by download counts) we can see that 62% of them use Git, while 22% of them use Mercurial, and 13% use something else. This has Git being just under 3 times as popular as Mercurial for the top 100 projects on PyPI. These numbers back up the anecdotal evidence for Git as the far more popular DVCS for open source projects. Choosing the more popular VCS has a number of positive benefits. For new contributors it increases the likelihood that they will have already learned the basics of Git as part of working with another project or if they are just now learning Git, that they'll be able to take that knowledge and apply it to other projects. Additionally a larger community means more people writing how to guides, answering questions, and writing articles about Git which makes it easier for a new user to find answers and information about the tool they are trying to learn and use. Given its popularity, there may also be more auxiliary tooling written *around* Git. This increases options for everything from GUI clients, helper scripts, repository hosting, etc. Further, the adoption of Git as the proposed back-end repository format doesn't prohibit the use of Mercurial by fans of that VCS! Mercurial users have the [#hg-git]_ plugin which allows them to push and pull from a Git server using the Mercurial front-end. It's a well-maintained and highly functional plugin that seems to be well-liked by Mercurial users. Repository Hosting ------------------ Where and how the official repositories for CPython are hosted is in someways determined by the choice of VCS. With Git there are several options. In fact, once the repository is hosted in Git, branches can be mirrored in many locations, within many free, open, and proprietary code hosting sites. It's still important for CPython to adopt a single, official repository, with a web front-end that allows for many convenient and common interactions entirely through the web, without always requiring local VCS manipulations. These interactions include as a minimum, code review with inline comments, branch diffing, CI integration, and auto-merging. This PEP proposes to adopt a [#GitLab]_ instance, run within the python.org domain, accessible to and with ultimate control from the PSF and the Python infrastructure team, but donated, hosted, and primarily maintained by GitLab, Inc. Why GitLab? Because it is a fully functional Git hosting system, that sports modern web interactions, software workflows, and CI integration. GitLab's Community Edition (CE) is open source software, and thus is closely aligned with the principles of the CPython community. Code Review ----------- Currently CPython uses a custom fork of Rietveld modified to not run on Google App Engine and which is currently only really maintained by one person. It is missing common features present in many modern code review tools. This PEP proposes to utilize GitLab's built-in merge requests and online code review features to facilitate reviews of all proposed changes. GitLab merge requests --------------------- The normal workflow for a GitLab hosted project is to submit a *merge request* asking that a feature or bug fix branch be merged into a target branch, usually one or more of the stable maintenance branches or the next-version master branch for new features. GitLab's merge requests are similar in form and function to GitHub's pull requests, so anybody who is already familiar with the latter should be able to immediately utilize the former. Once submitted, a conversation about the change can be had between the submitter and reviewer. This includes both general comments, and inline comments attached to a particular line of the diff between the source and target branches. Projects can also be configured to automatically run continuous integration on the submitted branch, the results of which are readily visible from the merge request page. Thus both the reviewer and submitter can immediately see the results of the tests, making it much easier to only land branches with passing tests. Each new push to the source branch (e.g. to respond to a commenter's feedback or to fix a failing test) results in a new run of the CI, so that the state of the request always reflects the latest commit. Merge requests have a fairly major advantage over the older "submit a patch to a bug tracker" model. They allow developers to work completely within the VCS using standard VCS tooling, without requiring the creation of a patch file or figuring out the right location to upload the patch to. This lowers the barrier for sending a change to be reviewed. Merge requests are far easier to review. For example, they provide nice syntax highlighted diffs which can operate in either unified or side by side views. They allow commenting inline and on the merge request as a whole and they present that in a nice unified way which will also hide comments which no longer apply. Comments can be hidden and revealed. Actually merging a merge request is quite simple, if the source branch applies cleanly to the target branch. A core reviewer simply needs to press the "Merge" button for GitLab to automatically perform the merge. The source branch can be optionally rebased, and once the merge is completed, the source branch can be automatically deleted. GitLab also has a good workflow for submitting pull requests to a project completely through their web interface. This would enable the Python documentation to have "Edit on GitLab" buttons on every page and people who discover things like typos, inaccuracies, or just want to make improvements to the docs they are currently reading. They can simply hit that button and get an in browser editor that will let them make changes and submit a merge request all from the comfort of their browser. Criticism ========= X is not written in Python -------------------------- One feature that the current tooling (Mercurial, Rietveld) has is that the primary language for all of the pieces are written in Python. This PEP focuses more on the *best* tools for the job and not necessarily on the *best* tools that happen to be written in Python. Volunteer time is the most precious resource for any open source project and we can best respect and utilize that time by focusing on the benefits and downsides of the tools themselves rather than what language their authors happened to write them in. One concern is the ability to modify tools to work for us, however one of the Goals here is to *not* modify software to work for us and instead adapt ourselves to a more standardized workflow. This standardization pays off in the ability to re-use tools out of the box freeing up developer time to actually work on Python itself as well as enabling knowledge sharing between projects. However, if we do need to modify the tooling, Git itself is largely written in C the same as CPython itself. It can also have commands written for it using any language, including Python. GitLab itself is largely written in Ruby and since it is Open Source software, we would have the ability to submit merge requests to the upstream Community Edition, albeit in language potentially unfamiliar to most Python programmers. Mercurial is better than Git ---------------------------- Whether Mercurial or Git is better on a technical level is a highly subjective opinion. This PEP does not state whether the mechanics of Git or Mercurial are better, and instead focuses on the network effect that is available for either option. While this PEP proposes switching to Git, Mercurial users are not left completely out of the loop. By using the hg-git extension for Mercurial, working with server-side Git repositories is fairly easy and straightforward. CPython Workflow is too Complicated ----------------------------------- One sentiment that came out of previous discussions was that the multi-branch model of CPython was too complicated for GitLab style merge requests. This PEP disagrees with that sentiment. Currently any particular change requires manually creating a patch for 2.7 and 3.x which won't change at all in this regards. If someone submits a fix for the current stable branch (e.g. 3.5) the merge request workflow can be used to create a request to merge the current stable branch into the master branch, assuming there is no merge conflicts. As always, merge conflicts must be manually and locally resolved. Because developers also have the *option* of performing the merge locally, this provides an improvement over the current situation where the merge *must* always happen locally. For fixes in the current development branch that must also be applied to stable release branches, it is possible in many situations to locally cherry pick and apply the change to other branches, with merge requests submitted for each stable branch. It is also possible just cherry pick and complete the merge locally. These are all accomplished with standard Git commands and techniques, with the advantage that all such changes can go through the review and CI test workflows, even for merges to stable branches. Minor changes may be easily accomplished in the GitLab web editor. No system can hide all the complexities involved in maintaining several long lived branches. The only thing that the tooling can do is make it as easy as possible to submit and commit changes. Open issues =========== * What level of hosted support will GitLab offer? The PEP author has been in contact with the GitLab CEO, with positive interest on their part. The details of the hosting offer would have to be discussed. * What happens to Roundup and do we switch to the GitLab issue tracker? Currently, this PEP is *not* suggesting we move from Roundup to GitLab issues. We have way too much invested in Roundup right now and migrating the data would be a huge effort. GitLab does support webhooks, so we will probably want to use webhooks to integrate merges and other events with updates to Roundup (e.g. to include pointers to commits, close issues, etc. similar to what is currently done). * What happens to wiki.python.org? Nothing! While GitLab does support wikis in repositories, there's no reason for us to migration our Moin wikis. * What happens to the existing GitHub mirrors? We'd probably want to regenerate them once the official upstream branches are natively hosted in Git. This may change commit ids, but after that, it should be easy to mirror the official Git branches and repositories far and wide. * Where would the GitLab instance live? Physically, in whatever hosting provider GitLab chooses. We would point gitlab.python.org (or git.python.org?) to this host. References ========== .. [#openhub-stats] `Open Hub Statistics <https://www.openhub.net/repositories/compare>` .. [#hg-git] `Hg-Git mercurial plugin <https://hg-git.github.io/>` .. [#GitLab] `https://about.gitlab.com/` Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: