Python project managers comparison
Until just a few years ago, the python packaging ecosystem was very confusing. To manage the dependencies of a project you had to manually create a file called requirements.txt, where you would put all of the libraries your project was dependant on. There was no command line utility to do this, and managing very big projects became very annoying.
If you wanted to create a library for other people, things were even more desperate. The standard in use was setuptools, which was everything but user friendly. And if you wanted to declare the dependencies of your library, you had to specify them in the configuration file of setuptools, setup.py. So dependencies were declared in two different places. There is no way this can't go wrong!!!
The tooling wasn't very good, especially compared to other programming languages tools (think about Rust Cargo, for example). But everyting changed the introduction of Pyproject.toml
Pyproject.toml
Pyproject.toml is a new python configuration file, standardized thanks to PEP518. It defines a way to declare dependencies for both libraries and applications, and greatly simplifies the installation process for a package. It is such a great advancemente in the python packaging world! And it paves the way for all the great tools that are popping up every other day in the Python ecosystem.
But Pyproject.toml is not only used to store the dependencies of your project, it also works as a centralized configuration for all python tools (ex. pytest, black, isort, flake8, mypy) as they move towards supporting it. This will help declutter your project from all your messy different configuration files, replacing them with only one.
Moving your dependencies in pyproject.toml means you can remove the very annoying requirements.txt from you project, alongside the MANIFEST.in file, which I personally hate, if the tool of your choice supports it.
Project managers
While searching for the perfect tool for my projects, I found out that there is really no up to date guide for it. I found some articles, but they were either old, so they mentioned project which are no more supported or widely used, or they talked about only the most popular ones, indicating a lack of thorough research. So I did a lot of research on my own. This is the result, I hope it to be helpful, so that you can avoid wasting a lot of time like I did!
Poetry
Poetry is the most famous project base on Pyproject.toml, with over 20k stars on github. A lot of the popularity of Pyproject.toml can be attributed to the success of this project. Poetry comes with a very nice CLI that lets you manage all the aspects of your project, from managing dependencies, searching them on remote repositories, to building and publishing the pakage on PyPI. It has also a pyproject.toml generator so you can get up and runnig with the new standard immediately! It's very difficult to find something that you need from a project manager that Poetry doesn't provide, or that isn't provided by one of its many plugins.
However Poetry it is not perfect, it has a few downsides. Having become such a big project it is very slow to add new features. The most glaring one is the fact that Poetry has still not updated to PEP621, which standardizes a few aspects of Pyproject.toml, and still uses its own custom Pyproject.toml format. The PEP has been finalized in June 2020, so two and a half years ago! This puts in persepective how slow is sometimes Poetry to add new features. This missing feature means that the project config generated by Poetry is somewhat different to the one generated by the other projects, that are very similar between each others. Still, Poetry as a very extensive documentation and a very large community of loyal users, so learning the differences is definetely not difficult.
Poetry is an amazing project managers which comes with almost all you could ask for in a tool like this. If you are unsure which one to choose from this list, this one can be a great entry point into the world of Pyproject.toml and the future of python pakaging and dependency declaration.
Hatch
Hatch is a project manager which supports all the latest PEPs about Pyproject.toml. It has way fewer stars than Poetry, but it is very much in active development, and is more up to date to all the modern standards. For example it fully supports PEP621, which standardizes dependency declaration among build tools.
In addition to this, Hatch has some nice features that Poetry doesn't suport. For example it can source the version of the library dinamically, for example from an __init__.py file. This with Poetry is impossible without a plugin. And talking about plugins, Hatch has fantastic documentation on how to write a new plugin and has a stable API for it; with Poetry the API is always changing, and there is little to no documentation about it. Hatch has also support for custom build hooks. All of this means that Hatch is very useful when you have a project with a complex build and installation process.
But the feature that really distinguish Hatch from all the other tools is its concept of Environment. You can have multiple virtual environments, each dedicated to a specific part of your workflow, for example dev, linting, testing, etc..., each one with a different set of additional dependencies, and you can run scripts in this environments without having to activate the virtual environment using the hatch CLI interface. Poetry has support for something like this, but you are limited to only one virtual environment, so I prefer to use other tools to isolate my environment. Hatch can provide better isolation, and can even replace Tox. This is really a killer feature!
Unfortunately even this project has some downsides. Because it aims to supports all the standard PEPs, it doesn't try to add features if they are not standardized. This means that there is still no support for lock files, so you can't have reproducible build environments. For me, and for most application developer, this is really a problem! The only thing we can do is wait and hope a PEP is proposed that aims to tackle this problem.
PDM
PDM is another Python project manager which has almost all of the features previously mentioned. Like Hatch is supports all the standard Pyproject.toml PEPs, custom build hooks, and good plugin support. Like Poetry, it supports creating a lock file, definetely a very important feature, and has a very good CLI.
The feature that really distinguishes PDM from all the other tools is its support for PEP582, a still in draft proposal which will enable managing dependencies for different projects without using virtual environments, but instead storing all of the packages inside a folder in your project root, something like what NPM does for NodeJs. PEP582 was proposed almost five years ago, and even though it is certanly interesting it doesn't have broad support across the Python world. If you are really into this, PDM is certainly the tool for you. Otherwise it is still a very capable tool, but loses its most distinguishing feature.
Setuptools
Even though setuptools was the older method to install packages, it doesn't mean that is going to fade away. Setuptools is in fact a project by Pypa, the Python Package Authority, the same organization that mantains Flit, Hatch, and a lot of related packages. It should surprise no one the fact that since those proposal was made setuptools has become a backend build system for pyproject.toml, like the ones we talked before. That means that you can move to pyproject.toml withough changing you project very much. You can even decide which part of the configuration move in pyproject.toml, and leave the rest in the legacy setup.py, without any problems.
It has however a lot of quirks. Because it is a very large project, it moves very slowly and a lot of features of pyproject.toml are still in beta. Moreover, unlike all the previous projects, it doesn't have a cli, because it is only a build system and doesn't care about project management. This can be solved by using a fronted like build, but still it doens't provide the nice features of the other projects, like auto generating a config file. This is certainly not a difficult task (here you can see how I migrated one of my projects), but certainly annoying.
Still setuptools has one major advantage. Because it supports all the legacy code from setup.py, it supports writing custom install commands. For most projects it isn't such a big deal, but if you are working on something that doesn't use only python, for example if you are developing a C extension or another project with non python dependencies, setuptools is definetely the easieast way to go. You also have the advantage that you don't have to modify your setup.py, so the transition to pyproject.toml can be done fairly quickly. Setuptools is not the only way to achieve this; you could write custom plugins for previous projects or use other niche tools, but if you are in this situation I would wholeheartedly recommend setuptools. But if you are not is this situation I would recommend the other tools as they are nicer to use.
Pipenv
I can't talk about python projects managers without talking about Pipenv. Pipenv moves in a different direction than Pyproject.toml, being developed much earlier. But even if it is very old, it doesn't mean it can't add value to your project. It concerns itself only with dependency management, and it is very good at it. It saves the dependencies declaration inside a file named Pipfile, and then generates a Pipfile.lock that contains the exact versions of all the libraries, to better reproduce the python environment across machines.
Pipenv has also native support for development dependencies and a very nice integration with venv, e.g you can spawn a virtual environment shell directly with a Pipenv command. It has a very simple but effective interface, and is very simple to drop in one of your projects without disrupting the pre existent workflow.
As you can see Pipenv doesn't care about building packages. It is geared towards application developers, even though library developers can rely on it for declaring all the dev dependencies. So if you are developing an end user application and don't need all the additional features of the other projects, Pipenv may be the right tool for you.
Flit
Flit is another project based on Pyproject.toml but moves on the opposite direction to Pipenv, it concerns itself mostly with building and publishing packages. It applies sane defaults that are very useful when working with packages, like automatically including subpackages and all the data files inside the working directory. This is a very big thing for package maintainers, because forgetting to include data inside the package may result in a lot of problems! Furthermore Flit comes with a pyproject.toml init tool, so you can get up and running with publishing your package in little to no time!
However Flit is no perfect, especially with its dependency management. Its CLI doesn't let you add or remove packages from pyproject.toml, you have to manually modify them. It consequentially doesn't come with commands to inspect the dependency tree. This may be a problem if you have a lot of intricate dependencies.
Flit can be a very good tool if you have a package with minimal dependencies that you want to publish with very little hassle. But if you have a more complex projects, the previous tools may be better suited for your use case.
Additional tools
Here is a brief list of other tools which may be useful to you, if you have specifc requirements
maturin: if you want to build bindings to Rust packages
scikit-build: glue code between setuptools and the CMake build tool
enscons: build python packages with the SCons build tool. Very niche package
pyflow: a tool written in Rust with support for PEP582. It also manages python virtual environments and python installations for you, so good if you want to use one tool for everyting. Unfortunately seems no more under development
Conclusion
This is really an exciting time in the Python ecosystem! I hope I managed to help you understand the vast landscape of tools that help you manage your Python projects. Cheers!