Combining Poetry and Conda 🐍

For one of my clients I had to convert a couple of pip-managed repositories to Poetry-managed ones. When I got to the last repo, there was one catch: it required Conda as well. This is quite straightforward, with the help of some packages and a couple of config changes.

Pip v.s. Poetry v.s. Conda

Lets quickly recap the 3 tools we will discuss today.

pip

Pip is probably one of the most known package managers for Python. It enabled developers to fetch and install Python packages from either the internet or a private repository. Pip has been the standard for a lot of years and will probably used for many years to come.

Poetry

Poetry has gained traction quickly of the past few years. Build upon pip, it's a package manager on steroids. With the help of pyproject.toml files you can define your packages (either individually or in groups), set the configuration for groups of packages, define settings for your favourite linters and tools and define the environment to build your packages. It can (and will so by default) also create your virtual environments.

Conda

Then there's also Conda. Conda has a lot of similarities with Poetry. One of the benefits of Conda (and this is also the reason my client uses it) is that repositories like conda-forge contain pre-compiled packages; something that's not always the case with PyPi (the public package repository that both pip and Poetry use by default).

Combining Poetry & Conda

When picking up this story, it sounded pretty straightforward. However, one of the issues is that both Poetry and Conda are both package managers as well as environment managers that can conflict with each other quite easily. So how do we set this up?

Since Poetry is quite straightforward to configure to the needs of Conda, we'll use Conda as the base environment. Then we modify Poetry to use this environment as well.

Lets start with setting up a base environment.yml to define our Conda setup:

name: poetry-and-conda
channels:
  - conda-forge
dependencies:
  - python==3.11.*
  - gmsh
  - conda-lock
  - poetry
platforms:
  - linux-64
  - osx-64
  - win-64

These packages are just some example, however there are a few important things to note here:

• We define a Python version, this should be kept the same as what we will define in our pyproject.toml.

• poetry will be installed as a dependency as well. This ensures that the poetry command which we will use later is used in the context of the Conda environment.

• We define platforms which will be used for our conda lock file, which will be generated with the conda-lock tool. You can remove the platforms you don't need.

With this out of the way we can configure a lock file. The lock file will be used to create reproducible environments across installations. We create the lock files with a temporary environment

conda create -p /tmp/condaenv -c conda-forge python=="3.11.*" conda-lock
conda activate /tmp/condaenv
conda-lock -k explicit
conda deactivate
rm -rf /tmp/condaenv

So what's happening here? We create an environment, using the conda-forge channel, and setup Python 3.11 (the same that we've defined in the environment.yml) and conda-lock. We will then activate the environment, run the conda-lock command which will pick-up the dependencies from the environment.yml, resolve them and put their exact versions and hashes into a file called conda-linux-64.lock. Using this file, we can create reprodusable environments, which is great when working in teams (everyone will get the exact same version of packages when setting up the environment).
After that's done we deactivate the environment and remove it from the system.

Now it's time to get the real environment up and running!

conda create --name poetry-and-conda -c conda-forge --file conda-osx-64.lock

First, we'll generate the environment from the lock file. We'll give it a name, specify the conda-forge channel and make use of the lock file we generated earlier to install the required packages. Please note that I'm using the conda-osx-64.lock file. If you're on Windows or Linux, you should use one those flavours instead. If you're just using a single platform, you can remove the others from the environment.yml file.

Now it's time to activate the environment and configure Poetry

conda activate poetry-and-conda

# Verify which Poetry we're using
which poetry
# Note that this will point to Poetry installed in the environment,
# not my globally installed Poetry.
# /opt/homebrew/Caskroom/miniconda/base/envs/poetry-and-conda/bin/poetry

# Make sure Poetry doesn't create another environment
poetry config --local virtualenvs.create false

# Start a Poetry project
poetry init -n

By running poetry config --local virtualenvs.create false we will get a poetry.toml that contains some local config that will prevent the creation of virtual environments for just this project. If you're working in a team it's important to commit this to your repository as well. When running the init command, make sure the Python version matches with what you've used to setup the Conda environment.

We can now use Poetry to install PyPi (or private repository, when configured) packages.

poetry add flask flask-login
poetry add --group dev pytest

Since we've configured Poetry to not create an environment for this project, it'll install the packages in the current environment - the Conda environment in this case.

With the setup above, we've added both conda(-forge) packages and PyPi packages. They're all installed in the same environment, so the interpeter is able to find the packages, regardless of which tool installed them. Now we can create a webapp (an API for example) with Flask (as installed with Poetry) that uses gmsh (that's installed with Conda).

And that's it! Now you've got a Conda and Poetry hybrid environment. I would recommend to stick with a single solution as much as possible, but sometimes you're required to combine the two.