Rasterio and pip 19

Sean Gillies

Hi all,

I was tripped up by the release of pip 19 and want to share what I learned so that you can avoid the same bruises.

Rasterio has a pyproject.toml file since version 1.0. This file specifies the build requirements of Rasterio, the packages you must have to build working modules from the source. These include Cython, because Rasterio is largely written in the Cython language, and Numpy, because Cython needs Numpy's C headers.

When pip 10 was released, it began to use this pyproject.toml file and you may have noticed that the old two-step build of Rasterio

  pip install numpy cython
  pip install --no-binary rasterio rasterio

wasn't necessary anymore. Pip would check the pyproject.toml file and install numpy and cython before moving on to Rasterio's own setup script. Note well that pip 10 would install numpy and cython into the site-packages folder of the currently active Python environment. Pip 19 changes this.

With pip 19, the default is to create a new temporary environment and install numpy and cython into it. And because I didn't specify versions in pyproject.toml, this means the latest numpy (1.16.0 as I type this). Extension modules compiled using the 1.16.0 headers aren't compatible with older versions of numpy. Importing them will result in a ValueError and a message about mismatched struct sizes.

In Rasterio's wheel-building system we have a different way of satisfying the build-time dependencies, but the new temporary environment snuck in, entailed a dependency on numpy 1.16.0 and then when we tested the wheels with versions of numpy that *should* have been compatible, we had failed tests. It was rather perplexing until I reminded myself to check the pip change log. The problem was easy enough to solve: we use the --no-build-isolation option of pip install and our builds go on as they did with earlier pip versions.

You'll see the same thing if you build wheels for production and then try to use them with an older version of numpy.

I'm hesitant to pin numpy in pyproject.toml because the right version sort of depends on your Python version. However, there's a potential stumbling block here that needs to be addressed. If anybody has more experience and wisdom to share about the new features of pip, I'm all ears.

Sean Gillies

Join main@rasterio.groups.io to automatically receive all group messages.