TL;DR: you can install some Python packages on ARMv7 platforms quicker now by using
pip install --extra-index-url=https://gergely.imreh.net/wheels/ <packagename>
for a number of packages, including
simplejson,… For the whole list and the hosted versions browse
This is work in progress! You’ll need
pip 8.1 version or later!
I’m using Python for a lot of my projects, and on ARM machines (like the Raspberry Pi), and when installing libraries which have components to be compiled (such as numpy), ARM is doubly at a disadvantage: it takes much longer compilation, and when there are precompiled versions available for x86 (so those machines don’t even need to compile anything just install) that’s not the case for ARM. So I thought it would be a good Friday Hack to change this and let ARM devices install native compiled packages faster.
The standard way to distribute precompiled Python libraries is using wheels. If you check the PyPi page for numpy for example, there are a bunch of wheels available to download for x86, and when you issue a
pip install numpy, if a compatible wheel is found,
pip will use that instead of working things out from scratch.
Inspired by the manylinux project, I’ve set up a similar organization: a base Docker container with Python, and separate containers for the different libraries that install dependencies and compile them. Unfortunately manylinux’s containers are based on CentOS, and no ARM versions of those are available, otherwise it’s very cool, how they organize things. To get things done in less than a day, I went instead with our base images, and modified the Python images to work from
armv7 directly, install Python, add
wheel, and create tags for pythons 2.7-3.6.
Then added Dockerfiles for “generic” Python packages which don’t need anything extra besides what’s normally in
build-essential (more precisely the
-buildpack-deps resin.io base images). This can be used to compile packages like RPi.GPIO, simplejson, Twisted, etc.
Dockerfile for each setup and the scripts are included in the arm-wheels repo on Github, though will have to fill out some more details.
archdirectory has the
Dockerfilesfor the Python bases (so far only
armv7hf, the results are on Dockerhub)
packagesdirectory have the Dockerfiles for specific Python packages and a
_generic, as mentioned above. Can be found on Dockerhub as well: generic, numpy, scipy, pillow.
You can use them by running the
cross.sh script as
./cross.sh <pythonversion> <packagename> where
<pythonversion> is one of
3.6 (or fewer, check the available tags for on Docker hub!), and
<packagename> is in the standard
pip format something like
./cross.sh script within spins up the container, does the work, and then spits out the resulting binaries to the
./target directory, so for example
docker run \ --rm \ --detach \ -v $PWD/target:/usr/src/target \ imrehg/armv7hf-python-numpy:$1 $2
I’ve uploaded the resulting files to my server at the moment, so the available packages and versions are at
https://gergely.imreh.net/wheels/, and you can use them in
pip 8.1 or later for example as:
pip install --extra-index-url=https://gergely.imreh.net/wheels/ numpy
This one line above should save ~40 minutes of compilation time easily (
scipy package saves a few hours).
It is a very rough, and work in progress project, and I learned a big bunch! Here are a few things that I’d like to change or improve on :
- reorganize the base images, similar to
manylinux: I think it would have a bunch of benefits to create a similar setup to manylinux, including
- a single base image Docker container with all available Linux versions installed, instead of using tags
- compile Python from scratch instead of using resin.io’s precompiled distributions, so could add more platforms, such as
- ideally upstream and add these abilities to
- have an organization that can run through this compilation automatically (a CI setup), and build all required packages (while checking which ones were already built), and ideally automatically distribute them too
- have a better way to distribute these packages, such as a
devpiserver hosted somewhere
- add more Python libraries, probably all that are listed on Python Wheels as having wheels, but maybe just those ones that really need compilation (non-fully-native packages). Suggestions for what other packages to build are welcome!
What do you think, do you find this helpful? Have any feedback?