Thank you for the reports and discussion, and big thanks to Alan Snow for the contributions around coordinate reference system interoperability.
I've finessed the Linux and OS X wheel builds so that for the first time we can include size and speed optimized GDAL shared libraries. An install of one of the Linux wheels now takes up "only" 52 MB, 39 MB of which are shared libraries. The wheels themselves are down to 15 MB.
