Tracking down a segmentation fault


Angus Dickey
 

We have a Chalice app that uses Rasterio to read COGs from an AWS S3 bucket, do some processing, and deliver the result as XYZ tiles.

Recently we upgraded the rasterio version and started seeing segmentation faults killing the chalice development server. Rasterio 1.2.3 and lower work fine, but 1.2.4 and up segfaults after a small number of requests.

The app uses a python 3.6 virtual environment and the rasterio manylinux wheels (which are awesome BTW). The backtrace (from gdb) seems to show the problem is in libgdal but doesn't provide much to go on:

Thread 13 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff9e5d8700 (LWP 3887)]
0x00007fffad92e0b3 in ?? () from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
(gdb) bt
#0  0x00007fffad92e0b3 in ?? () from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#1  0x00007fffad92e552 in ?? () from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#2  0x00007fffad92e5b1 in GDALProxyPoolDataset::~GDALProxyPoolDataset() ()
   from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#3  0x00007fffad92e679 in GDALProxyPoolDataset::~GDALProxyPoolDataset() ()
   from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#4  0x00007fffad8d0fcf in GDALDataset::ReleaseRef() () from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#5  0x00007fffad86448c in VRTSimpleSource::~VRTSimpleSource() () from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#6  0x00007fffad864569 in VRTComplexSource::~VRTComplexSource() () from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#7  0x00007fffad863e8a in VRTSourcedRasterBand::CloseDependentDatasets() ()
   from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#8  0x00007fffad863eeb in VRTSourcedRasterBand::~VRTSourcedRasterBand() ()
   from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#9  0x00007fffad863f59 in VRTSourcedRasterBand::~VRTSourcedRasterBand() ()
   from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#10 0x00007fffad8d67ef in GDALDataset::~GDALDataset() () from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#11 0x00007fffad8409f2 in VRTDataset::~VRTDataset() () from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#12 0x00007fffad840a99 in VRTDataset::~VRTDataset() () from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#13 0x00007fffad92dd11 in ?? () from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#14 0x00007fffad92e311 in ?? () from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#15 0x00007fffad92e62c in GDALProxyPoolDataset::~GDALProxyPoolDataset() ()
   from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#16 0x00007fffad92e679 in GDALProxyPoolDataset::~GDALProxyPoolDataset() ()
   from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#17 0x00007fffad8d0fcf in GDALDataset::ReleaseRef() () from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#18 0x00007fffad86448c in VRTSimpleSource::~VRTSimpleSource() () from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#19 0x00007fffad864569 in VRTComplexSource::~VRTComplexSource() () from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#20 0x00007fffad863e8a in VRTSourcedRasterBand::CloseDependentDatasets() ()
   from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#21 0x00007fffad863eeb in VRTSourcedRasterBand::~VRTSourcedRasterBand() ()
   from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#22 0x00007fffad863f59 in VRTSourcedRasterBand::~VRTSourcedRasterBand() ()
   from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#23 0x00007fffad8d67ef in GDALDataset::~GDALDataset() () from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#24 0x00007fffad8409f2 in VRTDataset::~VRTDataset() () from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#25 0x00007fffad840a99 in VRTDataset::~VRTDataset() () from /home/user/project/env/lib/python3.6/site-packages/rasterio/../rasterio.libs/libgdal-41ff680c.so.29.0.0
#26 0x00007fffaec0ac99 in ?? () from /home/user/project/env/lib/python3.6/site-packages/rasterio/_base.cpython-36m-x86_64-linux-gnu.so
#27 0x00007fffaec21216 in ?? () from /home/user/project/env/lib/python3.6/site-packages/rasterio/_base.cpython-36m-x86_64-linux-gnu.so
#28 0x00007fffaec2d5d0 in ?? () from /home/user/project/env/lib/python3.6/site-packages/rasterio/_base.cpython-36m-x86_64-linux-gnu.so
#29 0x0000000000566bbc in _PyCFunction_FastCallDict ()
#30 0x00000000005a4cd1 in _PyObject_FastCallDict ()
#31 0x00000000005a4fb8 in PyObject_CallFunctionObjArgs ()
#32 0x000000000050d698 in _PyEval_EvalFrameDefault ()
#33 0x00000000005095c8 in ?? ()
#34 0x000000000050a2fd in ?? ()
#35 0x000000000050beb4 in _PyEval_EvalFrameDefault ()
#36 0x00000000005095c8 in ?? ()
#37 0x000000000050a2fd in ?? ()
#38 0x000000000050beb4 in _PyEval_EvalFrameDefault ()
#39 0x0000000000507be4 in ?? ()
#40 0x0000000000588e5c in ?? ()
#41 0x000000000059fd0e in PyObject_Call ()
#42 0x000000000050d256 in _PyEval_EvalFrameDefault ()
#43 0x00000000005095c8 in ?? ()
#44 0x000000000050a2fd in ?? ()
#45 0x000000000050beb4 in _PyEval_EvalFrameDefault ()
#46 0x0000000000507be4 in ?? ()
#47 0x0000000000509900 in ?? ()
#48 0x000000000050a2fd in ?? ()
#49 0x000000000050beb4 in _PyEval_EvalFrameDefault ()
#50 0x0000000000507be4 in ?? ()
#51 0x0000000000509900 in ?? ()
#52 0x000000000050a2fd in ?? ()
#53 0x000000000050beb4 in _PyEval_EvalFrameDefault ()
#54 0x00000000005095c8 in ?? ()
#55 0x000000000050a2fd in ?? ()
#56 0x000000000050beb4 in _PyEval_EvalFrameDefault ()
#57 0x0000000000508cd5 in _PyFunction_FastCallDict ()
#58 0x0000000000594a01 in ?? ()
#59 0x000000000054a971 in ?? ()
#60 0x00000000005a9dac in _PyObject_FastCallKeywords ()
#61 0x000000000050a433 in ?? ()
#62 0x000000000050beb4 in _PyEval_EvalFrameDefault ()
#63 0x0000000000507be4 in ?? ()
#64 0x0000000000508ec2 in _PyFunction_FastCallDict ()
#65 0x0000000000594a01 in ?? ()
#66 0x000000000054a971 in ?? ()
#67 0x00000000005a9dac in _PyObject_FastCallKeywords ()
#68 0x000000000050a433 in ?? ()
#69 0x000000000050beb4 in _PyEval_EvalFrameDefault ()
#70 0x0000000000508cd5 in _PyFunction_FastCallDict ()
#71 0x0000000000594a01 in ?? ()
#72 0x000000000054a971 in ?? ()
#73 0x00000000005a9dac in _PyObject_FastCallKeywords ()
#74 0x000000000050a433 in ?? ()
#75 0x000000000050beb4 in _PyEval_EvalFrameDefault ()
#76 0x0000000000507be4 in ?? ()
#77 0x0000000000509900 in ?? ()
#78 0x000000000050a2fd in ?? ()
#79 0x000000000050cc96 in _PyEval_EvalFrameDefault ()
#80 0x00000000005095c8 in ?? ()
#81 0x000000000050a2fd in ?? ()
#82 0x000000000050beb4 in _PyEval_EvalFrameDefault ()
#83 0x00000000005095c8 in ?? ()
#84 0x000000000050a2fd in ?? ()
#85 0x000000000050beb4 in _PyEval_EvalFrameDefault ()
#86 0x00000000005095c8 in ?? ()
#87 0x000000000050a2fd in ?? ()
#88 0x000000000050beb4 in _PyEval_EvalFrameDefault ()
#89 0x00000000005095c8 in ?? ()
#90 0x000000000050a2fd in ?? ()
#91 0x000000000050beb4 in _PyEval_EvalFrameDefault ()
#92 0x0000000000507be4 in ?? ()
#93 0x0000000000508ec2 in _PyFunction_FastCallDict ()
#94 0x0000000000594a01 in ?? ()
#95 0x0000000000549e8f in ?? ()
#96 0x00000000005515c1 in ?? ()
#97 0x00000000005a48ec in _PyObject_FastCallDict ()
#98 0x00000000005f03bc in ?? ()
#99 0x00000000005a9dac in _PyObject_FastCallKeywords ()
#100 0x000000000050a433 in ?? ()
#101 0x000000000050beb4 in _PyEval_EvalFrameDefault ()
#102 0x00000000005095c8 in ?? ()
#103 0x000000000050a2fd in ?? ()
#104 0x000000000050beb4 in _PyEval_EvalFrameDefault ()
#105 0x0000000000508cd5 in _PyFunction_FastCallDict ()
#106 0x0000000000594a01 in ?? ()
#107 0x000000000059fd0e in PyObject_Call ()
#108 0x000000000050d256 in _PyEval_EvalFrameDefault ()
#109 0x00000000005095c8 in ?? ()
#110 0x000000000050a2fd in ?? ()
#111 0x000000000050beb4 in _PyEval_EvalFrameDefault ()
#112 0x00000000005095c8 in ?? ()
#113 0x000000000050a2fd in ?? ()
#114 0x000000000050beb4 in _PyEval_EvalFrameDefault ()
#115 0x0000000000508cd5 in _PyFunction_FastCallDict ()
#116 0x0000000000594a01 in ?? ()
#117 0x000000000059fd0e in PyObject_Call ()
#118 0x00000000005e12f2 in ?? ()
#119 0x0000000000631b94 in ?? ()
#120 0x00007ffff77ca6db in start_thread (arg=0x7fff9e5d8700) at pthread_create.c:463
#121 0x00007ffff7b0371f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95


Is there a location where the libgdal version for each resterio release is documented? I presume 1.2.4 uses a newer GDAl version.

Any tips for tracking down what the problem might be in rasterio/GDAL?

Thanks for any help,

Angus

Join main@rasterio.groups.io to automatically receive all group messages.