Numpy Import Fails On Multiarray Extension Library When Called From Embedded Python Within A C++ Application

I'm running a C++ application which tries to run python using the https://docs.python.org/3.5/extending/embedding.html function calls. This is the error that the application error

Solution 1:

Root Cause

This error occurs because multiarray.cpython-35m-x86_64-linux-gnu.so module in numpy depends on libpythonx.x.so, be it is not explicit link the libpythonx.x.so. So if you use ldd -d multiarray.cpython-35m-x86_64-linux-gnu.so you will not see the python in the list.

Python doesn't have issue because python binary depends on libpython.x.x.so, so when numpy load multiarray.cpython-35m-x86_64-linux-gnu.so by using dlopen. libdl.so will try to resolve the undefined symbols by checking the dependent shared library of the main program which is python. It will find it in libpython.x.x.so.

Solution

After knowing the root cause the solution is very easy, just help libdl.so to be able to find libpython.x.x.so. There are at least two ways to achieve that:

1. Use dlopen("libpythonx.x.so", RTLD_GLOBAL). After opening this so use RTLD_GLOBAL flag, it make symbol in libpythonx.x.so available for symbol resolution of subsequently loaded shared objects.
2. In main program which embed python, add the libpythonx.x.so into its dependency libraries.

Solution 2:

I had a similar error with linking an application against a libpython3.5m.a (archive, not dynamic). Once it loaded something like multiarray.cpython-35m-x86_64-linux-gnu.so, it would expect symbols like PyFloat_Type to exist.

In diagnosing why Python could be called directly and it would work, but my application would not, I noticed that readelf -s myapplication had a PyFloat_Type symbol in the .symtab table but not in the .dynsym table.

However, readelf -s /asb/path/to/python3 had a PyFloat_Type symbol in both tables.

Adding: target_link_options(myapplication PUBLIC "LINKER:-export-dynamic") in CMake ensured that the symbols needed were also available in the .dynsym table. After this, it the application worked correctly.