Author Topic: Pyinstaller issue with Python3.6 deployed via AIXTOOLs (Read 1206 times)

martind · « **on:** April 30, 2019, 02:31:20 PM »

Aixtools deployment of Python3.6 seemed normal as did the subsequent install and update of pip3.6. The install of pyinstaller via pip3.6 was problematic in that the wscript was not correct for AIX using the native AIX compiler when constructing the bootloader. Pyinstaller subsequently was installed correctly and appears to work:

# pyinstaller --onefile IPP_Testing.py
75 INFO: PyInstaller: 3.4
75 INFO: Python: 3.6.5
115 INFO: Platform: AIX-1-00F9B3C44C00-powerpc-64bit-COFF
120 INFO: wrote /home/dyer/IPP_Testing.spec
155 INFO: UPX is not available.
173 INFO: Extending PYTHONPATH with paths
['/home/dyer', '/home/dyer']
173 INFO: checking Analysis
173 INFO: Building Analysis because Analysis-00.toc is non existent
174 INFO: Initializing module dependency graph...
176 INFO: Initializing module graph hooks...
178 INFO: Analyzing base_library.zip ...
7046 INFO: running Analysis Analysis-00.toc
7420 INFO: Caching module hooks...
7431 INFO: Analyzing /home/dyer/IPP_Testing.py
8042 INFO: Loading module hooks...
8044 INFO: Loading module hook "hook-encodings.py"...
8206 INFO: Loading module hook "hook-pydoc.py"...
8207 INFO: Loading module hook "hook-xml.py"...
8337 INFO: Looking for ctypes DLLs
8337 INFO: Analyzing run-time hooks ...
8352 INFO: Looking for dynamic libraries
ldd: /usr/lib/libcrypto.a: File is an archive.
ldd: /usr/lib/libssl.a: File is an archive.
16669 INFO: Looking for eggs
16670 INFO: Python library not in binary dependencies. Doing additional searching...
17000 INFO: Using Python library /opt/lib/libpython3.6.a
17017 INFO: Warnings written to /home/dyer/build/IPP_Testing/warn-IPP_Testing.txt
17078 INFO: Graph cross-reference written to /home/dyer/build/IPP_Testing/xref-IPP_Testing.html
17134 INFO: checking PYZ
17135 INFO: Building PYZ because PYZ-00.toc is non existent
17135 INFO: Building PYZ (ZlibArchive) /home/dyer/build/IPP_Testing/PYZ-00.pyz
17823 INFO: Building PYZ (ZlibArchive) /home/dyer/build/IPP_Testing/PYZ-00.pyz completed successfully.
17834 INFO: checking PKG
17834 INFO: Building PKG because PKG-00.toc is non existent
17834 INFO: Building PKG (CArchive) PKG-00.pkg
42653 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
42668 INFO: Bootloader /opt/lib/python3.6/site-packages/PyInstaller-3.4-py3.6.egg/PyInstaller/bootloader/AIX-64bit/run
42668 INFO: checking EXE
42668 INFO: Building EXE because EXE-00.toc is non existent
42668 INFO: Building EXE from EXE-00.toc
42668 INFO: Appending archive to EXE /home/dyer/dist/IPP_Testing
42784 INFO: Building EXE from EXE-00.toc completed successfully.

however, when executing the IPP-Testing program:

# ./IPP_Testing
[4915392] Error loading Python lib '/tmp/_MEI_it7ea/libpython3.6.a(libpython3.6.so)': dlopen: 0509-022 Cannot load module /tmp/_MEI_it7ea/libpython3.6.a(libpython3.6.so).
0509-152 Member libpython3.6.so is not found in archive

The /opt/lib library contains:

lrwxrwxrwx 1 root system 24 Apr 30 03:40 libpython3.6.a -> /opt/lib/libpython3.6m.a
-r-xr-xr-x 1 bin bin 10006800 Jun 01 2018 libpython3.6m.a
-rw-r--r-- 1 bin bin 1683837 Feb 02 2017 libz.a
-rwxr-xr-x 1 root system 164632 Apr 29 01:12 libz.so.1
-rwxr-xr-x 1 root system 164594 Apr 29 01:12 libz.so.1.2.10
-rwxr-xr-x 1 root system 164632 Apr 29 01:12 libz.so.1.2.11
-r-xr-xr-x 1 root system 156861 Apr 29 01:12 libz.so.1.2.8
drwxr-xr-x 2 bin bin 256 Jun 01 2018 pkgconfig
drwxr-xr-x 35 bin bin 8192 Apr 26 08:52 python3.6

i added the softlink to the archive. There is no libpython.3.6m.so (or libpython3.6m.so) and the libpython3.6m.a does not contain a libpython shared object.

I was wondering why Pyinstaller would have a dependency on libpython3.6.so when ldd does not report it:

# ldd IPP_Testing
IPP_Testing needs:
/usr/lib/libdl.a(shr_64.o)
/usr/lib/libpthreads.a(shr_xpg5_64.o)
/usr/lib/libc.a(shr_64.o)
/unix
/usr/lib/libcrypt.a(shr_64.o)

The run and run_d bootloader components also do not reference any libpython sharded object.

any ideas would be appreciated.

Michael · « **Reply #1 on:** May 01, 2019, 12:21:14 PM »

I'll take a look. Never used pyinstaller before.

Just checking:
a) you are using the IBM compiler (xlc)
b) IPP_testing.py is part of the pyinstaller package.

Michael · « **Reply #2 on:** May 01, 2019, 12:48:48 PM »

OK. I expect it will be something specific with the PyInstaller package, and/or an issue with the way Lib/ctypes is used.

I'll have to try a manual build - as I guess you did - to understand the issue.

For record keeping - a summary of actions.
p.s. this is the 32bit version.
+-----------------------------------------------------------------------------+ Installing Software... +-----------------------------------------------------------------------------+ installp: APPLYING software for: aixtools.python3.rte 3.6.5.0 aixtools.python3.man.en_US 3.6.5.0 aixtools.python3.adt 3.6.5.0 ... +-----------------------------------------------------------------------------+ Summaries: +-----------------------------------------------------------------------------+ Installation Summary -------------------- Name Level Part Event Result ------------------------------------------------------------------------------- aixtools.python3.rte 3.6.5.0 USR APPLY SUCCESS aixtools.python3.man.en_US 3.6.5.0 USR APPLY SUCCESS aixtools.python3.adt 3.6.5.0 USR APPLY SUCCESS root@x067:[/tmp]pip3 list DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the

https://files.pythonhosted.org/packages/f9/fb/863012b13912709c13cf5cfdbfb304fa6c727659d6290438e1a88df9d848/pip-19.1-py2.py3-none-any.whl (1.4MB)
100% |################################| 1.4MB 80kB/s
Installing collected packages: pip
Found existing installation: pip 9.0.3
Uninstalling pip-9.0.3:
Successfully uninstalled pip-9.0.3
Successfully installed pip-19.1

I generally start with the download command, and see what else comes.

root@x067:[/tmp]pip download pyinstaller
Collecting pyinstaller
Downloading https://files.pythonhosted.org/packages/03/32/0e0de593f129bf1d1e77eed562496d154ef4460fd5cecfd78612ef39a0cc/PyInstaller-3.4.tar.gz (3.5MB)
|################################| 3.5MB 225kB/s
Saved ./PyInstaller-3.4.tar.gz
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... done
Collecting setuptools (from pyinstaller)
Using cached https://files.pythonhosted.org/packages/ec/51/f45cea425fd5cb0b0380f5b0f048ebc1da5b417e48d304838c02d6288a1e/setuptools-41.0.1-py2.py3-none-any.whl
Saved ./setuptools-41.0.1-py2.py3-none-any.whl
Collecting pefile>=2017.8.1 (from pyinstaller)
Downloading https://files.pythonhosted.org/packages/36/58/acf7f35859d541985f0a6ea3c34baaefbfaee23642cf11e85fe36453ae77/pefile-2019.4.18.tar.gz (62kB)
|################################| 71kB 196kB/s
Saved ./pefile-2019.4.18.tar.gz
Collecting macholib>=1.8 (from pyinstaller)
Downloading https://files.pythonhosted.org/packages/41/f1/6d23e1c79d68e41eb592338d90a33af813f98f2b04458aaf0b86908da2d8/macholib-1.11-py2.py3-none-any.whl
Saved ./macholib-1.11-py2.py3-none-any.whl
Collecting altgraph (from pyinstaller)
Downloading https://files.pythonhosted.org/packages/0a/cc/646187eac4b797069e2e6b736f14cdef85dbe405c9bfc7803ef36e4f62ef/altgraph-0.16.1-py2.py3-none-any.whl
Saved ./altgraph-0.16.1-py2.py3-none-any.whl
Collecting future (from pefile>=2017.8.1->pyinstaller)
Downloading https://files.pythonhosted.org/packages/90/52/e20466b85000a181e1e144fd8305caf2cf475e2f9674e797b222f8105f5f/future-0.17.1.tar.gz (829kB)
|################################| 829kB 81kB/s
Saved ./future-0.17.1.tar.gz
Successfully downloaded pyinstaller setuptools pefile macholib altgraph future

I am going to guess a bit, as to what is most 'root' to all, and install them one-by-one.

root@x067:[/tmp]pip list
Package Version
---------- -------
pip 19.1
setuptools 39.0.1

-rw-r--r-- 1 root system 3487849 May 01 12:34 PyInstaller-3.4.tar.gz
-rw-r--r-- 1 root system 575966 May 01 12:35 setuptools-41.0.1-py2.py3-none-any.whl
-rw-r--r-- 1 root system 62374 May 01 12:35 pefile-2019.4.18.tar.gz
-rw-r--r-- 1 root system 36562 May 01 12:35 macholib-1.11-py2.py3-none-any.whl
-rw-r--r-- 1 root system 20311 May 01 12:35 altgraph-0.16.1-py2.py3-none-any.whl
-rw-r--r-- 1 root system 829119 May 01 12:35 future-0.17.1.tar.gz

root@x067:[/tmp]pip install setuptools
Requirement already satisfied: setuptools in /opt/lib/python3.6/site-packages (39.0.1)

hmm

root@x067:[/tmp]pip install future
Collecting future
Using cached https://files.pythonhosted.org/packages/90/52/e20466b85000a181e1e144fd8305caf2cf475e2f9674e797b222f8105f5f/future-0.17.1.tar.gz
Installing collected packages: future
Running setup.py install for future ... done
Successfully installed future-0.17.1

root@x067:[/tmp]pip install altgraph
Collecting altgraph
Using cached https://files.pythonhosted.org/packages/0a/cc/646187eac4b797069e2e6b736f14cdef85dbe405c9bfc7803ef36e4f62ef/altgraph-0.16.1-py2.py3-none-any.whl
Installing collected packages: altgraph
Successfully installed altgraph-0.16.1

root@x067:[/tmp]pip install macholib
Collecting macholib
Using cached https://files.pythonhosted.org/packages/41/f1/6d23e1c79d68e41eb592338d90a33af813f98f2b04458aaf0b86908da2d8/macholib-1.11-py2.py3-none-any.whl
Requirement already satisfied: altgraph>=0.15 in /opt/lib/python3.6/site-packages (from macholib) (0.16.1)
Installing collected packages: macholib
Successfully installed macholib-1.11

root@x067:[/tmp]pip install macholib
Collecting macholib
Using cached https://files.pythonhosted.org/packages/41/f1/6d23e1c79d68e41eb592338d90a33af813f98f2b04458aaf0b86908da2d8/macholib-1.11-py2.py3-none-any.whl
Requirement already satisfied: altgraph>=0.15 in /opt/lib/python3.6/site-packages (from macholib) (0.16.1)
Installing collected packages: macholib
Successfully installed macholib-1.11

root@x067:[/tmp]pip install pefile
Collecting pefile
Using cached https://files.pythonhosted.org/packages/36/58/acf7f35859d541985f0a6ea3c34baaefbfaee23642cf11e85fe36453ae77/pefile-2019.4.18.tar.gz
Requirement already satisfied: future in /opt/lib/python3.6/site-packages (from pefile) (0.17.1)
Installing collected packages: pefile
Running setup.py install for pefile ... done
Successfully installed pefile-2019.4.18

So, even though there is a newer version of setuptools - at this point it is not needed.

+++++++
root@x067:[/tmp]pip install pyinstaller
Collecting pyinstaller
Using cached https://files.pythonhosted.org/packages/03/32/0e0de593f129bf1d1e77eed562496d154ef4460fd5cecfd78612ef39a0cc/PyInstaller-3.4.tar.gz
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... done
Requirement already satisfied: altgraph in /opt/lib/python3.6/site-packages (from pyinstaller) (0.16.1)
Requirement already satisfied: pefile>=2017.8.1 in /opt/lib/python3.6/site-packages (from pyinstaller) (2019.4.18)
Requirement already satisfied: macholib>=1.8 in /opt/lib/python3.6/site-packages (from pyinstaller) (1.11)
Requirement already satisfied: setuptools in /opt/lib/python3.6/site-packages (from pyinstaller) (39.0.1)
Requirement already satisfied: future in /opt/lib/python3.6/site-packages (from pefile>=2017.8.1->pyinstaller) (0.17.1)
Building wheels for collected packages: pyinstaller
Building wheel for pyinstaller (PEP 517) ... error
ERROR: Complete output from command /opt/bin/python3.6 /opt/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmp4rjg0pdd:
ERROR: running bdist_wheel
running build
running build_bootloader
Setting top to : /tmp/pip-install-3adnqnud/pyinstaller/bootloader
Setting out to : /tmp/pip-install-3adnqnud/pyinstaller/bootloader/build
Python Version : 3.6.5 (default, May 9 2018, 18:35:41) [C]
Checking for 'xlc' (C compiler) : /usr/vacpp/bin/xlc_r
Checking size of pointer : 4
Platform : AIX-32bit detected based on compiler
Checking for flags -q32 : yes
Checking for program '/usr/vacpp/bin/xlc_r' : /usr/vacpp/bin/xlc_r
Checking for program 'strip' : /usr/vacpp/bin/xlc_r
Checking for library dl : yes
Checking for library m : yes
Checking for library z : yes
Checking for function unsetenv : yes
Checking for function mkdtemp : yes
'configure' finished successfully (11.109s)
'all' finished successfully (0.000s)
'distclean' finished successfully (0.084s)
Setting top to : /tmp/pip-install-3adnqnud/pyinstaller/bootloader
Setting out to : /tmp/pip-install-3adnqnud/pyinstaller/bootloader/build
Python Version : 3.6.5 (default, May 9 2018, 18:35:41) [C]
Checking for 'xlc' (C compiler) : /usr/vacpp/bin/xlc_r
Checking size of pointer : 4
Platform : AIX-32bit detected based on compiler
Checking for flags -q32 : yes
Checking for program '/usr/vacpp/bin/xlc_r' : /usr/vacpp/bin/xlc_r
Checking for program 'strip' : /usr/vacpp/bin/xlc_r
Checking for library dl : yes
Checking for library m : yes
Checking for library z : yes
Checking for function unsetenv : yes
Checking for function mkdtemp : yes
'configure' finished successfully (3.684s)
'make_all' finished successfully (0.012s)
Waf: Entering directory `/tmp/pip-install-3adnqnud/pyinstaller/bootloader/build/debug'
[ 1/13] Compiling src/pyi_python.c
[ 2/13] Compiling src/main.c
[ 3/13] Compiling src/pyi_path.c
[ 4/13] Compiling src/pyi_pythonlib.c
[ 5/13] Compiling src/pyi_win32_utils.c
"../../src/pyi_win32_utils.c", line 574.21: 1506-356 (W) Compilation unit is empty.

[ 6/13] Compiling src/pyi_main.c
[ 7/13] Compiling src/pyi_launch.c
[ 8/13] Compiling src/pyi_python27_compat.c
[ 9/13] Compiling src/pyi_utils.c
[10/13] Compiling src/pyi_global.c
[11/13] Compiling src/pyi_archive.c
[12/13] Linking build/debug/run_d
ld: 0706-006 Cannot find or open library file: -l Z
ld:open(): A file or directory in the path name does not exist.

Waf: Leaving directory `/tmp/pip-install-3adnqnud/pyinstaller/bootloader/build/debug'
Build failed
-> task in 'run_d' failed with exit status 255 (run with -v to display more information)
No precompiled bootloader found. Trying to compile it for you ...
ERROR: Failed compiling the bootloader. Please compile manually and rerun setup.py
----------------------------------------
ERROR: Failed building wheel for pyinstaller
Running setup.py clean for pyinstaller
Failed to build pyinstaller
ERROR: Could not build wheels for pyinstaller which use PEP 517 and cannot be installed directly

[/tt][/list]

Michael · « **Reply #3 on:** May 01, 2019, 02:23:50 PM »

Quote

Aixtools deployment of Python3.6 seemed normal as did the subsequent install and update of pip3.6. The install of pyinstaller via pip3.6 was problematic in that the wscript was not correct for AIX using the native AIX compiler when constructing the bootloader. Pyinstaller subsequently was installed correctly and appears to work:

I am having some issues with getting wscript 'patched', so I still do not get boatloader built/installed.

Documenting for posterity...

[12/13] Linking build/debug/run_d ld: 0706-006 Cannot find or open library file: -l Z ld:open(): A file or directory in the path name does not exist. *** The first error (trying to ling -lZ is fixed here *** +763 else: +764 # Linux, Darwin (MacOSX), ... +765 # Only the libs found will actually be used, so it's safe to list all +766 # here. The decision if a lib is required for a specific platform is +767 # made in the configure phase. +768 libs = ['DL', 'M', 'Z', # 'z' - zlib, 'm' - math, +769 'THR'] # may be used on FreBSD +770 staticlibs = [] +771 if ctx.env.DEST_OS == 'aix': +772 # link statically with zlib +773 libs.remove('Z') +774 staticlibs.append('z') "bootloader/wscript" [Modified] The cursor is at line 774 of 825 Now step 12 of 13 passes, and step 13 fails with: [12/13] Linking build/debug/run_d [13/13] Processing build/debug/run_d /usr/vacpp/bin/xlc_r: 1501-216 (W) command option -32 is not recognized - passed to ld ld: 0706-012 The -X flag is not recognized. ld: 0706-012 The -3 flag is not recognized. ld: 0706-012 The -2 flag is not recognized. Skip the strip flags whatever... +459 if ctx.env.CC_NAME != 'msvc': +460 # This tool allows reducing the size of executables. +461 ctx.find_program([assoc_programm(ctx, 'strip')], var='STRIP') +462 ctx.load('strip', tooldir='tools') +463 # There is a strip flag for AIX environment +464 # if ctx.env.DEST_OS == 'aix': +465 # if ctx.env.PYI_ARCH == '32bit': +466 # ctx.env.append_value('STRIPFLAGS', '-X32') +467 # elif ctx.env.PYI_ARCH == '64bit': +468 # ctx.env.append_value('STRIPFLAGS', '-X64') +469 and the manual build process succeeds.. and the manual install succeeds... .... byte-compiling build/bdist.aix-7.1/egg/PyInstaller/utils/win32/winmanifest.py to winmanifest.cpython-36.pyc byte-compiling build/bdist.aix-7.1/egg/PyInstaller/utils/win32/winresource.py to winresource.cpython-36.pyc byte-compiling build/bdist.aix-7.1/egg/PyInstaller/utils/win32/winutils.py to winutils.cpython-36.pyc creating build/bdist.aix-7.1/egg/EGG-INFO copying PyInstaller.egg-info/PKG-INFO -> build/bdist.aix-7.1/egg/EGG-INFO copying PyInstaller.egg-info/SOURCES.txt -> build/bdist.aix-7.1/egg/EGG-INFO copying PyInstaller.egg-info/dependency_links.txt -> build/bdist.aix-7.1/egg/EGG-INFO copying PyInstaller.egg-info/entry_points.txt -> build/bdist.aix-7.1/egg/EGG-INFO copying PyInstaller.egg-info/not-zip-safe -> build/bdist.aix-7.1/egg/EGG-INFO copying PyInstaller.egg-info/requires.txt -> build/bdist.aix-7.1/egg/EGG-INFO copying PyInstaller.egg-info/top_level.txt -> build/bdist.aix-7.1/egg/EGG-INFO creating dist creating 'dist/PyInstaller-3.4-py3.6.egg' and adding 'build/bdist.aix-7.1/egg' to it removing 'build/bdist.aix-7.1/egg' (and everything under it) Processing PyInstaller-3.4-py3.6.egg creating /opt/lib/python3.6/site-packages/PyInstaller-3.4-py3.6.egg Extracting PyInstaller-3.4-py3.6.egg to /opt/lib/python3.6/site-packages Adding PyInstaller 3.4 to easy-install.pth file Installing pyi-archive_viewer script to /opt/bin Installing pyi-bindepend script to /opt/bin Installing pyi-grab_version script to /opt/bin Installing pyi-makespec script to /opt/bin Installing pyi-set_version script to /opt/bin Installing pyinstaller script to /opt/bin Installed /opt/lib/python3.6/site-packages/PyInstaller-3.4-py3.6.egg Processing dependencies for PyInstaller==3.4 Searching for altgraph==0.16.1 Best match: altgraph 0.16.1 Adding altgraph 0.16.1 to easy-install.pth file Using /opt/lib/python3.6/site-packages Searching for macholib==1.11 Best match: macholib 1.11 Adding macholib 1.11 to easy-install.pth file Installing macho_dump script to /opt/bin Installing macho_find script to /opt/bin Installing macho_standalone script to /opt/bin Using /opt/lib/python3.6/site-packages Searching for pefile==2019.4.18 Best match: pefile 2019.4.18 Adding pefile 2019.4.18 to easy-install.pth file Using /opt/lib/python3.6/site-packages Searching for setuptools==39.0.1 Best match: setuptools 39.0.1 Adding setuptools 39.0.1 to easy-install.pth file Installing easy_install script to /opt/bin Installing easy_install-3.6 script to /opt/bin Using /opt/lib/python3.6/site-packages Searching for future==0.17.1 Best match: future 0.17.1 Adding future 0.17.1 to easy-install.pth file Installing futurize script to /opt/bin Installing pasteurize script to /opt/bin Using /opt/lib/python3.6/site-packages Finished processing dependencies for PyInstaller==3.4 root@x067:[/home/root/python/PyInstaller-3.4]

So far, everything seems normal - only change made is to bootloader/wscript

Michael · « **Reply #4 on:** May 01, 2019, 02:26:13 PM »

Ready to proceed - but do not find your example

Finished processing dependencies for PyInstaller==3.4 root@x067:[/home/root/python/PyInstaller-3.4]cd root@x067:[/home/root]find / -fstype jfs2 -name IPP_Testing.py root@x067:[/home/root]

Michael · « **Reply #5 on:** May 01, 2019, 02:40:53 PM »

Quote

The run and run_d bootloader components also do not reference any libpython sharded object.

any ideas would be appreciated.

Look in code for calls to "dlopen", as it states in the bootloader/wscript for AIX this needs to be in the form:

+550 elif ctx.env.DEST_OS == 'aix':
+551 ctx.env.append_value('DEFINES', 'AIX')
+552 # On AIX some APIs are restricted if _ALL_SOURCE is not defined.
+553 # In the case of PyInstaller, we need the AIX specific flag RTLD_MEMBER
+554 # for dlopen() which is used to load a shared object from a library
+555 # archive. We need to load the Python library like this:
+556 # dlopen("libpython2.7.a(libpython2.7.so)", RTLD_MEMBER)
+557 ctx.env.append_value('DEFINES', '_ALL_SOURCE')

I actually have a patch added to this, to fix the issue that ctypes.find_library never worked.

root@x067:[/home/root]python3 /opt/lib/python3.6/ctypes/util.py
None
libc.a(shr.o)
libbz2.a(libbz2.so)
Using CDLL(name, RTLD_MEMBER): <CDLL 'libc.a(shr.o)', handle e at 0x301cc910>
Using cdll.LoadLibrary(): <CDLL 'libc.a(shr.o)', handle f at 0x301cc910>
rpm: librpm.so <CDLL 'librpm.so', handle 10 at 0x301cc910>
crypt :: libcrypt.a(shr.o)
crypt :: <CDLL 'libcrypt.a(shr.o)', handle 11 at 0x301cc910>
crypto:: libcrypto.a(libcrypto.so)
crypto:: <CDLL 'libcrypto.a(libcrypto.so)', handle 12 at 0x301cc910>

To disable this patch just comment the following lines:

+83 # elif sys.platform.startswith("aix"):
+84 # AIX has two styles of storing shared libraries
+85 # GNU auto_tools refer to these as svr4 and aix
+86 # svr4 (System V Release 4) is a regular file, often with .so as suffix
+87 # AIX style uses an archive (suffix .a) with members (e.g., shr.o, libssl.so)
+88 # see issue#26439 and _aix.py for more details
+89
+90 # from ctypes._aix import find_library

Now the self test runs as:

root@x067:[/home/root]python3 /opt/lib/python3.6/ctypes/util.py
None
None
None
Using CDLL(name, RTLD_MEMBER): <CDLL 'libc.a(shr.o)', handle 13 at 0x300adf10>
Using cdll.LoadLibrary(): <CDLL 'libc.a(shr.o)', handle 14 at 0x300adf10>
rpm: None <CDLL 'librpm.so', handle 15 at 0x301f4fd0>
crypt :: None
crypt :: <CDLL 'None', handle 16 at 0x300adf10>
crypto:: None
crypto:: <CDLL 'None', handle 17 at 0x30102ed0>

So, maybe your test expects this to still be 'broken' - and you could test by commenting out the lines 83 and 90.
To be absolutely complete, also comment lines 341 to 353 in Lib/ctypes/__init__.py

+341 if _sys.platform.startswith("aix"):
+342 """When the name contains ".a(" and ends with ")",
+343 e.g., "libFOO.a(libFOO.so)" - this is taken to be an
+344 archive(member) syntax for dlopen(), and the mode is adjusted.
+345 Otherwise, name is presented to dlopen() as a file argument.
+346 """
+347 # from _ctypes import RTLD_MEMBER, RTLD_NOW
+348 # RTLD_MEMBER and RTLD_NOW are suppossed to be in _ctypes, but cannot find exact logic
+349 # so hard-code it here, to get the patch done
+350 RTLD_MEMBER = 0x00040000
+351 RTLD_NOW = 0x00000002
+352 if name and name.endswith(")") and ".a(" in name:
+353 mode |= ( RTLD_MEMBER | RTLD_NOW )

FYI: the rest of the patch is in the file Lib/ctypes/_aix.py - and it is included in the 3.7 branch (and later). Slightly different, because they wanted to use a new string type, but functionally, the same.

Michael · « **Reply #6 on:** May 01, 2019, 03:17:45 PM »

p.s. - just repeated the process using the 64-bit build. No time to figure out how to add -bloadmap or -bnoquiet, but this is a (minor I hope) difference:

[13/13] Processing build/release/run ld: 0711-224 WARNING: Duplicate symbol: __start ld: 0711-224 WARNING: Duplicate symbol: .__start ld: 0711-224 WARNING: Duplicate symbol: __threads_init ld: 0711-224 WARNING: Duplicate symbol: .__threads_init ld: 0711-345 Use the -bloadmap or -bnoquiet option to obtain more information. And the ctypes/util.py test() shows the 64-bit dlopen call values - as expected. Using /opt/lib/python3.6/site-packages Finished processing dependencies for PyInstaller==3.4 root@x067:[/home/root/python/PyInstaller-3.4]pip list Package Version ----------- --------- altgraph 0.16.1 future 0.17.1 macholib 1.11 pefile 2019.4.18 pip 19.1 PyInstaller 3.4 setuptools 39.0.1 root@x067:[/home/root/python/PyInstaller-3.4]python3 /opt/lib/python3.6/ctypes/util.py None libc.a(shr_64.o) libbz2.a(libbz2.so.1) Using CDLL(name, os.RTLD_MEMBER): %s <CDLL 'libc.a(shr_64.o)', handle e at 0xa000000002934a8> Using cdll.LoadLibrary(): %s <CDLL 'libc.a(shr_64.o)', handle f at 0xa000000002934a8> crypt :: libcrypt.a(shr_64.o) crypt :: <CDLL 'libcrypt.a(shr_64.o)', handle 10 at 0xa000000002934a8> crypto:: libcrypto.a(libcrypto.so) crypto:: <CDLL 'libcrypto.a(libcrypto.so)', handle 11 at 0xa000000002934a8>

Michael · « **Reply #7 on:** May 03, 2019, 12:36:15 PM »

Starting again - with Python3-3.6.8.

Normally, I build with ./configure ... --without-computed-gotos (needed for the xlc compiler) and assumed the library built was shared. I am rebuilding now, adding the flag --enable-shared and see if that creates the file the module is looking for - as it creates whatever it creates.

More later.

martind · « **Reply #8 on:** May 03, 2019, 02:13:21 PM »

I thought pyinstaller allowed the execution of a 'compiled' python program without having to deplopy python. Therefore, is it expected that the libpython3.6.so library will be included as part of the AIXTOOLs Python package?

Michael · « **Reply #9 on:** May 03, 2019, 04:35:18 PM »

As I mentioned before - never user pyinstaller. The new packaging includes the shared library, just uses the name Pythin gives it (with a m in the name).

Will test first, then repackage with a link if needed.

root@x067:[/home/root/python/hello]lslpp -w | grep libpython /opt/lib/python3.6/config-3.6m/libpython3.6m.a /opt/lib/libpython3.6m.so

++++

With the new packaging, and some manual help - it can work. I would rather not write in /usr/lib, if I can avoid that. And I want to check with Python - why the library name it generates is 3.6m.so rather than 3.6.so (and, better, why it is not in libpython.a as member libpython3.6.so).

Will have a new package asap, bit I doubt this evening.

++++
root@x067:[/home/root/python/hello]ln -s /opt/lib/libpython3.6m.so /usr/lib/libpython3.6.so root@x067:[/home/root/python/hello]pyinstaller --onefile hello.py 94 INFO: PyInstaller: 3.4 94 INFO: Python: 3.6.8 137 INFO: Platform: AIX-1-00C291F54C00-powerpc-32bit 140 INFO: wrote /home/root/python/hello/hello.spec 176 INFO: UPX is not available. 180 INFO: Extending PYTHONPATH with paths ['/home/root/python/hello', '/home/root/python/hello'] 180 INFO: checking Analysis 180 INFO: Building Analysis because Analysis-00.toc is non existent 180 INFO: Initializing module dependency graph... 183 INFO: Initializing module graph hooks... 186 INFO: Analyzing base_library.zip ... 11142 INFO: running Analysis Analysis-00.toc 12376 INFO: Caching module hooks... 12391 INFO: Analyzing /home/root/python/hello/hello.py 12404 INFO: Loading module hooks... 12405 INFO: Loading module hook "hook-encodings.py"... 12603 INFO: Loading module hook "hook-pydoc.py"... 12605 INFO: Loading module hook "hook-xml.py"... 13425 INFO: Looking for ctypes DLLs 13426 INFO: Analyzing run-time hooks ... 13447 INFO: Looking for dynamic libraries ldd: /usr/lib/libcrypto.a: File is an archive. ldd: /usr/lib/libssl.a: File is an archive. 33700 INFO: Looking for eggs 33700 INFO: Python library not in binary dependencies. Doing additional searching... 34521 INFO: Using Python library /lib/libpython3.6.so 34536 INFO: Warnings written to /home/root/python/hello/build/hello/warn-hello.txt 34625 INFO: Graph cross-reference written to /home/root/python/hello/build/hello/xref-hello.html 34696 INFO: checking PYZ 34696 INFO: Building PYZ because PYZ-00.toc is non existent 34696 INFO: Building PYZ (ZlibArchive) /home/root/python/hello/build/hello/PYZ-00.pyz 35625 INFO: Building PYZ (ZlibArchive) /home/root/python/hello/build/hello/PYZ-00.pyz completed successfully. 35634 INFO: checking PKG 35634 INFO: Building PKG because PKG-00.toc is non existent 35634 INFO: Building PKG (CArchive) PKG-00.pkg 68882 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully. 68922 INFO: Bootloader /opt/lib/python3.6/site-packages/PyInstaller-3.4-py3.6.egg/PyInstaller/bootloader/AIX-32bit/run 68922 INFO: checking EXE 68922 INFO: Building EXE because EXE-00.toc is non existent 68923 INFO: Building EXE from EXE-00.toc 68923 INFO: Appending archive to EXE /home/root/python/hello/dist/hello 69707 INFO: Building EXE from EXE-00.toc completed successfully. root@x067:[/home/root/python/hello]

News

Author Topic: Pyinstaller issue with Python3.6 deployed via AIXTOOLs (Read 1206 times)

martind

Pyinstaller issue with Python3.6 deployed via AIXTOOLs

Michael

Re: Pyinstaller issue with Python3.6 deployed via AIXTOOLs

Michael

Re: Pyinstaller issue with Python3.6 deployed via AIXTOOLs

Michael

Re: Pyinstaller issue with Python3.6 deployed via AIXTOOLs

Michael

Re: Pyinstaller issue with Python3.6 deployed via AIXTOOLs

Michael

Re: Pyinstaller issue with Python3.6 deployed via AIXTOOLs

Michael

Re: Pyinstaller issue with Python3.6 deployed via AIXTOOLs

Michael

Re: Pyinstaller issue with Python3.6 deployed via AIXTOOLs

martind

Re: Pyinstaller issue with Python3.6 deployed via AIXTOOLs

Michael

Re: Pyinstaller issue with Python3.6 deployed via AIXTOOLs