Please login or register. December 11, 2019, 10:02:10 PM

Author Topic: Pyinstaller issue with Python3.6 deployed via AIXTOOLs  (Read 3417 times)

0 Members and 1 Guest are viewing this topic.

martind

  • Jr. Member
  • **
  • Posts: 6
  • Karma: +0/-0
Pyinstaller issue with Python3.6 deployed via AIXTOOLs
« on: April 30, 2019, 02:31:20 PM »
Aixtools deployment of Python3.6 seemed normal as did the subsequent install and update of pip3.6. The install of pyinstaller via pip3.6 was problematic in that the wscript was not correct for AIX using the native AIX compiler when constructing the bootloader. Pyinstaller subsequently was installed correctly and appears to work:

# pyinstaller --onefile IPP_Testing.py
75 INFO: PyInstaller: 3.4
75 INFO: Python: 3.6.5
115 INFO: Platform: AIX-1-00F9B3C44C00-powerpc-64bit-COFF
120 INFO: wrote /home/dyer/IPP_Testing.spec
155 INFO: UPX is not available.
173 INFO: Extending PYTHONPATH with paths
['/home/dyer', '/home/dyer']
173 INFO: checking Analysis
173 INFO: Building Analysis because Analysis-00.toc is non existent
174 INFO: Initializing module dependency graph...
176 INFO: Initializing module graph hooks...
178 INFO: Analyzing base_library.zip ...
7046 INFO: running Analysis Analysis-00.toc
7420 INFO: Caching module hooks...
7431 INFO: Analyzing /home/dyer/IPP_Testing.py
8042 INFO: Loading module hooks...
8044 INFO: Loading module hook "hook-encodings.py"...
8206 INFO: Loading module hook "hook-pydoc.py"...
8207 INFO: Loading module hook "hook-xml.py"...
8337 INFO: Looking for ctypes DLLs
8337 INFO: Analyzing run-time hooks ...
8352 INFO: Looking for dynamic libraries
ldd: /usr/lib/libcrypto.a: File is an archive.
ldd: /usr/lib/libssl.a: File is an archive.
16669 INFO: Looking for eggs
16670 INFO: Python library not in binary dependencies. Doing additional searching...
17000 INFO: Using Python library /opt/lib/libpython3.6.a
17017 INFO: Warnings written to /home/dyer/build/IPP_Testing/warn-IPP_Testing.txt
17078 INFO: Graph cross-reference written to /home/dyer/build/IPP_Testing/xref-IPP_Testing.html
17134 INFO: checking PYZ
17135 INFO: Building PYZ because PYZ-00.toc is non existent
17135 INFO: Building PYZ (ZlibArchive) /home/dyer/build/IPP_Testing/PYZ-00.pyz
17823 INFO: Building PYZ (ZlibArchive) /home/dyer/build/IPP_Testing/PYZ-00.pyz completed successfully.
17834 INFO: checking PKG
17834 INFO: Building PKG because PKG-00.toc is non existent
17834 INFO: Building PKG (CArchive) PKG-00.pkg
42653 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
42668 INFO: Bootloader /opt/lib/python3.6/site-packages/PyInstaller-3.4-py3.6.egg/PyInstaller/bootloader/AIX-64bit/run
42668 INFO: checking EXE
42668 INFO: Building EXE because EXE-00.toc is non existent
42668 INFO: Building EXE from EXE-00.toc
42668 INFO: Appending archive to EXE /home/dyer/dist/IPP_Testing
42784 INFO: Building EXE from EXE-00.toc completed successfully.

however, when executing the IPP-Testing program:

# ./IPP_Testing
[4915392] Error loading Python lib '/tmp/_MEI_it7ea/libpython3.6.a(libpython3.6.so)': dlopen:   0509-022 Cannot load module /tmp/_MEI_it7ea/libpython3.6.a(libpython3.6.so).
        0509-152   Member libpython3.6.so is not found in archive

The /opt/lib library contains:

lrwxrwxrwx    1 root     system           24 Apr 30 03:40 libpython3.6.a -> /opt/lib/libpython3.6m.a
-r-xr-xr-x    1 bin      bin        10006800 Jun 01 2018  libpython3.6m.a
-rw-r--r--    1 bin      bin         1683837 Feb 02 2017  libz.a
-rwxr-xr-x    1 root     system       164632 Apr 29 01:12 libz.so.1
-rwxr-xr-x    1 root     system       164594 Apr 29 01:12 libz.so.1.2.10
-rwxr-xr-x    1 root     system       164632 Apr 29 01:12 libz.so.1.2.11
-r-xr-xr-x    1 root     system       156861 Apr 29 01:12 libz.so.1.2.8
drwxr-xr-x    2 bin      bin             256 Jun 01 2018  pkgconfig
drwxr-xr-x   35 bin      bin            8192 Apr 26 08:52 python3.6

i added the softlink to the archive. There is no libpython.3.6m.so (or libpython3.6m.so) and the libpython3.6m.a does not contain a libpython shared object.

I was wondering why Pyinstaller would have a dependency on libpython3.6.so when ldd does not report it:

# ldd IPP_Testing
IPP_Testing needs:
         /usr/lib/libdl.a(shr_64.o)
         /usr/lib/libpthreads.a(shr_xpg5_64.o)
         /usr/lib/libc.a(shr_64.o)
         /unix
         /usr/lib/libcrypt.a(shr_64.o)

The run and run_d bootloader components also do not reference any libpython sharded object.

any ideas would be appreciated.


Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1188
  • Karma: +0/-0
Re: Pyinstaller issue with Python3.6 deployed via AIXTOOLs
« Reply #1 on: May 01, 2019, 12:21:14 PM »
I'll take a look. Never used pyinstaller before.

Just checking:
a) you are using the IBM compiler (xlc)
b) IPP_testing.py is part of the pyinstaller package.

Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1188
  • Karma: +0/-0
Re: Pyinstaller issue with Python3.6 deployed via AIXTOOLs
« Reply #2 on: May 01, 2019, 12:48:48 PM »
OK. I expect it will be something specific with the PyInstaller package, and/or an issue with the way Lib/ctypes is used.

I'll have to try a manual build - as I guess you did - to understand the issue.

For record keeping - a summary of actions.
p.s. this is the 32bit version.

+-----------------------------------------------------------------------------+
                         Installing Software...
+-----------------------------------------------------------------------------+

installp:  APPLYING software for:
        aixtools.python3.rte 3.6.5.0
        aixtools.python3.man.en_US 3.6.5.0
        aixtools.python3.adt 3.6.5.0

...
+-----------------------------------------------------------------------------+
                                Summaries:
+-----------------------------------------------------------------------------+

Installation Summary
--------------------
Name                        Level           Part        Event       Result
-------------------------------------------------------------------------------
aixtools.python3.rte        3.6.5.0         USR         APPLY       SUCCESS
aixtools.python3.man.en_US  3.6.5.0         USR         APPLY       SUCCESS
aixtools.python3.adt        3.6.5.0         USR         APPLY       SUCCESS

root@x067:[/tmp]pip3 list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the
    section) to disable this warning.
    pip (9.0.3)
    setuptools (39.0.1)
    You are using pip version 9.0.3, however version 19.1 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.

    root@x067:[/tmp]pip3 install --upgrade pip
    Collecting pip
      Downloading
https://files.pythonhosted.org/packages/f9/fb/863012b13912709c13cf5cfdbfb304fa6c727659d6290438e1a88df9d848/pip-19.1-py2.py3-none-any.whl (1.4MB)
    100% |################################| 1.4MB 80kB/s
Installing collected packages: pip
  Found existing installation: pip 9.0.3
    Uninstalling pip-9.0.3:
      Successfully uninstalled pip-9.0.3
Successfully installed pip-19.1

I generally start with the download command, and see what else comes.

root@x067:[/tmp]pip download pyinstaller
Collecting pyinstaller
  Downloading https://files.pythonhosted.org/packages/03/32/0e0de593f129bf1d1e77eed562496d154ef4460fd5cecfd78612ef39a0cc/PyInstaller-3.4.tar.gz (3.5MB)
     |################################| 3.5MB 225kB/s
  Saved ./PyInstaller-3.4.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting setuptools (from pyinstaller)
  Using cached https://files.pythonhosted.org/packages/ec/51/f45cea425fd5cb0b0380f5b0f048ebc1da5b417e48d304838c02d6288a1e/setuptools-41.0.1-py2.py3-none-any.whl
  Saved ./setuptools-41.0.1-py2.py3-none-any.whl
Collecting pefile>=2017.8.1 (from pyinstaller)
  Downloading https://files.pythonhosted.org/packages/36/58/acf7f35859d541985f0a6ea3c34baaefbfaee23642cf11e85fe36453ae77/pefile-2019.4.18.tar.gz (62kB)
     |################################| 71kB 196kB/s
  Saved ./pefile-2019.4.18.tar.gz
Collecting macholib>=1.8 (from pyinstaller)
  Downloading https://files.pythonhosted.org/packages/41/f1/6d23e1c79d68e41eb592338d90a33af813f98f2b04458aaf0b86908da2d8/macholib-1.11-py2.py3-none-any.whl
  Saved ./macholib-1.11-py2.py3-none-any.whl
Collecting altgraph (from pyinstaller)
  Downloading https://files.pythonhosted.org/packages/0a/cc/646187eac4b797069e2e6b736f14cdef85dbe405c9bfc7803ef36e4f62ef/altgraph-0.16.1-py2.py3-none-any.whl
  Saved ./altgraph-0.16.1-py2.py3-none-any.whl
Collecting future (from pefile>=2017.8.1->pyinstaller)
  Downloading https://files.pythonhosted.org/packages/90/52/e20466b85000a181e1e144fd8305caf2cf475e2f9674e797b222f8105f5f/future-0.17.1.tar.gz (829kB)
     |################################| 829kB 81kB/s
  Saved ./future-0.17.1.tar.gz
Successfully downloaded pyinstaller setuptools pefile macholib altgraph future


I am going to guess a bit, as to what is most 'root' to all, and install them one-by-one.

root@x067:[/tmp]pip list
Package    Version
---------- -------
pip        19.1
setuptools 39.0.1

-rw-r--r--    1 root     system      3487849 May 01 12:34 PyInstaller-3.4.tar.gz
-rw-r--r--    1 root     system       575966 May 01 12:35 setuptools-41.0.1-py2.py3-none-any.whl
-rw-r--r--    1 root     system        62374 May 01 12:35 pefile-2019.4.18.tar.gz
-rw-r--r--    1 root     system        36562 May 01 12:35 macholib-1.11-py2.py3-none-any.whl
-rw-r--r--    1 root     system        20311 May 01 12:35 altgraph-0.16.1-py2.py3-none-any.whl
-rw-r--r--    1 root     system       829119 May 01 12:35 future-0.17.1.tar.gz

root@x067:[/tmp]pip install setuptools
Requirement already satisfied: setuptools in /opt/lib/python3.6/site-packages (39.0.1)

hmm

root@x067:[/tmp]pip install future
Collecting future
  Using cached https://files.pythonhosted.org/packages/90/52/e20466b85000a181e1e144fd8305caf2cf475e2f9674e797b222f8105f5f/future-0.17.1.tar.gz
Installing collected packages: future
  Running setup.py install for future ... done
Successfully installed future-0.17.1

root@x067:[/tmp]pip install altgraph
Collecting altgraph
  Using cached https://files.pythonhosted.org/packages/0a/cc/646187eac4b797069e2e6b736f14cdef85dbe405c9bfc7803ef36e4f62ef/altgraph-0.16.1-py2.py3-none-any.whl
Installing collected packages: altgraph
Successfully installed altgraph-0.16.1

root@x067:[/tmp]pip install macholib
Collecting macholib
  Using cached https://files.pythonhosted.org/packages/41/f1/6d23e1c79d68e41eb592338d90a33af813f98f2b04458aaf0b86908da2d8/macholib-1.11-py2.py3-none-any.whl
Requirement already satisfied: altgraph>=0.15 in /opt/lib/python3.6/site-packages (from macholib) (0.16.1)
Installing collected packages: macholib
Successfully installed macholib-1.11

root@x067:[/tmp]pip install macholib
Collecting macholib
  Using cached https://files.pythonhosted.org/packages/41/f1/6d23e1c79d68e41eb592338d90a33af813f98f2b04458aaf0b86908da2d8/macholib-1.11-py2.py3-none-any.whl
Requirement already satisfied: altgraph>=0.15 in /opt/lib/python3.6/site-packages (from macholib) (0.16.1)
Installing collected packages: macholib
Successfully installed macholib-1.11

root@x067:[/tmp]pip install pefile
Collecting pefile
  Using cached https://files.pythonhosted.org/packages/36/58/acf7f35859d541985f0a6ea3c34baaefbfaee23642cf11e85fe36453ae77/pefile-2019.4.18.tar.gz
Requirement already satisfied: future in /opt/lib/python3.6/site-packages (from pefile) (0.17.1)
Installing collected packages: pefile
  Running setup.py install for pefile ... done
Successfully installed pefile-2019.4.18

So, even though there is a newer version of setuptools - at this point it is not needed.

+++++++
root@x067:[/tmp]pip install pyinstaller
Collecting pyinstaller
  Using cached https://files.pythonhosted.org/packages/03/32/0e0de593f129bf1d1e77eed562496d154ef4460fd5cecfd78612ef39a0cc/PyInstaller-3.4.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: altgraph in /opt/lib/python3.6/site-packages (from pyinstaller) (0.16.1)
Requirement already satisfied: pefile>=2017.8.1 in /opt/lib/python3.6/site-packages (from pyinstaller) (2019.4.18)
Requirement already satisfied: macholib>=1.8 in /opt/lib/python3.6/site-packages (from pyinstaller) (1.11)
Requirement already satisfied: setuptools in /opt/lib/python3.6/site-packages (from pyinstaller) (39.0.1)
Requirement already satisfied: future in /opt/lib/python3.6/site-packages (from pefile>=2017.8.1->pyinstaller) (0.17.1)
Building wheels for collected packages: pyinstaller
  Building wheel for pyinstaller (PEP 517) ... error
  ERROR: Complete output from command /opt/bin/python3.6 /opt/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmp4rjg0pdd:
  ERROR: running bdist_wheel
  running build
  running build_bootloader
  Setting top to                           : /tmp/pip-install-3adnqnud/pyinstaller/bootloader
  Setting out to                           : /tmp/pip-install-3adnqnud/pyinstaller/bootloader/build
  Python Version                           : 3.6.5 (default, May  9 2018, 18:35:41) [C]
  Checking for 'xlc' (C compiler)          : /usr/vacpp/bin/xlc_r
  Checking size of pointer                 : 4
  Platform                                 : AIX-32bit detected based on compiler
  Checking for flags -q32                  : yes
  Checking for program '/usr/vacpp/bin/xlc_r' : /usr/vacpp/bin/xlc_r
  Checking for program 'strip'                : /usr/vacpp/bin/xlc_r
  Checking for library dl                     : yes
  Checking for library m                      : yes
  Checking for library z                      : yes
  Checking for function unsetenv              : yes
  Checking for function mkdtemp               : yes
  'configure' finished successfully (11.109s)
  'all' finished successfully (0.000s)
  'distclean' finished successfully (0.084s)
  Setting top to                           : /tmp/pip-install-3adnqnud/pyinstaller/bootloader
  Setting out to                           : /tmp/pip-install-3adnqnud/pyinstaller/bootloader/build
  Python Version                           : 3.6.5 (default, May  9 2018, 18:35:41) [C]
  Checking for 'xlc' (C compiler)          : /usr/vacpp/bin/xlc_r
  Checking size of pointer                 : 4
  Platform                                 : AIX-32bit detected based on compiler
  Checking for flags -q32                  : yes
  Checking for program '/usr/vacpp/bin/xlc_r' : /usr/vacpp/bin/xlc_r
  Checking for program 'strip'                : /usr/vacpp/bin/xlc_r
  Checking for library dl                     : yes
  Checking for library m                      : yes
  Checking for library z                      : yes
  Checking for function unsetenv              : yes
  Checking for function mkdtemp               : yes
  'configure' finished successfully (3.684s)
  'make_all' finished successfully (0.012s)
  Waf: Entering directory `/tmp/pip-install-3adnqnud/pyinstaller/bootloader/build/debug'
  [ 1/13] Compiling src/pyi_python.c
  [ 2/13] Compiling src/main.c
  [ 3/13] Compiling src/pyi_path.c
  [ 4/13] Compiling src/pyi_pythonlib.c
  [ 5/13] Compiling src/pyi_win32_utils.c
  "../../src/pyi_win32_utils.c", line 574.21: 1506-356 (W) Compilation unit is empty.

  [ 6/13] Compiling src/pyi_main.c
  [ 7/13] Compiling src/pyi_launch.c
  [ 8/13] Compiling src/pyi_python27_compat.c
  [ 9/13] Compiling src/pyi_utils.c
  [10/13] Compiling src/pyi_global.c
  [11/13] Compiling src/pyi_archive.c
  [12/13] Linking build/debug/run_d
  ld: 0706-006 Cannot find or open library file: -l Z
        ld:open(): A file or directory in the path name does not exist.

  Waf: Leaving directory `/tmp/pip-install-3adnqnud/pyinstaller/bootloader/build/debug'
  Build failed
   -> task in 'run_d' failed with exit status 255 (run with -v to display more information)
  No precompiled bootloader found. Trying to compile it for you ...
  ERROR: Failed compiling the bootloader. Please compile manually and rerun setup.py
  ----------------------------------------
  ERROR: Failed building wheel for pyinstaller
  Running setup.py clean for pyinstaller
Failed to build pyinstaller
ERROR: Could not build wheels for pyinstaller which use PEP 517 and cannot be installed directly

[/tt][/list]

Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1188
  • Karma: +0/-0
Re: Pyinstaller issue with Python3.6 deployed via AIXTOOLs
« Reply #3 on: May 01, 2019, 02:23:50 PM »
Quote
Aixtools deployment of Python3.6 seemed normal as did the subsequent install and update of pip3.6. The install of pyinstaller via pip3.6 was problematic in that the wscript was not correct for AIX using the native AIX compiler when constructing the bootloader. Pyinstaller subsequently was installed correctly and appears to work:

I am having some issues with getting wscript 'patched', so I still do not get boatloader built/installed.

Documenting for posterity...

[12/13] Linking build/debug/run_d
ld: 0706-006 Cannot find or open library file: -l Z
        ld:open(): A file or directory in the path name does not exist.

*** The first error (trying to ling -lZ is fixed here ***
  +763      else:
  +764          # Linux, Darwin (MacOSX), ...
  +765          # Only the libs found will actually be used, so it's safe to list all
  +766          # here. The decision if a lib is required for a specific platform is
  +767          # made in the configure phase.
  +768          libs = ['DL', 'M', 'Z',  # 'z' - zlib, 'm' - math,
  +769                  'THR']  # may be used on FreBSD
  +770          staticlibs = []
  +771          if ctx.env.DEST_OS == 'aix':
  +772              # link statically with zlib
  +773              libs.remove('Z')
  +774              staticlibs.append('z')
"bootloader/wscript" [Modified] The cursor is at line 774 of 825

Now step 12 of 13 passes, and step 13 fails with:

[12/13] Linking build/debug/run_d
[13/13] Processing build/debug/run_d
/usr/vacpp/bin/xlc_r: 1501-216 (W) command option -32 is not recognized - passed to ld
ld: 0706-012 The -X flag is not recognized.
ld: 0706-012 The -3 flag is not recognized.
ld: 0706-012 The -2 flag is not recognized.

Skip the strip flags whatever...

  +459      if ctx.env.CC_NAME != 'msvc':
  +460          # This tool allows reducing the size of executables.
  +461          ctx.find_program([assoc_programm(ctx, 'strip')], var='STRIP')
  +462          ctx.load('strip', tooldir='tools')
  +463          # There is a strip flag for AIX environment
  +464          # if ctx.env.DEST_OS == 'aix':
  +465          #     if ctx.env.PYI_ARCH == '32bit':
  +466          #         ctx.env.append_value('STRIPFLAGS', '-X32')
  +467          #     elif ctx.env.PYI_ARCH == '64bit':
  +468          #         ctx.env.append_value('STRIPFLAGS', '-X64')
  +469

and the manual build process succeeds..

and the manual install succeeds...

....
byte-compiling build/bdist.aix-7.1/egg/PyInstaller/utils/win32/winmanifest.py to winmanifest.cpython-36.pyc
byte-compiling build/bdist.aix-7.1/egg/PyInstaller/utils/win32/winresource.py to winresource.cpython-36.pyc
byte-compiling build/bdist.aix-7.1/egg/PyInstaller/utils/win32/winutils.py to winutils.cpython-36.pyc
creating build/bdist.aix-7.1/egg/EGG-INFO
copying PyInstaller.egg-info/PKG-INFO -> build/bdist.aix-7.1/egg/EGG-INFO
copying PyInstaller.egg-info/SOURCES.txt -> build/bdist.aix-7.1/egg/EGG-INFO
copying PyInstaller.egg-info/dependency_links.txt -> build/bdist.aix-7.1/egg/EGG-INFO
copying PyInstaller.egg-info/entry_points.txt -> build/bdist.aix-7.1/egg/EGG-INFO
copying PyInstaller.egg-info/not-zip-safe -> build/bdist.aix-7.1/egg/EGG-INFO
copying PyInstaller.egg-info/requires.txt -> build/bdist.aix-7.1/egg/EGG-INFO
copying PyInstaller.egg-info/top_level.txt -> build/bdist.aix-7.1/egg/EGG-INFO
creating dist
creating 'dist/PyInstaller-3.4-py3.6.egg' and adding 'build/bdist.aix-7.1/egg' to it
removing 'build/bdist.aix-7.1/egg' (and everything under it)
Processing PyInstaller-3.4-py3.6.egg
creating /opt/lib/python3.6/site-packages/PyInstaller-3.4-py3.6.egg
Extracting PyInstaller-3.4-py3.6.egg to /opt/lib/python3.6/site-packages
Adding PyInstaller 3.4 to easy-install.pth file
Installing pyi-archive_viewer script to /opt/bin
Installing pyi-bindepend script to /opt/bin
Installing pyi-grab_version script to /opt/bin
Installing pyi-makespec script to /opt/bin
Installing pyi-set_version script to /opt/bin
Installing pyinstaller script to /opt/bin

Installed /opt/lib/python3.6/site-packages/PyInstaller-3.4-py3.6.egg
Processing dependencies for PyInstaller==3.4
Searching for altgraph==0.16.1
Best match: altgraph 0.16.1
Adding altgraph 0.16.1 to easy-install.pth file

Using /opt/lib/python3.6/site-packages
Searching for macholib==1.11
Best match: macholib 1.11
Adding macholib 1.11 to easy-install.pth file
Installing macho_dump script to /opt/bin
Installing macho_find script to /opt/bin
Installing macho_standalone script to /opt/bin

Using /opt/lib/python3.6/site-packages
Searching for pefile==2019.4.18
Best match: pefile 2019.4.18
Adding pefile 2019.4.18 to easy-install.pth file

Using /opt/lib/python3.6/site-packages
Searching for setuptools==39.0.1
Best match: setuptools 39.0.1
Adding setuptools 39.0.1 to easy-install.pth file
Installing easy_install script to /opt/bin
Installing easy_install-3.6 script to /opt/bin

Using /opt/lib/python3.6/site-packages
Searching for future==0.17.1
Best match: future 0.17.1
Adding future 0.17.1 to easy-install.pth file
Installing futurize script to /opt/bin
Installing pasteurize script to /opt/bin

Using /opt/lib/python3.6/site-packages
Finished processing dependencies for PyInstaller==3.4
root@x067:[/home/root/python/PyInstaller-3.4]


So far, everything seems normal - only change made is to bootloader/wscript



Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1188
  • Karma: +0/-0
Re: Pyinstaller issue with Python3.6 deployed via AIXTOOLs
« Reply #4 on: May 01, 2019, 02:26:13 PM »
Ready to proceed - but do not find your example


Finished processing dependencies for PyInstaller==3.4
root@x067:[/home/root/python/PyInstaller-3.4]cd
root@x067:[/home/root]find / -fstype jfs2 -name IPP_Testing.py
root@x067:[/home/root]

Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1188
  • Karma: +0/-0
Re: Pyinstaller issue with Python3.6 deployed via AIXTOOLs
« Reply #5 on: May 01, 2019, 02:40:53 PM »
Quote
The run and run_d bootloader components also do not reference any libpython sharded object.

any ideas would be appreciated.

Look in code for calls to "dlopen", as it states in the bootloader/wscript for AIX this needs to be in the form:

  +550      elif ctx.env.DEST_OS == 'aix':
  +551          ctx.env.append_value('DEFINES', 'AIX')
  +552          # On AIX some APIs are restricted if _ALL_SOURCE is not defined.
  +553          # In the case of PyInstaller, we need the AIX specific flag RTLD_MEMBER
  +554          # for dlopen() which is used to load a shared object from a library
  +555          # archive. We need to load the Python library like this:
  +556          #  dlopen("libpython2.7.a(libpython2.7.so)", RTLD_MEMBER)
  +557          ctx.env.append_value('DEFINES', '_ALL_SOURCE')

I actually have a patch added to this, to fix the issue that ctypes.find_library never worked.

root@x067:[/home/root]python3 /opt/lib/python3.6/ctypes/util.py
None
libc.a(shr.o)
libbz2.a(libbz2.so)
Using CDLL(name, RTLD_MEMBER): <CDLL 'libc.a(shr.o)', handle e at 0x301cc910>
Using cdll.LoadLibrary(): <CDLL 'libc.a(shr.o)', handle f at 0x301cc910>
rpm: librpm.so <CDLL 'librpm.so', handle 10 at 0x301cc910>
crypt :: libcrypt.a(shr.o)
crypt :: <CDLL 'libcrypt.a(shr.o)', handle 11 at 0x301cc910>
crypto:: libcrypto.a(libcrypto.so)
crypto:: <CDLL 'libcrypto.a(libcrypto.so)', handle 12 at 0x301cc910>

To disable this patch just comment the following lines:

   +83  # elif sys.platform.startswith("aix"):
   +84      # AIX has two styles of storing shared libraries
   +85      # GNU auto_tools refer to these as svr4 and aix
   +86      # svr4 (System V Release 4) is a regular file, often with .so as suffix
   +87      # AIX style uses an archive (suffix .a) with members (e.g., shr.o, libssl.so)
   +88      # see issue#26439 and _aix.py for more details
   +89
   +90  #     from ctypes._aix import find_library

Now the self test runs as:

root@x067:[/home/root]python3 /opt/lib/python3.6/ctypes/util.py
None
None
None
Using CDLL(name, RTLD_MEMBER): <CDLL 'libc.a(shr.o)', handle 13 at 0x300adf10>
Using cdll.LoadLibrary(): <CDLL 'libc.a(shr.o)', handle 14 at 0x300adf10>
rpm: None <CDLL 'librpm.so', handle 15 at 0x301f4fd0>
crypt :: None
crypt :: <CDLL 'None', handle 16 at 0x300adf10>
crypto:: None
crypto:: <CDLL 'None', handle 17 at 0x30102ed0>

So, maybe your test expects this to still be 'broken' - and you could test by commenting out the lines 83 and 90.
To be absolutely complete, also comment lines 341 to 353 in Lib/ctypes/__init__.py

  +341          if _sys.platform.startswith("aix"):
  +342              """When the name contains ".a(" and ends with ")",
  +343                 e.g., "libFOO.a(libFOO.so)" - this is taken to be an
  +344                 archive(member) syntax for dlopen(), and the mode is adjusted.
  +345                 Otherwise, name is presented to dlopen() as a file argument.
  +346              """
  +347              # from _ctypes import RTLD_MEMBER, RTLD_NOW
  +348              # RTLD_MEMBER and RTLD_NOW are suppossed to be in _ctypes, but cannot find exact logic
  +349              # so hard-code it here, to get the patch done
  +350              RTLD_MEMBER = 0x00040000
  +351              RTLD_NOW    = 0x00000002
  +352              if name and name.endswith(")") and ".a(" in name:
  +353                  mode |= ( RTLD_MEMBER | RTLD_NOW )

FYI: the rest of the patch is in the file Lib/ctypes/_aix.py - and it is included in the 3.7 branch (and later). Slightly different, because they wanted to use a new string type, but functionally, the same.

Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1188
  • Karma: +0/-0
Re: Pyinstaller issue with Python3.6 deployed via AIXTOOLs
« Reply #6 on: May 01, 2019, 03:17:45 PM »
p.s. - just repeated the process using the 64-bit build. No time to figure out how to add -bloadmap or -bnoquiet, but this is a (minor I hope) difference:

[13/13] Processing build/release/run
ld: 0711-224 WARNING: Duplicate symbol: __start
ld: 0711-224 WARNING: Duplicate symbol: .__start
ld: 0711-224 WARNING: Duplicate symbol: __threads_init
ld: 0711-224 WARNING: Duplicate symbol: .__threads_init
ld: 0711-345 Use the -bloadmap or -bnoquiet option to obtain more information.

And the ctypes/util.py test() shows the 64-bit dlopen call values - as expected.

Using /opt/lib/python3.6/site-packages
Finished processing dependencies for PyInstaller==3.4
root@x067:[/home/root/python/PyInstaller-3.4]pip list
Package     Version
----------- ---------
altgraph    0.16.1
future      0.17.1
macholib    1.11
pefile      2019.4.18
pip         19.1
PyInstaller 3.4
setuptools  39.0.1

root@x067:[/home/root/python/PyInstaller-3.4]python3 /opt/lib/python3.6/ctypes/util.py
None
libc.a(shr_64.o)
libbz2.a(libbz2.so.1)
Using CDLL(name, os.RTLD_MEMBER): %s <CDLL 'libc.a(shr_64.o)', handle e at 0xa000000002934a8>
Using cdll.LoadLibrary(): %s <CDLL 'libc.a(shr_64.o)', handle f at 0xa000000002934a8>
crypt :: libcrypt.a(shr_64.o)
crypt :: <CDLL 'libcrypt.a(shr_64.o)', handle 10 at 0xa000000002934a8>
crypto:: libcrypto.a(libcrypto.so)
crypto:: <CDLL 'libcrypto.a(libcrypto.so)', handle 11 at 0xa000000002934a8>

Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1188
  • Karma: +0/-0
Re: Pyinstaller issue with Python3.6 deployed via AIXTOOLs
« Reply #7 on: May 03, 2019, 12:36:15 PM »
Starting again - with Python3-3.6.8.

Normally, I build with ./configure ... --without-computed-gotos (needed for the xlc compiler) and assumed the library built was shared. I am rebuilding now, adding the flag --enable-shared and see if that creates the file the module is looking for - as it creates whatever it creates.

More later.

martind

  • Jr. Member
  • **
  • Posts: 6
  • Karma: +0/-0
Re: Pyinstaller issue with Python3.6 deployed via AIXTOOLs
« Reply #8 on: May 03, 2019, 02:13:21 PM »
I thought pyinstaller allowed the execution of a 'compiled' python program without having to deplopy python. Therefore, is it expected that the libpython3.6.so library will be included as part of the AIXTOOLs Python package?

Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1188
  • Karma: +0/-0
Re: Pyinstaller issue with Python3.6 deployed via AIXTOOLs
« Reply #9 on: May 03, 2019, 04:35:18 PM »
As I mentioned before - never user pyinstaller. The new packaging includes the shared library, just uses the name Pythin gives it (with a m in the name).

Will test first, then repackage with a link if needed.

root@x067:[/home/root/python/hello]lslpp -w | grep libpython
  /opt/lib/python3.6/config-3.6m/libpython3.6m.a
  /opt/lib/libpython3.6m.so


++++

With the new packaging, and some manual help - it can work. I would rather not write in /usr/lib, if I can avoid that. And I want to check with Python - why the library name it generates is 3.6m.so rather than 3.6.so (and, better, why it is not in libpython.a as member libpython3.6.so).

Will have a new package asap, bit I doubt this evening.

++++
root@x067:[/home/root/python/hello]ln -s /opt/lib/libpython3.6m.so /usr/lib/libpython3.6.so
root@x067:[/home/root/python/hello]pyinstaller --onefile hello.py
94 INFO: PyInstaller: 3.4
94 INFO: Python: 3.6.8
137 INFO: Platform: AIX-1-00C291F54C00-powerpc-32bit
140 INFO: wrote /home/root/python/hello/hello.spec
176 INFO: UPX is not available.
180 INFO: Extending PYTHONPATH with paths
['/home/root/python/hello', '/home/root/python/hello']
180 INFO: checking Analysis
180 INFO: Building Analysis because Analysis-00.toc is non existent
180 INFO: Initializing module dependency graph...
183 INFO: Initializing module graph hooks...
186 INFO: Analyzing base_library.zip ...
11142 INFO: running Analysis Analysis-00.toc
12376 INFO: Caching module hooks...
12391 INFO: Analyzing /home/root/python/hello/hello.py
12404 INFO: Loading module hooks...
12405 INFO: Loading module hook "hook-encodings.py"...
12603 INFO: Loading module hook "hook-pydoc.py"...
12605 INFO: Loading module hook "hook-xml.py"...
13425 INFO: Looking for ctypes DLLs
13426 INFO: Analyzing run-time hooks ...
13447 INFO: Looking for dynamic libraries
ldd: /usr/lib/libcrypto.a: File is an archive.
ldd: /usr/lib/libssl.a: File is an archive.
33700 INFO: Looking for eggs
33700 INFO: Python library not in binary dependencies. Doing additional searching...
34521 INFO: Using Python library /lib/libpython3.6.so
34536 INFO: Warnings written to /home/root/python/hello/build/hello/warn-hello.txt
34625 INFO: Graph cross-reference written to /home/root/python/hello/build/hello/xref-hello.html
34696 INFO: checking PYZ
34696 INFO: Building PYZ because PYZ-00.toc is non existent
34696 INFO: Building PYZ (ZlibArchive) /home/root/python/hello/build/hello/PYZ-00.pyz
35625 INFO: Building PYZ (ZlibArchive) /home/root/python/hello/build/hello/PYZ-00.pyz completed successfully.
35634 INFO: checking PKG
35634 INFO: Building PKG because PKG-00.toc is non existent
35634 INFO: Building PKG (CArchive) PKG-00.pkg
68882 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
68922 INFO: Bootloader /opt/lib/python3.6/site-packages/PyInstaller-3.4-py3.6.egg/PyInstaller/bootloader/AIX-32bit/run
68922 INFO: checking EXE
68922 INFO: Building EXE because EXE-00.toc is non existent
68923 INFO: Building EXE from EXE-00.toc
68923 INFO: Appending archive to EXE /home/root/python/hello/dist/hello
69707 INFO: Building EXE from EXE-00.toc completed successfully.
root@x067:[/home/root/python/hello]