Date: 22-10-21  Time: 22:48 PM

Author Topic: Python 3.8 - shared library support  (Read 638 times)

0 Members and 1 Guest are viewing this topic.

Jordan.Stopford

  • New Member
  • *
  • Posts: 3
  • Karma: +0/-0
Python 3.8 - shared library support
« on: August 17, 2021, 10:38:43 AM »
Is there a build of Python 3.8 with shared library support? I've tried compiling this myself but to no avail.

I've got a project that I need to use with PyInstaller hence the need for shared-library support.

I did try with a Python 3.6.8 build I found from AIXTOOLS that was done with shared-library support but this fails to compile pynacl and other things, so thought it might be best to just get the latest.

Any assistance helpful.

Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1315
  • Karma: +0/-0
Re: Python 3.8 - shared library support
« Reply #1 on: August 17, 2021, 11:37:21 AM »
What failure were you getting? During the compile.

Jordan.Stopford

  • New Member
  • *
  • Posts: 3
  • Karma: +0/-0
Re: Python 3.8 - shared library support
« Reply #2 on: August 17, 2021, 12:53:40 PM »
It was failing to compile the _ctypes module due to missing libffi libs/headers. I am using your aixtools.libffi package to get these onto my system and then setting CPPFLAGS and LDFLAGS appropriately for the make command but it still fails to find them. I have seen various things online that other people had struggled with this but I was not able to use their hints (such as using CPPFLAGS instead of CFLAGS, and setting PKG_CONFIG_PATH to the location of the libffi libs).

What I have done is this:

Install aixtools.libffi latest from your website
Downloaded Python-3.8.5 source package from the Python website
Ran the configuration using CC=xlc_r OPT="-O2" ./configure --without-computed-gotos --enable-shared --enable-optimizations
make (fails)
CFLAGS="-I/opt/include/ffi" make (fails)
CPPFLAGS="-I/opt/include/ffi" make (fails)
CPPFLAGS="-I/opt/include/ffi" LDFLAGS="-R/opt/lib -L/opt/lib" make (fails)
CPPFLAGS="-I/opt/include/ffi" LDFLAGS="-R/opt/lib -L/opt/lib" PKG_CONFIG_PATH="/opt/lib/pkgconfig" make (fails)

As this didn't work, I reverted to using your Python 3.6.8 build with shared-library support, I then get different errors depending on the version of pynacl I try to use:
With the latest pynacl:
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
libtool:   error: require no space between '-L' and '/usr/lib'
  make: 1254-004 The error code from the last command is 1.


  Stop.
  make: 1254-004 The error code from the last command is 1.


  Stop.
  make: 1254-004 The error code from the last command is 1.


  Stop.
  make: 1254-004 The error code from the last command is 1.


  Stop.
  Traceback (most recent call last):
    File "/home/jenkins/workspace/env/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 349, in <module>
      main()
    File "/home/jenkins/workspace/env/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 331, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/jenkins/workspace/env/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 249, in build_wheel
      metadata_directory)
    File "/tmp/pip-build-env-sm9lt_ja/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 222, in build_wheel
      wheel_directory, config_settings)
    File "/tmp/pip-build-env-sm9lt_ja/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 207, in _build_with_temp_dir
      self.run_setup()
    File "/tmp/pip-build-env-sm9lt_ja/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 150, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 264, in <module>
      "Programming Language :: Python :: 3.8",
    File "/tmp/pip-build-env-sm9lt_ja/overlay/lib/python3.6/site-packages/setuptools/__init__.py", line 153, in setup
      return distutils.core.setup(**attrs)
    File "/opt/lib/python3.6/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/opt/lib/python3.6/distutils/dist.py", line 955, in run_commands
      self.run_command(cmd)
    File "/opt/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-sm9lt_ja/overlay/lib/python3.6/site-packages/wheel/bdist_wheel.py", line 299, in run
      self.run_command('build')
    File "/opt/lib/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/opt/lib/python3.6/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/opt/lib/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "setup.py", line 184, in run
      subprocess.check_call(["make"] + make_args, cwd=build_temp)
    File "/opt/lib/python3.6/subprocess.py", line 311, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['make']' returned non-zero exit status 2.
  ----------------------------------------
  ERROR: Failed building wheel for pynacl
Failed to build pynacl
ERROR: Could not build wheels for pynacl which use PEP 517 and cannot be installed directly
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

With pynacl pegged at version 1.3.0 (saw this as a suggestion somewhere) it seems to get further but fails when running the unit tests:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  PASS: aead_aes256gcm
  PASS: aead_chacha20poly1305
  PASS: aead_xchacha20poly1305
  PASS: auth
  PASS: auth2
  PASS: auth3
  PASS: auth5
  PASS: auth6
  PASS: auth7
  PASS: box
  PASS: box2
  PASS: box7
  PASS: box8
  PASS: box_easy
  PASS: box_easy2
  PASS: box_seal
  PASS: box_seed
  PASS: chacha20
  PASS: codecs
  PASS: core1
  PASS: core2
  PASS: core3
  PASS: core4
  PASS: core5
  PASS: core6
  PASS: ed25519_convert
  PASS: generichash
  PASS: generichash2
  PASS: generichash3
  PASS: hash
  PASS: hash3
  PASS: kdf
  PASS: keygen
  PASS: kx
  PASS: metamorphic
  PASS: misuse
  PASS: onetimeauth
  PASS: onetimeauth2
  PASS: onetimeauth7
  PASS: pwhash_argon2i
  PASS: pwhash_argon2id
  PASS: randombytes
  PASS: scalarmult
  PASS: scalarmult2
  PASS: scalarmult5
  PASS: scalarmult6
  PASS: scalarmult7
  PASS: secretbox
  PASS: secretbox2
  PASS: secretbox7
  PASS: secretbox8
  PASS: secretbox_easy
  PASS: secretbox_easy2
  PASS: secretstream
  PASS: shorthash
  PASS: sign
  PASS: sodium_core
  PASS: sodium_utils
  PASS: sodium_version
  PASS: stream
  PASS: stream2
  PASS: stream3
  PASS: stream4
  PASS: verify1
  PASS: sodium_utils2
  PASS: sodium_utils3
  PASS: core_ed25519
  /tmp/pip-install-ory_sfel/pynacl_459235fbc55848d9a60d79c75b09ead8/src/libsodium/build-aux/test-driver[107]: 8192002 Killed
  FAIL: pwhash_scrypt
  PASS: pwhash_scrypt_ll
  PASS: scalarmult_ed25519
  PASS: siphashx24
  PASS: xchacha20
  ============================================================================
  Testsuite summary for libsodium 1.0.16
  ============================================================================
  # TOTAL: 72
  # PASS:  71
  # SKIP:  0
  # XFAIL: 0
  # FAIL:  1
  # XPASS: 0
  # ERROR: 0
  ============================================================================
  See test/default/test-suite.log
  Please report to https://github.com/jedisct1/libsodium/issues
  ============================================================================
  make: 1254-004 The error code from the last command is 1.


  Stop.
  make: 1254-004 The error code from the last command is 2.


  Stop.
  make: 1254-004 The error code from the last command is 2.


  Stop.
  make: 1254-004 The error code from the last command is 1.


  Stop.
  make: 1254-004 The error code from the last command is 1.


  Stop.
  Traceback (most recent call last):
    File "/home/jenkins/workspace/env/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 349, in <module>
      main()
    File "/home/jenkins/workspace/env/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 331, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/jenkins/workspace/env/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 249, in build_wheel
      metadata_directory)
    File "/tmp/pip-build-env-g6p48ucn/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 222, in build_wheel
      wheel_directory, config_settings)
    File "/tmp/pip-build-env-g6p48ucn/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 207, in _build_with_temp_dir
      self.run_setup()
    File "/tmp/pip-build-env-g6p48ucn/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 259, in run_setup
      self).run_setup(setup_script=setup_script)
    File "/tmp/pip-build-env-g6p48ucn/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 150, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 255, in <module>
      "Programming Language :: Python :: 3.7",
    File "/tmp/pip-build-env-g6p48ucn/overlay/lib/python3.6/site-packages/setuptools/__init__.py", line 153, in setup
      return distutils.core.setup(**attrs)
    File "/opt/lib/python3.6/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/opt/lib/python3.6/distutils/dist.py", line 955, in run_commands
      self.run_command(cmd)
    File "/opt/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-g6p48ucn/overlay/lib/python3.6/site-packages/wheel/bdist_wheel.py", line 299, in run
      self.run_command('build')
    File "/opt/lib/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/opt/lib/python3.6/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/opt/lib/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "setup.py", line 179, in run
      subprocess.check_call(["make", "check"] + make_args, cwd=build_temp)
    File "/opt/lib/python3.6/subprocess.py", line 311, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['make', 'check']' returned non-zero exit status 2.
  ----------------------------------------
  ERROR: Failed building wheel for pynacl
Failed to build pynacl
ERROR: Could not build wheels for pynacl which use PEP 517 and cannot be installed directly
----------------------------------------------------------------------------------------------------------------------------------------------------------------------

Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1315
  • Karma: +0/-0
Re: Python 3.8 - shared library support
« Reply #3 on: August 17, 2021, 02:25:50 PM »
a) if using 64-bit python, be sure to use

export OBJECT_MODE=64

before you start your build.

b) I see my `static` libffi is still at version 3.2.1, (http://download.aixtools.net/static/static.libffi.3.2.1.0.I) and I frequently use that version (forgetting that I had a newer version at 3.3). Yout might try that, and, iirc, the correct CFLAGS would be -I/opt/include (as the programs should be using #include <ffi/fileIwant.h>)

Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1315
  • Karma: +0/-0
Re: Python 3.8 - shared library support
« Reply #4 on: August 17, 2021, 02:39:35 PM »
Anyway, I am building py38-3.8.11 - with a patch adding wheel support.

I see if I can also build and 'wheel' pynacl for you.

Jordan.Stopford

  • New Member
  • *
  • Posts: 3
  • Karma: +0/-0
Re: Python 3.8 - shared library support
« Reply #5 on: August 17, 2021, 02:49:40 PM »
Thanks Michael that is greatly appreciated. For reference I was going by some documentation that I presume you had written here:

https://github.com/aixtools/pythonAIXDOC/tree/master/Misc

Although I'm not sure whether the python build process has changed since you wrote that guide...!

Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1315
  • Karma: +0/-0
Re: Python 3.8 - shared library support
« Reply #6 on: August 17, 2021, 03:18:10 PM »
Some things have changed. Not too great iirc. I posted it on github MUCH later than I actually wrote it. And queries about it have not been made, so I was not too anxious to update it.

Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1315
  • Karma: +0/-0
Re: Python 3.8 - shared library support
« Reply #7 on: August 17, 2021, 04:54:04 PM »
FYI.

Testing new py38 (64-bit) build, and I get:

make[4]: Entering directory `/home/aixtools/download/PyNaCl-1.4.0/build/temp.aix-5312-1241-64-3.8/test/default'
PASS: aead_aes256gcm
PASS: aead_aes256gcm2
PASS: aead_chacha20poly1305
PASS: aead_chacha20poly13052
PASS: aead_xchacha20poly1305
PASS: auth
PASS: auth2
PASS: auth3
PASS: auth5
PASS: auth6
PASS: auth7
PASS: box
PASS: box2
PASS: box7
PASS: box8
PASS: box_easy
PASS: box_easy2
PASS: box_seal
PASS: box_seed
PASS: chacha20
PASS: codecs
PASS: core1
PASS: core2
PASS: core3
PASS: core4
PASS: core5
PASS: core6
PASS: ed25519_convert
PASS: generichash
PASS: generichash2
PASS: generichash3
PASS: hash
PASS: hash3
PASS: kdf
PASS: keygen
PASS: kx
PASS: metamorphic
PASS: misuse
PASS: onetimeauth
PASS: onetimeauth2
PASS: onetimeauth7
PASS: pwhash_argon2i
PASS: pwhash_argon2id
PASS: randombytes
PASS: scalarmult
PASS: scalarmult2
PASS: scalarmult5
PASS: scalarmult6
PASS: scalarmult7
PASS: scalarmult8
PASS: secretbox
PASS: secretbox2
PASS: secretbox7
PASS: secretbox8
PASS: secretbox_easy
PASS: secretbox_easy2
PASS: secretstream
PASS: shorthash
PASS: sign
PASS: sodium_core
PASS: sodium_utils
PASS: sodium_version
PASS: stream
PASS: stream2
PASS: stream3
PASS: stream4
PASS: verify1
PASS: sodium_utils2
PASS: sodium_utils3
PASS: core_ed25519
PASS: core_ristretto255
PASS: pwhash_scrypt
PASS: pwhash_scrypt_ll
PASS: scalarmult_ed25519
PASS: scalarmult_ristretto255
PASS: siphashx24
PASS: xchacha20
============================================================================
Testsuite summary for libsodium 1.0.18
============================================================================
# TOTAL: 77
# PASS:  77
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================

This is a virtualenv:

(py38) aixtools@x065:[/home/aixtools/download/PyNaCl-1.4.0]pip3 list
Package    Version
---------- -------
cffi       1.14.6
pip        21.2.3
pycparser  2.20
PyNaCl     1.4.0
setuptools 57.4.0
six        1.16.0
wheel      0.37.0