Date: 12-07-20  Time: 23:34 PM

Author Topic: python3 pynacl module install just hangs - trying to compile libsodium  (Read 311 times)

0 Members and 1 Guest are viewing this topic.

hammertime

  • Jr. Member
  • **
  • Posts: 9
  • Karma: +0/-0
Hi Michael,

Have you had any joy compiling this yourself?  Ultimately I'm trying to install module pysftp.  I've found a few hits online but no joy so far so any advice appreciated.

For me it hangs as below -
 # pip3 install  pynacl
Collecting pynacl
  Downloading PyNaCl-1.4.0.tar.gz (3.4 MB)
     |################################| 3.4 MB 3.2 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: cffi>=1.4.1 in /opt/lib/python3.8/site-packages (from pynacl) (1.14.0)
Requirement already satisfied: six in /opt/lib/python3.8/site-packages (from pynacl) (1.14.0)
Requirement already satisfied: pycparser in /opt/lib/python3.8/site-packages (from cffi>=1.4.1->pynacl) (2.20)
Building wheels for collected packages: pynacl
  Building wheel for pynacl (PEP 517) ... -


 ps shows -
    root  8323076  8716394 120 00:39:28  pts/1  0:00 /bin/sh ../../libtool --silent --tag=CC --mode=compile xlc_r -DPACKAGE_NAME="libsodium" -DPACKAGE_TARNAME="libsodium" -DPACKAGE_VERSION="1.0.18" -DPACKAGE_STRING="libsodium 1.0.18" -DPACKAGE_BUGREPORT="https://github.com/jedisct1/libsodium/issues" -DPACKAGE_URL="https://github.com/jedisct1/libsodium" -DPACKAGE="libsodium" -DVERSION="1.0.18" -DHAVE_PTHREAD_PRIO_INHERIT=1 -DHAVE_PTHREAD=1 -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -D__EXTENSIONS__=1 -D_ALL_SOURCE=1 -D_GNU_SOURCE=1 -D_POSIX_PTHREAD_SEMANTICS=1 -D_TANDEM_SOURCE=1 -DHAVE_C_VARARRAYS=1 -DHAVE_CATCHABLE_SEGV=1 -DHAVE_CATCHABLE_ABRT=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=".libs/" -DHAVE_SYS_MMAN_H=1 -DNATIVE_BIG_ENDIAN=1 -DHAVE_INLINE_ASM=1 -DCPU_UNALIGNED_ACCESS=1 -DHAVE_ATOMIC_OPS=1 -DHAVE_ALLOCA_H=1 -DHAVE_ALLOCA=1 -DHAVE_MMAP=1 -DHAVE_MLOCK=1 -DHAVE_MADVISE=1 -DHAVE_MPROTECT=1 -DHAVE_POSIX_MEMALIGN=1 -DHAVE_GETPID=1 -DHAVE_NANOSLEEP=1 -DHAVE_MEMSET_S=1 -DCONFIGURED=1 -I. -I/tmp/pip-install-ub689zph/pynacl/src/libsodium/src/libsodium -I/tmp/pip-install-ub689zph/pynacl/src/libsodium/src/libsodium/include/sodium -I./include/sodium -D_FORTIFY_SOURCE=2 -DNDEBUG -O -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -q64 -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -q64 -c -o crypto_stream/chacha20/ref/libsodium_la-chacha20_ref.lo /tmp/pip-install-ub689zph/pynacl/src/libsodium/src/libsodium/crypto_stream/chacha20/ref/chacha20_ref.c


Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1256
  • Karma: +0/-0
Years ago - I had no issues with sodium - that I can recall - but I'll give it a shot.
Which Python3 version are you using?
Michael

hammertime

  • Jr. Member
  • **
  • Posts: 9
  • Karma: +0/-0
Thanks Michael,

# lslpp -l aixtools.python.py38.*
  Fileset                      Level  State      Description
  ----------------------------------------------------------------------------
Path: /usr/lib/objrepos
  aixtools.python.py38.adt   3.8.2.0  COMMITTED  python py38 ADT files
  aixtools.python.py38.man.en_US
                             3.8.2.0  COMMITTED  python py38 man pages
  aixtools.python.py38.rte   3.8.2.0  COMMITTED  python py38 31-Mar-2020

Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1256
  • Karma: +0/-0
I had guessed as much...
I have a clean system (well almost - experimenting with jenkins), but no compiler, so it has errors of course.
First thing I do is create a "clean" virtualenv that I switch to - to not disturb anything at a system level.

But my first phase is always to just 'download' the target and see where issues come up.
So, since I do not have a compiler - I run into cffi as an issue - no C compiler to build it:
Code: [Select]
root@x072:[/home/root]py38
(py38) root@x072:[/home/root]mkdir py38
(py38) root@x072:[/home/root]pip download pynacl
Collecting pynacl
  Downloading PyNaCl-1.4.0.tar.gz (3.4 MB)
     |################################| 3.4 MB 3.1 MB/s
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/python/py38/bin/python /opt/python/py38/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-_57fryna/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel 'cffi>=1.4.1; python_implementation != '"'"'PyPy'"'"''

hammertime

  • Jr. Member
  • **
  • Posts: 9
  • Karma: +0/-0
Indeed, I have installed a trial version of C compiler so cffi installs for me.

# pip3 install cffi
Collecting cffi
  Downloading cffi-1.14.0.tar.gz (463 kB)
     |################################| 463 kB 3.1 MB/s
Requirement already satisfied: pycparser in /opt/lib/python3.8/site-packages (from cffi) (2.20)
Using legacy setup.py install for cffi, since package 'wheel' is not installed.
Installing collected packages: cffi
    Running setup.py install for cffi ... done
Successfully installed cffi-1.14.0


I also installed your libffi package as a prereq.

# lslpp -l aixtools.libffi.*
  Fileset                      Level  State      Description
  ----------------------------------------------------------------------------
Path: /usr/lib/objrepos
  aixtools.libffi.man.en_US  3.2.1.1  COMMITTED  aixtools libffi man pages
  aixtools.libffi.rte        3.2.1.1  COMMITTED  aixtools libffi 14-Mar-2017
  aixtools.libffi.share      3.2.1.1  COMMITTED  aixtools libffi universal
                                                 files

Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1256
  • Karma: +0/-0
So, on a new system with compiler - on my second attempt I am at:
Code: [Select]
(py38) root@x065:[/data/prj/python/py38/download]export OBJECT_MODE=64
(py38) root@x065:[/data/prj/python/py38/download]pip download pynacl
Collecting pynacl
  Using cached PyNaCl-1.4.0.tar.gz (3.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Saved ./PyNaCl-1.4.0.tar.gz
Collecting six
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
  Saved ./six-1.15.0-py2.py3-none-any.whl
Collecting cffi>=1.4.1
  Using cached cffi-1.14.0.tar.gz (463 kB)
  Saved ./cffi-1.14.0.tar.gz
Collecting pycparser
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
  Saved ./pycparser-2.20-py2.py3-none-any.whl
Successfully downloaded pynacl six cffi pycparser
At this point, I unpack the package (PyNaCl-1.4.0.tar.gz) and call setup.py manually.
And - I want to make sure I have wheels of all the packages just installed - for my compilerless target.

Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1256
  • Karma: +0/-0
Heads Up: this package needs GNU tar!
The @ in a filename is a giveaway!
Code: [Select]
(py38) root@x065:[/data/prj/python/py38/download]gzip -dc PyNaCl-1.4.0.tar.gz | tar tf -
././@PaxHeader
PyNaCl-1.4.0/
././@PaxHeader
PyNaCl-1.4.0/CHANGELOG.rst

After installation:
Code: [Select]
Installation Summary
--------------------
Name                        Level           Part        Event       Result
-------------------------------------------------------------------------------
aixtools.gnu.tar.share      1.26.0.0        USR         APPLY       SUCCESS
aixtools.gnu.tar.rte        1.26.0.0        USR         APPLY       SUCCESS
aixtools.gnu.tar.adt        1.26.0.0        USR         APPLY       SUCCESS
(py38) root@x065:[/data/httpd/tools/gnu]cd -
/data/prj/python/py38/download
(py38) root@x065:[/data/prj/python/py38/download]which tar
/opt/bin/tar
(py38) root@x065:[/data/prj/python/py38/download]gzip -dc PyNaCl-1.4.0.tar.gz | tar tf - | head
PyNaCl-1.4.0/
PyNaCl-1.4.0/CHANGELOG.rst
PyNaCl-1.4.0/INSTALL.rst
PyNaCl-1.4.0/LICENSE
PyNaCl-1.4.0/MANIFEST.in
PyNaCl-1.4.0/PKG-INFO
PyNaCl-1.4.0/README.rst
PyNaCl-1.4.0/docs/
PyNaCl-1.4.0/docs/Makefile
PyNaCl-1.4.0/docs/_static/

Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1256
  • Karma: +0/-0
I come quite far, I think - and shall try creating a wheel - BUT - I am nervous something else will get in way.
Code: [Select]
(py38) root@x065:[/data/prj/python/py38/download/PyNaCl-1.4.0]python setup.py build
running build
running build_py
creating build
creating build/lib.aix-5307-0747-64-3.8
creating build/lib.aix-5307-0747-64-3.8/nacl
copying src/nacl/__init__.py -> build/lib.aix-5307-0747-64-3.8/nacl
copying src/nacl/encoding.py -> build/lib.aix-5307-0747-64-3.8/nacl
...PASS: xchacha20============================================================================
Testsuite summary for libsodium 1.0.18
============================================================================
# TOTAL: 77
# PASS:  77
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================

...
generating cffi module 'build/temp.aix-5307-0747-64-3.8/_sodium.c'
building '_sodium' extension
creating build/temp.aix-5307-0747-64-3.8/build
creating build/temp.aix-5307-0747-64-3.8/build/temp.aix-5307-0747-64-3.8
xlc_r -DNDEBUG -O -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -q64 -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -q64 -DNDEBUG -O -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -q64 -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -q64 -I/data/prj/python/py38/include -I/opt/include/python3.8 -Ibuild/temp.aix-5307-0747-64-3.8/include -c build/temp.aix-5307-0747-64-3.8/_sodium.c -o build/temp.aix-5307-0747-64-3.8/build/temp.aix-5307-0747-64-3.8/_sodium.o
/opt/lib/python3.8/config-3.8/ld_so_aix xlc_r -bI:/opt/lib/python3.8/config-3.8/python.exp -DNDEBUG -O -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -q64 -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -q64 build/temp.aix-5307-0747-64-3.8/build/temp.aix-5307-0747-64-3.8/_sodium.o -Lbuild/temp.aix-5307-0747-64-3.8/lib -Lbuild/temp.aix-5307-0747-64-3.8/lib64 -Lbuild/temp.aix-5307-0747-64-3.8 -lsodium -lsodium -o build/lib.aix-5307-0747-64-3.8/nacl/_sodium.abi3.so
ld: 0711-327 WARNING: Entry point not found: PyInit__sodium.abi3
(py38) root@x065:[/data/prj/python/py38/download/PyNaCl-1.4.0]echo $?
0

Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1256
  • Karma: +0/-0
Actually, it looks successful...
Code: [Select]
(py38) root@x065:[/data/prj/python/py38/download/PyNaCl-1.4.0]ls -l ../*.gz dist
-rw-r--r--   1 root     felt        3416950 Jun 27 08:56 ../PyNaCl-1.4.0.tar.gz
-rw-r--r--   1 root     felt         463065 Jun 27 08:56 ../cffi-1.14.0.tar.gz

dist:
total 7184
-rw-r--r--   1 root     system       256173 Jun 27 09:25 PyNaCl-1.4.0-cp38-cp38-aix_5307_0747_64.whl
-rw-r--r--   1 root     system      3415014 Jun 27 09:23 PyNaCl-1.4.0.tar.gz
So, I'll rename the cffi and PyNaCl wheels and install them on my test server.

hammertime

  • Jr. Member
  • **
  • Posts: 9
  • Karma: +0/-0
Thanks Michael.  I'm not sure I'd have ever worked out the need for GNU tar.

Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1256
  • Karma: +0/-0
Maybe this helps - having a clean build system (that can be trashed, as the Try and Buy compiler can not be reinstalled - they hide it's history that it was installed "somewhere")
especially with a build systems that is "lowest level" supported by your environment!
Build there (as wheels), then rename (aka link) on the target host using this first draft -
Code: [Select]
#!/usr/bin/env python3
"""
Examine arguments for aix pep425 tags
and propose ln (link) commands to give a new name
to an existing wheel for AIX.

Copyright 2020 by AIXTOOLS.NET, Michael Felt
"""

import sys, glob, sysconfig

# save runttime-tag and reverse a copy for comparision
rt_tag = sysconfig.get_platform().split("-")
_tag = list(rt_tag)
_tag.reverse()

# loop through all arguments and look for a AIX tags
for idx in range(1,len(sys.argv)):
  for arg in glob.glob(sys.argv[idx]):
    wheel=arg.split("-")
    if not wheel[-1].startswith("aix_"):
        continue
    # have a potential tag, reverse tag for comparision
    file_tag=wheel[-1].split(".")[0].split("_")
    file_tag.reverse()

    # if the tags are equal, or they are not 32-bit|64-bit - continue
    if (_tag == file_tag) or _tag[0] not in ["32", "64"]:
      continue

    # if the running tag is greater AND same bit-size link names
    elif (_tag > file_tag) and _tag[0] == file_tag[0]:
      wheel[-1] = "_".join(rt_tag) + ".whl"
      print("ln {} {}".format(arg, "-".join(wheel)))
And, as I am looking at a more economical way to support aixtools - I am going to experiment, at least as far as python goes - with opening issues on https://github.com/aixtools/cpython - for things that can be solved via packaging (of Python).
I guess I'll do the same with https://github.com/aixtools/packaging (which is pypa (aka pip et al)) related.

hammertime

  • Jr. Member
  • **
  • Posts: 9
  • Karma: +0/-0
Thank you ever so much time for your guidance Michael!

Not working for me quite yet.  I shall try again from the top following the steps you ran.

<snip>
PASS: core_ristretto255
ERROR: pwhash_scrypt
ERROR: pwhash_scrypt_ll
PASS: scalarmult_ed25519
PASS: scalarmult_ristretto255
PASS: siphashx24
PASS: xchacha20
============================================================================
Testsuite summary for libsodium 1.0.18
============================================================================
# TOTAL: 77
# PASS:  75
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 2
<SNIP>

Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1256
  • Karma: +0/-0
Did you get this working, i.e. PASS all tests?
If not, would you want my .whl?