AIXTOOLS, IBM AIX and POWER Portal
AIX => AIXTOOLS => Topic started by: hammertime on June 26, 2020, 11:41:32 PM
-
Hi Michael,
Have you had any joy compiling this yourself? Ultimately I'm trying to install module pysftp. I've found a few hits online but no joy so far so any advice appreciated.
For me it hangs as below -
# pip3 install pynacl
Collecting pynacl
Downloading PyNaCl-1.4.0.tar.gz (3.4 MB)
|################################| 3.4 MB 3.2 MB/s
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... done
Requirement already satisfied: cffi>=1.4.1 in /opt/lib/python3.8/site-packages (from pynacl) (1.14.0)
Requirement already satisfied: six in /opt/lib/python3.8/site-packages (from pynacl) (1.14.0)
Requirement already satisfied: pycparser in /opt/lib/python3.8/site-packages (from cffi>=1.4.1->pynacl) (2.20)
Building wheels for collected packages: pynacl
Building wheel for pynacl (PEP 517) ... -
ps shows -
root 8323076 8716394 120 00:39:28 pts/1 0:00 /bin/sh ../../libtool --silent --tag=CC --mode=compile xlc_r -DPACKAGE_NAME="libsodium" -DPACKAGE_TARNAME="libsodium" -DPACKAGE_VERSION="1.0.18" -DPACKAGE_STRING="libsodium 1.0.18" -DPACKAGE_BUGREPORT="https://github.com/jedisct1/libsodium/issues" -DPACKAGE_URL="https://github.com/jedisct1/libsodium" -DPACKAGE="libsodium" -DVERSION="1.0.18" -DHAVE_PTHREAD_PRIO_INHERIT=1 -DHAVE_PTHREAD=1 -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -D__EXTENSIONS__=1 -D_ALL_SOURCE=1 -D_GNU_SOURCE=1 -D_POSIX_PTHREAD_SEMANTICS=1 -D_TANDEM_SOURCE=1 -DHAVE_C_VARARRAYS=1 -DHAVE_CATCHABLE_SEGV=1 -DHAVE_CATCHABLE_ABRT=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=".libs/" -DHAVE_SYS_MMAN_H=1 -DNATIVE_BIG_ENDIAN=1 -DHAVE_INLINE_ASM=1 -DCPU_UNALIGNED_ACCESS=1 -DHAVE_ATOMIC_OPS=1 -DHAVE_ALLOCA_H=1 -DHAVE_ALLOCA=1 -DHAVE_MMAP=1 -DHAVE_MLOCK=1 -DHAVE_MADVISE=1 -DHAVE_MPROTECT=1 -DHAVE_POSIX_MEMALIGN=1 -DHAVE_GETPID=1 -DHAVE_NANOSLEEP=1 -DHAVE_MEMSET_S=1 -DCONFIGURED=1 -I. -I/tmp/pip-install-ub689zph/pynacl/src/libsodium/src/libsodium -I/tmp/pip-install-ub689zph/pynacl/src/libsodium/src/libsodium/include/sodium -I./include/sodium -D_FORTIFY_SOURCE=2 -DNDEBUG -O -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -q64 -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -q64 -c -o crypto_stream/chacha20/ref/libsodium_la-chacha20_ref.lo /tmp/pip-install-ub689zph/pynacl/src/libsodium/src/libsodium/crypto_stream/chacha20/ref/chacha20_ref.c
-
Years ago - I had no issues with sodium - that I can recall - but I'll give it a shot.
Which Python3 version are you using?
Michael
-
Thanks Michael,
# lslpp -l aixtools.python.py38.*
Fileset Level State Description
----------------------------------------------------------------------------
Path: /usr/lib/objrepos
aixtools.python.py38.adt 3.8.2.0 COMMITTED python py38 ADT files
aixtools.python.py38.man.en_US
3.8.2.0 COMMITTED python py38 man pages
aixtools.python.py38.rte 3.8.2.0 COMMITTED python py38 31-Mar-2020
-
I had guessed as much...
I have a clean system (well almost - experimenting with jenkins), but no compiler, so it has errors of course.
First thing I do is create a "clean" virtualenv that I switch to - to not disturb anything at a system level.
But my first phase is always to just 'download' the target and see where issues come up.
So, since I do not have a compiler - I run into cffi as an issue - no C compiler to build it:
root@x072:[/home/root]py38
(py38) root@x072:[/home/root]mkdir py38
(py38) root@x072:[/home/root]pip download pynacl
Collecting pynacl
Downloading PyNaCl-1.4.0.tar.gz (3.4 MB)
|################################| 3.4 MB 3.1 MB/s
Installing build dependencies ... error
ERROR: Command errored out with exit status 1:
command: /opt/python/py38/bin/python /opt/python/py38/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-_57fryna/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel 'cffi>=1.4.1; python_implementation != '"'"'PyPy'"'"''
-
Indeed, I have installed a trial version of C compiler so cffi installs for me.
# pip3 install cffi
Collecting cffi
Downloading cffi-1.14.0.tar.gz (463 kB)
|################################| 463 kB 3.1 MB/s
Requirement already satisfied: pycparser in /opt/lib/python3.8/site-packages (from cffi) (2.20)
Using legacy setup.py install for cffi, since package 'wheel' is not installed.
Installing collected packages: cffi
Running setup.py install for cffi ... done
Successfully installed cffi-1.14.0
I also installed your libffi package as a prereq.
# lslpp -l aixtools.libffi.*
Fileset Level State Description
----------------------------------------------------------------------------
Path: /usr/lib/objrepos
aixtools.libffi.man.en_US 3.2.1.1 COMMITTED aixtools libffi man pages
aixtools.libffi.rte 3.2.1.1 COMMITTED aixtools libffi 14-Mar-2017
aixtools.libffi.share 3.2.1.1 COMMITTED aixtools libffi universal
files
-
So, on a new system with compiler - on my second attempt I am at:
(py38) root@x065:[/data/prj/python/py38/download]export OBJECT_MODE=64
(py38) root@x065:[/data/prj/python/py38/download]pip download pynacl
Collecting pynacl
Using cached PyNaCl-1.4.0.tar.gz (3.4 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... done
Saved ./PyNaCl-1.4.0.tar.gz
Collecting six
Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Saved ./six-1.15.0-py2.py3-none-any.whl
Collecting cffi>=1.4.1
Using cached cffi-1.14.0.tar.gz (463 kB)
Saved ./cffi-1.14.0.tar.gz
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Saved ./pycparser-2.20-py2.py3-none-any.whl
Successfully downloaded pynacl six cffi pycparser
At this point, I unpack the package (PyNaCl-1.4.0.tar.gz) and call setup.py manually.
And - I want to make sure I have wheels of all the packages just installed - for my compilerless target.
-
Heads Up: this package needs GNU tar!
The @ in a filename is a giveaway!
(py38) root@x065:[/data/prj/python/py38/download]gzip -dc PyNaCl-1.4.0.tar.gz | tar tf -
././@PaxHeader
PyNaCl-1.4.0/
././@PaxHeader
PyNaCl-1.4.0/CHANGELOG.rst
After installation:Installation Summary
--------------------
Name Level Part Event Result
-------------------------------------------------------------------------------
aixtools.gnu.tar.share 1.26.0.0 USR APPLY SUCCESS
aixtools.gnu.tar.rte 1.26.0.0 USR APPLY SUCCESS
aixtools.gnu.tar.adt 1.26.0.0 USR APPLY SUCCESS
(py38) root@x065:[/data/httpd/tools/gnu]cd -
/data/prj/python/py38/download
(py38) root@x065:[/data/prj/python/py38/download]which tar
/opt/bin/tar
(py38) root@x065:[/data/prj/python/py38/download]gzip -dc PyNaCl-1.4.0.tar.gz | tar tf - | head
PyNaCl-1.4.0/
PyNaCl-1.4.0/CHANGELOG.rst
PyNaCl-1.4.0/INSTALL.rst
PyNaCl-1.4.0/LICENSE
PyNaCl-1.4.0/MANIFEST.in
PyNaCl-1.4.0/PKG-INFO
PyNaCl-1.4.0/README.rst
PyNaCl-1.4.0/docs/
PyNaCl-1.4.0/docs/Makefile
PyNaCl-1.4.0/docs/_static/
-
I come quite far, I think - and shall try creating a wheel - BUT - I am nervous something else will get in way.
(py38) root@x065:[/data/prj/python/py38/download/PyNaCl-1.4.0]python setup.py build
running build
running build_py
creating build
creating build/lib.aix-5307-0747-64-3.8
creating build/lib.aix-5307-0747-64-3.8/nacl
copying src/nacl/__init__.py -> build/lib.aix-5307-0747-64-3.8/nacl
copying src/nacl/encoding.py -> build/lib.aix-5307-0747-64-3.8/nacl
...PASS: xchacha20============================================================================
Testsuite summary for libsodium 1.0.18
============================================================================
# TOTAL: 77
# PASS: 77
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
============================================================================
...
generating cffi module 'build/temp.aix-5307-0747-64-3.8/_sodium.c'
building '_sodium' extension
creating build/temp.aix-5307-0747-64-3.8/build
creating build/temp.aix-5307-0747-64-3.8/build/temp.aix-5307-0747-64-3.8
xlc_r -DNDEBUG -O -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -q64 -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -q64 -DNDEBUG -O -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -q64 -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -q64 -I/data/prj/python/py38/include -I/opt/include/python3.8 -Ibuild/temp.aix-5307-0747-64-3.8/include -c build/temp.aix-5307-0747-64-3.8/_sodium.c -o build/temp.aix-5307-0747-64-3.8/build/temp.aix-5307-0747-64-3.8/_sodium.o
/opt/lib/python3.8/config-3.8/ld_so_aix xlc_r -bI:/opt/lib/python3.8/config-3.8/python.exp -DNDEBUG -O -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -q64 -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -q64 build/temp.aix-5307-0747-64-3.8/build/temp.aix-5307-0747-64-3.8/_sodium.o -Lbuild/temp.aix-5307-0747-64-3.8/lib -Lbuild/temp.aix-5307-0747-64-3.8/lib64 -Lbuild/temp.aix-5307-0747-64-3.8 -lsodium -lsodium -o build/lib.aix-5307-0747-64-3.8/nacl/_sodium.abi3.so
ld: 0711-327 WARNING: Entry point not found: PyInit__sodium.abi3
(py38) root@x065:[/data/prj/python/py38/download/PyNaCl-1.4.0]echo $?
0
-
Actually, it looks successful...
(py38) root@x065:[/data/prj/python/py38/download/PyNaCl-1.4.0]ls -l ../*.gz dist
-rw-r--r-- 1 root felt 3416950 Jun 27 08:56 ../PyNaCl-1.4.0.tar.gz
-rw-r--r-- 1 root felt 463065 Jun 27 08:56 ../cffi-1.14.0.tar.gz
dist:
total 7184
-rw-r--r-- 1 root system 256173 Jun 27 09:25 PyNaCl-1.4.0-cp38-cp38-aix_5307_0747_64.whl
-rw-r--r-- 1 root system 3415014 Jun 27 09:23 PyNaCl-1.4.0.tar.gzSo, I'll rename the cffi and PyNaCl wheels and install them on my test server.
-
Thanks Michael. I'm not sure I'd have ever worked out the need for GNU tar.
-
Maybe this helps - having a clean build system (that can be trashed, as the Try and Buy compiler can not be reinstalled - they hide it's history that it was installed "somewhere")
especially with a build systems that is "lowest level" supported by your environment!
Build there (as wheels), then rename (aka link) on the target host using this first draft -
#!/usr/bin/env python3
"""
Examine arguments for aix pep425 tags
and propose ln (link) commands to give a new name
to an existing wheel for AIX.
Copyright 2020 by AIXTOOLS.NET, Michael Felt
"""
import sys, glob, sysconfig
# save runttime-tag and reverse a copy for comparision
rt_tag = sysconfig.get_platform().split("-")
_tag = list(rt_tag)
_tag.reverse()
# loop through all arguments and look for a AIX tags
for idx in range(1,len(sys.argv)):
for arg in glob.glob(sys.argv[idx]):
wheel=arg.split("-")
if not wheel[-1].startswith("aix_"):
continue
# have a potential tag, reverse tag for comparision
file_tag=wheel[-1].split(".")[0].split("_")
file_tag.reverse()
# if the tags are equal, or they are not 32-bit|64-bit - continue
if (_tag == file_tag) or _tag[0] not in ["32", "64"]:
continue
# if the running tag is greater AND same bit-size link names
elif (_tag > file_tag) and _tag[0] == file_tag[0]:
wheel[-1] = "_".join(rt_tag) + ".whl"
print("ln {} {}".format(arg, "-".join(wheel)))
And, as I am looking at a more economical way to support aixtools - I am going to experiment, at least as far as python goes - with opening issues on https://github.com/aixtools/cpython - for things that can be solved via packaging (of Python).
I guess I'll do the same with https://github.com/aixtools/packaging (which is pypa (aka pip et al)) related.
-
Thank you ever so much time for your guidance Michael!
Not working for me quite yet. I shall try again from the top following the steps you ran.
<snip>
PASS: core_ristretto255
ERROR: pwhash_scrypt
ERROR: pwhash_scrypt_ll
PASS: scalarmult_ed25519
PASS: scalarmult_ristretto255
PASS: siphashx24
PASS: xchacha20
============================================================================
Testsuite summary for libsodium 1.0.18
============================================================================
# TOTAL: 77
# PASS: 75
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 2
<SNIP>
-
Did you get this working, i.e. PASS all tests?
If not, would you want my .whl?
