Date: 23-10-20  Time: 06:10 AM

Author Topic: Python and wheels continuing story  (Read 2222 times)

0 Members and 1 Guest are viewing this topic.

Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1267
  • Karma: +0/-0
Python and wheels continuing story
« on: December 29, 2019, 06:58:18 PM »
While it may take some investigation to get environment variables correct - eventually this does work!
For example - one of the scientific Python modules - numpy.

root@x069:[/python]pip wheel numpy
Collecting numpy
  Using cached https://files.pythonhosted.org/packages/31/0a/5df350c29a06835d534a6c4f5681075304da38d85f1c69e5226a635a67ce/numpy-1.18.0.zip
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: numpy
  Building wheel for numpy (PEP 517) ... done
  Created wheel for numpy: filename=numpy-1.18.0-cp37-cp37m-AIX_7104_1845_64.whl size=7130503 sha256=b5456e8aefb80a0ff714532d697f4f115bcbe7d8bd2871593c0bb0b93cac10fd
  Stored in directory: /python
Successfully built numpy

Wheels means no compiler needed for later installs and an installation in a virtualenv is also possible (aka python -m pip --user ....)

raghugr

  • Jr. Member
  • **
  • Posts: 7
  • Karma: +0/-0
Re: Python and wheels continuing story
« Reply #1 on: July 01, 2020, 08:17:04 PM »
Michael this is great. I have been trying to do this on the aix but to no avail.. Can I get this numpy wheel package hosted as I am not able to compile on our aix 7.1..

Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1267
  • Karma: +0/-0
Re: Python and wheels continuing story
« Reply #2 on: July 01, 2020, 08:21:45 PM »
Not tonight - at least I don't think so.
Which version of Python are you using? As wheels are - usually - Python version dependent.

raghugr

  • Jr. Member
  • **
  • Posts: 7
  • Karma: +0/-0
Re: Python and wheels continuing story
« Reply #3 on: July 01, 2020, 09:46:41 PM »
Thank you again Michael for your reply..

Python 3.7 is what I got installed. Below are the list of packages. My end goal is to have pandas. I hope once I have numpy installed, should have pandas issues corrected. Please advise if not so. I am novice to the AIX world.

bcrypt       3.1.7
cffi         1.13.2
cryptography 2.8
Cython       0.29.20
paramiko     2.6.0
pip          20.1.1
pycparser    2.19
PyNaCl       1.3.0
setuptools   40.8.0
six          1.13.0
wheel        0.34.2


Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1267
  • Karma: +0/-0
Re: Python and wheels continuing story
« Reply #4 on: July 02, 2020, 05:23:36 AM »
OK. I'll have to build it seperately. Do not have it for py37 yet.
p.s. - what is the oslevel your run on? (oslevel -s -q | head -5).
p.p.s. I assume you are using my packaging of Python.

raghugr

  • Jr. Member
  • **
  • Posts: 7
  • Karma: +0/-0
Re: Python and wheels continuing story
« Reply #5 on: July 02, 2020, 07:54:25 AM »
Thank you again Michael. I did install the packages from aixtools. I will have a word with our admin if it is the same as they have been trying to resolve the issues I got with installing numpy,

oslevel

Known Service Packs
-------------------
7100-05-02-1810
7100-05-02-1806
7100-05-01-1731
7100-05-00-0000
7100-04-06-1806


EDIT: I confirmed with my admin that all the data were from aixtools. He is currently installing the lapack and openblas.
« Last Edit: July 02, 2020, 03:52:41 PM by raghugr »

Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1267
  • Karma: +0/-0
Re: Python and wheels continuing story
« Reply #6 on: July 03, 2020, 06:56:14 AM »
OK. My update is that I am building a new build system. I need a clean system to reinstall the try and buy Fortran compiler.
Should have more news later today.

raghugr

  • Jr. Member
  • **
  • Posts: 7
  • Karma: +0/-0
Re: Python and wheels continuing story
« Reply #7 on: July 03, 2020, 11:06:11 AM »
Thanks Michael for your effort on this. Alternatively, you can also tell me the right list of packages so I ask our team to get it right as well. Some of them I see is,

openblas
gcc
gcc-c++
gcc-gfortran
lapack

Any other?

EDIT: After support prolonged support from our team, I was able to install numpy using easy_install at last.. Problem is not sure which has caused the resolution :D .. between now trying to install pandas package and seeing some errors for missing gc++ I guess.. Hopefully, will sort it.. I will keep you posted if I learnt something on this
« Last Edit: July 03, 2020, 03:15:44 PM by raghugr »

Michael

  • Administrator
  • Hero Member
  • *****
  • Posts: 1267
  • Karma: +0/-0
Re: Python and wheels continuing story
« Reply #8 on: July 03, 2020, 03:57:04 PM »
Actually, the only package I built myself is lapack - as a static library.
bos.adt.libm contains the IBM provided libblas.a - and I try to get it to use that.
and for compilers, I have my own v11 vac(pp) compiler, and I use the "try and buy" xlf (15.2) compiler for the fortran bits.
Results:
With a 32-bit build
numpy-1.19.0 runtests.py finishes with:87 failed, 10629 passed, 94 skipped, 108 deselected, 21 xfailed, 78 errors

Ah - so you are looking for pandas - I have done that before.
But since you mention gcc and gcc-fortran - I suspect you are using the AIX Toolbox, or Perzl's packaging of python.
Try this:
(py37) aixtools@x070:[/home/aixtools/python/download/numpy-1.19.0]pip debug
WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without notice.
pip version: pip 20.1.1 from /opt/lib/python3.7/site-packages/pip (python 3.7)
sys.version: 3.7.7 (default, Jul  3 2020, 06:10:55) [C]
sys.executable: /opt/bin/python3.7
sys.getdefaultencoding: utf-8
sys.getfilesystemencoding: iso8859-1
locale.getpreferredencoding: ISO8859-1
sys.platform: aix6
sys.implementation:
  name: cpython
'cert' config value: Not specified
REQUESTS_CA_BUNDLE: None
CURL_CA_BUNDLE: None
pip._vendor.certifi.where(): /opt/lib/python3.7/site-packages/pip/_vendor/certifi/cacert.pem
pip._vendor.DEBUNDLED: False
vendored library versions:
  appdirs==1.4.3
...
  webencodings==0.5.1 (Unable to locate actual module version, using vendor.txt specified version)
Compatible tags: 27
  cp37-cp37m-aix_6106_1043_32
  cp37-abi3-aix_6106_1043_32
  cp37-none-aix_6106_1043_32
  cp36-abi3-aix_6106_1043_32
  cp35-abi3-aix_6106_1043_32
  cp34-abi3-aix_6106_1043_32
  cp33-abi3-aix_6106_1043_32
  cp32-abi3-aix_6106_1043_32
  py37-none-aix_6106_1043_32
  py3-none-aix_6106_1043_32
  ...
  [First 10 tags shown. Pass --verbose to show all.]

I am particularly curious about the tags you see - old, or new (as in New in Python3.9, and backported by me down to Python3.6).
If `pip debug` does not work, you need to update your pip.

raghugr

  • Jr. Member
  • **
  • Posts: 7
  • Karma: +0/-0
Re: Python and wheels continuing story
« Reply #9 on: July 04, 2020, 04:02:57 AM »
Thank you again Michael for your time and efforts on this.

I am sorry when my Admin specified aixtools I thought it was from yours. I am wrong, he had meant the aixtools folder from IBM FTP. So finally I got it to work. This was all the files needed gcc, gcc-c++,gcc-gfortran,openblas,lapack,python3-devel

However, somehow pip got corrupted and unable to fix this. Below is the report I am receiving when I am issuing a 'pip debug'

pip debug
Traceback (most recent call last):
  File "/opt/freeware/bin/pip", line 11, in <module>
    load_entry_point('pip==20.1.1', 'console_scripts', 'pip')()
  File "/opt/freeware/lib/python3.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/opt/freeware/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
    return ep.load()
  File "/opt/freeware/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2443, in load
    return self.resolve()
  File "/opt/freeware/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2449, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/opt/freeware/lib/python3.7/site-packages/pip/_internal/__init__.py", line 40, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/opt/freeware/lib/python3.7/site-packages/pip/_internal/cli/autocompletion.py", line 8, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/opt/freeware/lib/python3.7/site-packages/pip/_internal/cli/main_parser.py", line 11, in <module>
    from pip._internal.commands import (
  File "/opt/freeware/lib/python3.7/site-packages/pip/_internal/commands/__init__.py", line 6, in <module>
    from pip._internal.commands.completion import CompletionCommand
  File "/opt/freeware/lib/python3.7/site-packages/pip/_internal/commands/completion.py", line 6, in <module>
    from pip._internal.cli.base_command import Command
  File "/opt/freeware/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 26, in <module>
    from pip._internal.index import PackageFinder
ImportError: cannot import name 'PackageFinder' from 'pip._internal.index' (/opt/freeware/lib/python3.7/site-packages/pip/_internal/index/__init__.py)


Because of this I had to resort to 'easy_install' for both numpy and Pandas. It was successful. Any suggestions you wish to provide for fixing the pip? I tried to install pip via easy_install but that did not work either.

Once again thank you for your time and efforts on this :)