Python 3.8, ModuleNotFoundError: No module named '_sqlite3'

[hammertime]

Apologies if I'm doing something silly here.  Trying to import some Python modules and getting the _sqlite3 error.  The first hits from a web search suggest that sqlite-devel would need to have been installed on the server where Python distribution was compiled.  Does this sound likely?

[ts01]:root:/root # pip3 list
Package    Version
---------- -------
nltk       3.4.5
pip        20.0.2
setuptools 46.1.3
six        1.14.0

[ts01]:root:/root # python3
Python 3.8.0 (default, Oct 26 2019, 17:53:18) [C] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import nltk
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/lib/python3.8/site-packages/nltk/__init__.py", line 150, in <module>
    from nltk.translate import *
  File "/opt/lib/python3.8/site-packages/nltk/translate/__init__.py", line 23, in <module>
    from nltk.translate.meteor_score import meteor_score as meteor
  File "/opt/lib/python3.8/site-packages/nltk/translate/meteor_score.py", line 10, in <module>
    from nltk.stem.porter import PorterStemmer
  File "/opt/lib/python3.8/site-packages/nltk/stem/__init__.py", line 29, in <module>
    from nltk.stem.snowball import SnowballStemmer
  File "/opt/lib/python3.8/site-packages/nltk/stem/snowball.py", line 32, in <module>
    from nltk.corpus import stopwords
  File "/opt/lib/python3.8/site-packages/nltk/corpus/__init__.py", line 66, in <module>
    from nltk.corpus.reader import *
  File "/opt/lib/python3.8/site-packages/nltk/corpus/reader/__init__.py", line 105, in <module>
    from nltk.corpus.reader.panlex_lite import *
  File "/opt/lib/python3.8/site-packages/nltk/corpus/reader/panlex_lite.py", line 15, in <module>
    import sqlite3
  File "/opt/lib/python3.8/sqlite3/__init__.py", line 23, in <module>
    from sqlite3.dbapi2 import *
  File "/opt/lib/python3.8/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'
>>>

[ts01]:root:/root # lslpp -L |grep python3
  aixtools.python3.adt       3.8.0.0    C     F    python3 ADT files
  aixtools.python3.man.en_US
                             3.8.0.0    C     F    python3 man pages
  aixtools.python3.rte       3.8.0.0    C     F    python3 26-Oct-2019

[ts01]:root:/root # oslevel -s
7100-05-04-1914

[Michael]

No, nothing wrong on your side. Noone (before) has wanted sqllite. However, I have been building new versions.
I'll get a new version out asap - 3.8.2 iirc. And I'll have an additional patch installed so that you can work with wheels.
Note: I am changing the fileset name - to allow multiple versions to be installed.
aixtools.python.py3X (so in this case, would be aixtools.python.py38).
I'll try and have it refuse to install if aixtools.python3.* is already installed. but not sure if I'll have time to get that built-in today.

[Michael]

The unpatched version (i.e., no wheel support) says:
pip version: pip 19.2.3 from /opt/lib/python3.8/site-packages/pip (python 3.
sys.version: 3.8.2 (default, Mar  6 2020, 17:40:06) [C]
sys.executable: /opt/bin/python3.8
sys.getdefaultencoding: utf-8
sys.getfilesystemencoding: iso8859-1
locale.getpreferredencoding: ISO8859-1
sys.platform: aix
sys.implementation:
  name: cpython
Compatible tags: 22
  cp38-cp38-aix_5_3
  cp38-abi3-aix_5_3
  cp38-none-aix_5_3
  cp37-abi3-aix_5_3
  cp36-abi3-aix_5_3
  cp35-abi3-aix_5_3
  cp34-abi3-aix_5_3
  cp33-abi3-aix_5_3
  cp32-abi3-aix_5_3
  py3-none-aix_5_3

So, I am going to add the PEP425 support that IS included in the Python3.9 release.
More soon.

[Michael]

So, with the correct pep425 code patch pip3 debug says:

Code: [Select]

root@x065:[/data/prj/python/py38-3.8.2]pip3 debug
WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without notice.
pip version: pip 19.2.3 from /opt/lib/python3.8/site-packages/pip (python 3.
sys.version: 3.8.2 (default, Mar 31 2020, 16:07:53) [C]
sys.executable: /opt/bin/python3.8
sys.getdefaultencoding: utf-8
sys.getfilesystemencoding: iso8859-1
locale.getpreferredencoding: ISO8859-1
sys.platform: aix
sys.implementation:
  name: cpython
Compatible tags: 22
  cp38-cp38-aix_5307_0747_64
  cp38-abi3-aix_5307_0747_64
  cp38-none-aix_5307_0747_64
  cp37-abi3-aix_5307_0747_64
  cp36-abi3-aix_5307_0747_64
  cp35-abi3-aix_5307_0747_64
  cp34-abi3-aix_5307_0747_64
  cp33-abi3-aix_5307_0747_64
  cp32-abi3-aix_5307_0747_64
  py3-none-aix_5307_0747_64
Or something like that - as what you see depends on what your system is currently at!

[Michael]

And - nltk will load in this new Python package:

Code: [Select]

(py38) root@x065:[/data/prj/python/virtual]python
Python 3.8.2 (default, Mar 31 2020, 16:07:53) [C] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import nltk
>>>

[hammertime]

Many thanks @Michael for your speedy responses.

Some very quick testing from me and all looking good.

Code: [Select]

[ts01]:root:/root # python3
Python 3.8.2 (default, Mar 31 2020, 16:07:53) [C] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import nltk
>>> import twilio
>>> import phonetics
>>> import cx_Oracle
>>> import textblob
>>>

[Michael]

Your pep425 tag will probably be different (for cx_oracle) - but you can try using:

Code: [Select]

export OBJECT_MODE=64
pip3 wheel twilio phonetics cx_oracle textblobThis should create or download several wheels, e.g.

Code: [Select]

(py38) root@x065:[/data/prj/python/virtual]ls -l *.whl
-rw-r--r--   1 root     felt          18928 Apr 01 08:14 PyJWT-1.7.1-py2.py3-none-any.whl
-rw-r--r--   1 root     felt         156030 Apr 01 08:14 certifi-2019.11.28-py2.py3-none-any.whl
-rw-r--r--   1 root     felt         133356 Apr 01 08:14 chardet-3.0.4-py2.py3-none-any.whl
-rw-r--r--   1 root     felt         239828 Apr 01 08:17 cx_Oracle-7.3.0-cp38-cp38-aix_5307_0747_64.whl
-rw-r--r--   1 root     felt          58388 Apr 01 08:14 idna-2.9-py2.py3-none-any.whl
-rw-r--r--   1 root     felt        1449905 Mar 31 17:20 nltk-3.4.5-py3-none-any.whl
-rw-r--r--   1 root     felt           8712 Apr 01 08:15 phonetics-1.0.5-py2.py3-none-any.whl
-rw-r--r--   1 root     felt         509158 Apr 01 08:14 pytz-2019.3-py2.py3-none-any.whl
-rw-r--r--   1 root     felt          58406 Apr 01 08:14 requests-2.23.0-py2.py3-none-any.whl
-rw-r--r--   1 root     felt          10938 Mar 31 17:20 six-1.14.0-py2.py3-none-any.whl
-rw-r--r--   1 root     felt         636507 Apr 01 08:14 textblob-0.15.3-py2.py3-none-any.whl
-rw-r--r--   1 root     felt        1091094 Apr 01 08:15 twilio-6.37.0-py2.py3-none-any.whl
-rw-r--r--   1 root     felt         125642 Apr 01 08:14 urllib3-1.25.8-py2.py3-none-any.whlNow - if you look carefully, you will see my prompt has (py38) in it's name. That is because BEFORE I created these wheels I installed virtualenv and created a virtualenv.
So, as root - install virtualenv, and as a regular user (e.g., michael) create a virtualenv (e.g., py38) using the command

Code: [Select]

$ virtualenv $HOME/py38; . $HOME/py38/bin/activateAnd then - install these wheels - NOT as root!! - great for testing!

Code: [Select]

(py38) $ pip install *.whlCurious about your mileage!

[hammertime]

Thanks @Michael .  I have tested the commands you listed and all work perfectly.  For information here are the wheel files that were created -

-rw-r--r--    1 root     system       156030 01 Apr 22:03 certifi-2019.11.28-py2.py3-none-any.whl
-rw-r--r--    1 root     system       133356 01 Apr 22:03 chardet-3.0.4-py2.py3-none-any.whl
-rw-r--r--    1 root     system       255894 01 Apr 22:22 cx_Oracle-7.3.0-cp38-cp38-aix_7105_1913_64.whl
-rw-r--r--    1 root     system        58388 01 Apr 22:03 idna-2.9-py2.py3-none-any.whl
-rw-r--r--    1 root     system      1449902 01 Apr 22:22 nltk-3.4.5-py3-none-any.whl
-rw-r--r--    1 root     system         8712 01 Apr 22:22 phonetics-1.0.5-py2.py3-none-any.whl
-rw-r--r--    1 root     system        18928 01 Apr 22:03 PyJWT-1.7.1-py2.py3-none-any.whl
-rw-r--r--    1 root     system       509158 01 Apr 22:03 pytz-2019.3-py2.py3-none-any.whl
-rw-r--r--    1 root     system        58406 01 Apr 22:03 requests-2.23.0-py2.py3-none-any.whl
-rw-r--r--    1 root     system        10938 01 Apr 22:03 six-1.14.0-py2.py3-none-any.whl
-rw-r--r--    1 root     system       636507 01 Apr 22:03 textblob-0.15.3-py2.py3-none-any.whl
-rw-r--r--    1 root     system      1105877 01 Apr 22:22 twilio-6.38.0-py2.py3-none-any.whl
-rw-r--r--    1 root     system       125642 01 Apr 22:03 urllib3-1.25.8-py2.py3-none-any.whl

I did then copy the cx_Oracle whl file to another server with a slight different patch level so had to rename the _1913_ part of the filename.  So thanks again as I picked this up from reading your other threads.

As to mileage...  at the moment I only have one developer asking to use Python on AIX.  So hard to say.  It is good to see what can be done so please keep up your excellent work!

[Michael]

Thanks for the update. And also the verification that renaming the files works.
I need to get that to be automated, but pypa is very slow to incorporate the changes - even after the base was added to to Python3.9.
Anyway, I'll start with a program to "rename" files that would be binary compatible .Three checks of running system compared with "package" system:

• both are 64 or 32 bit
• running system builddate >= than the filename
• running system AIX-TL >= than AIX-TL in the filename.

[Michael]

Here is a sample script - does not DO anything. You could pipe the output into (k)sh to automate renaming wheels made on other (older) systems so that you can load them on the running system.

Code: [Select]

#!/usr/bin/env python3
"""
Examine arguments for aix pep425 tags
and propose ln (link) commands to give a new name
to an existing wheel for AIX.

Copyright 2020 by AIXTOOLS.NET, Michael Felt
"""

import sys, glob, sysconfig

# save runttime-tag and reverse a copy for comparision
rt_tag = sysconfig.get_platform().split("-")
_tag = list(rt_tag)
_tag.reverse()

# loop through all arguments and look for a AIX tags
for idx in range(1,len(sys.argv)):
  for arg in glob.glob(sys.argv[idx]):
    wheel=arg.split("-")
    if not wheel[-1].startswith("aix_"):
        continue
    # have a potential tag, reverse tag for comparision
    file_tag=wheel[-1].split(".")[0].split("_")
    file_tag.reverse()

    # if the tags are equal, or they are not 32-bit|64-bit - continue
    if (_tag == file_tag) or _tag[0] not in ["32", "64"]:
      continue

    # if the running tag is greater AND same bit-size link names
    elif (_tag > file_tag) and _tag[0] == file_tag[0]:
      wheel[-1] = "_".join(rt_tag) + ".whl"
      print("ln {} {}".format(arg, "-".join(wheel)))