R for AIX

[Michael]

R is a statistical processing language - that has been around for a long while.

The support for AIX port/packaging seems to have been intermittent - looking at the comments in files. There has been work done during AIX 4.2, or shortly thereafter - based on some of the comments I have read.

Some of the choices made THEN are not working as well as hoped TODAY - and I am working on finding "a better way".

If you are working on R on AIX, or would like to be - your comments here will help shape focus on where to place effort.

Right now my first goal is simple: identify issues and find a resolution that leads to a successful build in both 32-bit and 64-bit modes on POWER5 and later processors.

Michael

[Michael]

Thanks to some help from "r-project" one big show-stopper has been solved.

Right now I am having trouble understanding the root cause of many linker messages - both unresolved and duplicate symbols when using gcc.

Since I am making headway - and it has been enriching my knowledge of how applications work - I shall make more frequent updates re: my progress.

The current status is that 32-bit builds are successful, but 64-bit attempts (when using gcc that I built from sources) does not. I am going to try and build using the IBM compilers - to search for similarities and differences.

[Michael]

I have installed xlc v11 and xlfortran V15.1.2. And I have copied a test libiconv.a to /opt/lib

Setting only the environment variable OBJECT_MODE=64 I ran configure with the following arguments:

/data/prj/cran/${RELEASE}/configure --disable-lto --prefix=/opt \
        --enable-R-shlib \
        --with-cairo=no --with-libpng=no --with-jpeglib=no --with-libtiff=no \
        --with-readline=no --with-x=no

This failed to complete with addig the following:

# ln -s /opt /usr/local - as it seems configure is ignoring the --prefix:/opt argument

After that configure finished with:
R is now configured for powerpc-ibm-aix7.1.3.0

  Source directory:          /data/prj/cran/R-3.2.3
  Installation directory:    /opt

  C compiler:                cc -qlanglvl=extc89 -qlanglvl=extc99  -g
  Fortran 77 compiler:       f95  -g

  C++ compiler:              xlC_r  -g
  C++ 11 compiler:             
  Fortran 90/95 compiler:    xlf95 -g
  Obj-C compiler:            cc

  Interfaces supported:     
  External libraries:       
  Additional capabilities:   NLS
  Options enabled:           shared R library, R profiling

  Capabilities skipped:      PNG, JPEG, TIFF, cairo, ICU
  Options not enabled:       shared BLAS, memory profiling

  Recommended packages:      yes

configure: WARNING: f95 and cc -qlanglvl=extc89 -qlanglvl=extc99 disagree on double complex
configure: WARNING: you cannot build info or HTML versions of the R manuals
configure: WARNING: you cannot build PDF versions of the R manuals
configure: WARNING: you cannot build PDF versions of vignettes and help pages
configure: WARNING: I could not determine a browser
configure: WARNING: I could not determine a PDF viewer

I do not expect make to finish. I am running it in this way to learn what additional arguments are needed for CFLAGS, FFLAGS and/or LDFLAGS.

[Michael]

That ended with syntax errors from the f95 compiler - so obviously some changes are needed.

Again, from r-project some suggestions:

export OBJECT_MODE=64
export CC=xlc_r
export CXX=xlc++_r
export F77=xlf_r
export FC=xlf95_r

/data/prj/cran/${RELEASE}/configure --disable-lto --prefix=/opt \
        --disable-R-shlib --enable-R-static-lib \
        --with-cairo=no --with-libpng=no --with-jpeglib=no --with-libtiff=no \
        --with-readline=no --with-x=no


R is now configured for powerpc-ibm-aix7.1.3.0

  Source directory:          /data/prj/cran/R-3.2.3
  Installation directory:    /opt

  C compiler:                xlc_r  -g
  Fortran 77 compiler:       xlf_r  -g

  C++ compiler:              xlc++_r  -g
  C++ 11 compiler:             
  Fortran 90/95 compiler:    xlf95_r -g
  Obj-C compiler:            cc

  Interfaces supported:     
  External libraries:       
  Additional capabilities:   NLS
  Options enabled:           shared R library, R profiling

  Capabilities skipped:      PNG, JPEG, TIFF, cairo, ICU
  Options not enabled:       shared BLAS, memory profiling

  Recommended packages:      yes

configure: WARNING: you cannot build info or HTML versions of the R manuals
configure: WARNING: you cannot build PDF versions of the R manuals
configure: WARNING: you cannot build PDF versions of vignettes and help pages
configure: WARNING: I could not determine a browser
configure: WARNING: I could not determine a PDF viewer

[Michael]

Results to date:

A. It looks like I am going to need a newer compiler for C - xlc/xlC V11 apparently does not understand this code:

"/data/prj/cran/R-3.2.3/src/main/memory.c", line 2149.31: 1506-046 (S) Syntax error.

I will have to check if R-devel has different code before asking for assistence.

 +2139  #ifdef HAVE_STDALIGN_H
 +2140  # include <stdalign.h>
 +2141  #endif
 +2142
 +2143  #include <stdint.h>
 +2144
 +2145  long double *R_allocLD(size_t nelem)
 +2146  {
 +2147  #if __alignof_is_defined
 +2148      // This is C11: picky compilers may warn.
 +2149      size_t ld_align = alignof(long double);
 +2150  #elif __GNUC__
 +2151      // This is C99, but do not rely on it.
 +2152      size_t ld_align = offsetof(struct { char __a; long double __b; }, __b);
 +2153  #else
 +2154      size_t ld_align = 0x0F; // value of x86_64, known others are 4 or 8
 +2155  #endif
 +2156      if (ld_align > {
 +2157          uintptr_t tmp = (uintptr_t) R_alloc(nelem + 1, sizeof(long double));
 +2158          tmp = (tmp + ld_align - 1) & ~ld_align;
 +2159          return (long double *) tmp;
 +2160      } else {
 +2161          return (long double *) R_alloc(nelem, sizeof(long double));
 +2162      }
 +2163  }

B. There is a very big difference in the way libraries are made when gcc/gfortran are not used.

When gcc is being used "everything" is being turned into a shared library. The only library I seem to be able
to affect via configure is libR.so (yes/no). The snip here shows that all members of the .a archives are
"non-shared" objects rather that a combined group of .o files into a single .so shared object.

So, it is not surprising that there are no duplicate symbols.

cran@x068:[/home/cran/64/R]dump -H src/*/*.a | head

src/appl/libappl.a[integrate.o]:

Loader section is not available

src/appl/libappl.a[interv.o]:

Loader section is not available

src/appl/libappl.a[maxcol.o]:

[Michael]

+2147  #if __alignof_is_defined
 +2148      // This is C11: picky compilers may warn.
 +2149      size_t ld_align = alignof(long double);
 +2150  #elif __GNUC__

I have been able to patch with:

Code: [Select]

+2147  #if __alignof_is_defined
 +2148      // This is C11: picky compilers may warn.
            long double ldbl = 0.0;
 +2149      size_t ld_align = alignof(ldbl);
 +2150  #elif __GNUC__
the IBM xlc compiler supports a C99 extension __alignof__(), but not for so-called non-standard type specifications. Using a variable set to that type works in that place. C11 syntax is (I expect) more recent than my xlc v11 compiler (xlc v13 is the current version - mine was new about 6 years ago).

The other point is that the configure script does many things differently, i.e., depending on the compilers used (GNU or IBM) - and is an "on going study" - and slow-going at that.

More to come!

[Michael]

It continues to be quite frustrating - questions that are answered slowly -

What follows is the last work, for now - on the 'recent' releases.

FYI: workaround #1 would be to not build the recommended packages right away - as the core is building ok.

Note also - what follows is more about the AIX 32-bit memory models than about R - imho.

++++++++++++++++
default memory model:
256MB data for data, bss and stack (0x20000000 - 0x2fffffff)

cran@x068:[/home/cran/32/R-3.2.3]dump -o bin/exec/R

bin/exec/R:

                        ***Object Module Header***
# Sections      Symbol Ptr      # Symbols       Opt Hdr Len     Flags
         7      0x0072542c         112376                72     0x1002
Timestamp = 1452258883
Magic = 0x1df

                        ***Optional Header***
Tsize       Dsize       Bsize       Tstart      Dstart
0x003ca77f  0x0003e7d9  0x0009f01c  0x100001a0  0x2000091f

SNloader    SNentry     SNtext      SNtoc       SNdata
0x0004      0x0002      0x0001      0x0002      0x0002

TXTalign    DATAalign   TOC         vstamp      entry
0x0007      0x0003      0x2003da88  0x0001      0x20037588

maxSTACK    maxDATA     SNbss       magic       modtype
0x00000000  0x00000000  0x0003      0x010b        1L

** help
*** installing help indices
** building package indices
Loading required package: Matrix
Error: C stack usage  589826336 is too close to the limit

 *** caught segfault ***
address e0dba1b4, cause 'memory not mapped'

Note: what is happening here - the 0xe0000000 is 'reserved' for
0xe0000000 to 0xefffffff    Available for use by shared memory or mmap services.

Traceback:
 1: tryCatchOne(expr, names, parentenv, handlers[[1L]])
 2: tryCatchList(expr, classes, parentenv, handlers)
 3: tryCatch(expr, error = function(e) {    call <- conditionCall(e)    if (!is.null(call)) {        if (identical(call[[1L]], quote(doTryCatch)))             call <- sys.call(-4L)        dcall <- deparse(call)[1L]        prefix <- paste("Error in", dcall, ": ")        LONG <- 75L        msg <- conditionMessage(e)        sm <- strsplit(msg, "\n")[[1L]]        w <- 14L + nchar(dcall, type = "w") + nchar(sm[1L], type = "w")        if (is.na(w))             w <- 14L + nchar(dcall, type = "b") + nchar(sm[1L],                 type = "b")        if (w > LONG)             prefix <- paste0(prefix, "\n  ")    }    else prefix <- "Error : "    msg <- paste0(prefix, conditionMessage(e), "\n")    .Internal(seterrmessage(msg[1L]))    if (!silent && identical(getOption("show.error.messages"),         TRUE)) {        cat(msg, file = stderr())        .Internal(printDeferredWarnings())    }    invisible(structure(msg, class = "try-error", condition = e))})
 4: try(.install_package_indices(".", instdir))
 5: do_install_source(pkg_name, instdir, pkg, desc)
 6: do_install(pkg)
 7: tools:::.install_packages()
aborting ...

Modified maxdata (to 0xd000000 as maxdata, plus /dsa aka "Very Large Data Model"

To allow a program to use the very large address-space model, you must specify a maxdata value and the dynamic segment allocation (dsa) property. Use either the ld command or the LDR_CNTRL environment variable to specify a maxdata value and the DSA option.

If a maxdata value is specified, the very large address-space model follows the large-address space model
in that a program's data is read into memory starting with segment 3, and occupies as many segments as needed.
The remaining data segments, however, are not reserved for the data area at execution time, but are obtained dynamically.
Until a segment is needed for a program's data area, it can be used by the shmat or mmap subroutines.
With the very large address-space model, a program can a maximum of 13 segments or 3.25 GB of data.
Of these 13 segments, 12 segments or 3 GB, are available for shmat and mmap subroutine purposes.

But are allocated from high to low addresses, rather than low to high


cran@x068:[/home/cran/32/R]dump -o bin/exec/R

bin/exec/R:

                        ***Object Module Header***
# Sections      Symbol Ptr      # Symbols       Opt Hdr Len     Flags
         7      0x0078cff0         125995                72     0x1042
Timestamp = 1452028679
Magic = 0x1df

                        ***Optional Header***
Tsize       Dsize       Bsize       Tstart      Dstart
0x003ff497  0x0003ed95  0x0009f048  0x100001a0  0x30000637

SNloader    SNentry     SNtext      SNtoc       SNdata
0x0004      0x0002      0x0001      0x0002      0x0002

TXTalign    DATAalign   TOC         vstamp      entry
0x0007      0x0003      0x3003db78  0x0001      0x30037850

maxSTACK    maxDATA     SNbss       magic       modtype
0x00000000  0xd0000000  0x0003      0x010b        1L


** building package indices
Loading required package: Matrix
Error: C stack usage  1944285993 is too close to the limit

 *** caught segfault ***
address c4866d8c, cause 'memory not mapped'

Is this a problem with mmap calls (as 0xc4866d8c is < 0xd0000000 - and 0xd0000000 is the top of the addressable memory range)

Traceback:
 1: simpleError(msg, call)
 2: tryCatchOne(expr, names, parentenv, handlers[[1L]])
 3: tryCatchList(expr, classes, parentenv, handlers)
 4: tryCatch(expr, error = function(e) {    call <- conditionCall(e)    if (!is.null(call)) {        if (identical(call[[1L]], quote(doTryCatch)))             call <- sys.call(-4L)        dcall <- deparse(call)[1L]        prefix <- paste("Error in", dcall, ": ")        LONG <- 75L        msg <- conditionMessage(e)        sm <- strsplit(msg, "\n")[[1L]]        w <- 14L + nchar(dcall, type = "w") + nchar(sm[1L], type = "w")        if (is.na(w))             w <- 14L + nchar(dcall, type = "b") + nchar(sm[1L],                 type = "b")        if (w > LONG)             prefix <- paste0(prefix, "\n  ")    }    else prefix <- "Error : "    msg <- paste0(prefix, conditionMessage(e), "\n")    .Internal(seterrmessage(msg[1L]))    if (!silent && identical(getOption("show.error.messages"),         TRUE)) {        cat(msg, file = stderr())        .Internal(printDeferredWarnings())    }    invisible(structure(msg, class = "try-error", condition = e))})
 5: try(.install_package_indices(".", instdir))
 6: do_install_source(pkg_name, instdir, pkg, desc)
 7: do_install(pkg)
 8: tools:::.install_packages()
aborting ...
/home/cran/32/R/bin/INSTALL[34]: 14483552 Segmentation fault(coredump)
make: 1254-004 The error code from the last command is 1.


One final test - with Large Memory, rather than Very Large (i.e., no /dsa) - results are identical to Very Large Memory Model.

** help
*** installing help indices
** building package indices
Loading required package: Matrix
Error: C stack usage  1944285993 is too close to the limit

 *** caught segfault ***
address c4866a8c, cause 'memory not mapped'

Traceback:
 1: simpleError(msg, call)
 2: tryCatchOne(expr, names, parentenv, handlers[[1L]])
 3: tryCatchList(expr, classes, parentenv, handlers)
 4: tryCatch(expr, error = function(e) {    call <- conditionCall(e)    if (!is.null(call)) {        if (identical(call[[1L]], quote(doTryCatch)))             call <- sys.call(-4L)        dcall <- deparse(call)[1L]        prefix <- paste("Error in", dcall, ": ")        LONG <- 75L        msg <- conditionMessage(e)        sm <- strsplit(msg, "\n")[[1L]]        w <- 14L + nchar(dcall, type = "w") + nchar(sm[1L], type = "w")        if (is.na(w))             w <- 14L + nchar(dcall, type = "b") + nchar(sm[1L],                 type = "b")        if (w > LONG)             prefix <- paste0(prefix, "\n  ")    }    else prefix <- "Error : "    msg <- paste0(prefix, conditionMessage(e), "\n")    .Internal(seterrmessage(msg[1L]))    if (!silent && identical(getOption("show.error.messages"),         TRUE)) {        cat(msg, file = stderr())        .Internal(printDeferredWarnings())    }    invisible(structure(msg, class = "try-error", condition = e))})
 5: try(.install_package_indices(".", instdir))
 6: do_install_source(pkg_name, instdir, pkg, desc)
 7: do_install(pkg)
 8: tools:::.install_packages()
aborting ...
/home/cran/32/R/bin/INSTALL[34]: 15269932 Segmentation fault(coredump)
make: 1254-004 The error code from the last command is 1.


Stop.
make: 1254-004 The error code from the last command is 2.


Stop.
make: 1254-004 The error code from the last command is 2.


Stop.
cran@x068:[/home/cran/32/R]dump -o bin/exec/R

bin/exec/R:

                        ***Object Module Header***
# Sections      Symbol Ptr      # Symbols       Opt Hdr Len     Flags
         7      0x0072542c         112376                72     0x1002
Timestamp = 1452266630
Magic = 0x1df

                        ***Optional Header***
Tsize       Dsize       Bsize       Tstart      Dstart
0x003ca77f  0x0003e7d9  0x0009f01c  0x100001a0  0x3000091f

SNloader    SNentry     SNtext      SNtoc       SNdata
0x0004      0x0002      0x0001      0x0002      0x0002

TXTalign    DATAalign   TOC         vstamp      entry
0x0007      0x0003      0x3003da88  0x0001      0x30037588

maxSTACK    maxDATA     SNbss       magic       modtype
0x00000000  0x80000000  0x0003      0x010b        1L
cran@x068:[/home/cran/32/R]