OpenSSH Issue

[Thiru]

Good day All,

My first post, and a very strange one.

I have installed 4 new LPARs on a p570 box.
All having similar resources, OS and apps.

- AIX 5300-10-01-0921
- OpenSSH_5.0p1, OpenSSL 0.9.8k-fips 25 Mar 2009

I am having 2 strange problems, all pertaining to SSH.
All problems seem to only be between LPARs 1; 2 & 3.
LPAR 4 seems fine.
LPAR 1 & 4 are DB2 servers, and LPAR's 2 & 3 are WAS Servers.

1.

Specifying a BANNER in /etc/ssh/sshd_config causes the login prompt to hang when trying to ssh between the LPARs.
Removing it also removes this problem.

2.

SCP between LPARs don't work, it just stalls.
The first file is created on the target server with a size of 0, but you have to eventually cancel.

[u3tm3] /tmp/ssh/tn # scp u3tm:/tmp/ssh/* .
root@u3tm's password:
gskjs.rte                                       0%    0     0.0KB/s - stalled -

Only works fine if they are copying between LPAR 4

I have uninstalled and re-installed SSL & SSH, but nothing works.
I have no idea why this problem is not occuring on LPAR 4.

[Thiru]

There is also a 3rd issue.

If I ssh to LPAR1, then to LPAR2, and try to vi a file or open smitty . . . . it hangs, and eventually timesout and closes connection back to LPAR1.

[Thiru]

OK, I figured out the problem 

I had configured Etherchannel on the first 3 LPARs.
I did not configure it on LPAR 4 as yet, so that got me thinking it was the Etherchannel.
I configured them with Standard mode, but I changed it to use JUMBO FRAMES.
Once I changed JUMBO FRAMES to NO, problem solved.

Since JUMBO FRAMES changes the MTU from 1500 to 9000, I guess the switch then needs to be configured to allow this.

I hope someone else has also been able to learn from this.

 

[Michael]

You have certainly given us all something to remember. Out of curiosity - where there any entries in the errlog about MTU mismatches, and/or netstat -v (or entstat -d entX).

Thanks for the full report!

[Thiru]

I see no errors in the errorlog, and did not check netstat before i corrected the problem.

But I noticed, that athough I switched Jumbo Frames to NO,if I do a entstat, it says Jumbo Frames Enabled?

EtherChannel Attributes:

EtherChannel / Link Aggregation: ent3
Status: Available
Attributes:
      adapter_names   ent0           EtherChannel Adapters
      alt_addr        0x000000000000 Alternate EtherChannel Address
      auto_recovery   yes            Enable automatic recovery after failover
      backup_adapter  ent1           Adapter used when whole channel fails
      hash_mode       default        Determines how outgoing adapter is chosen
      mode            standard       EtherChannel mode of operation
      netaddr         0              Address to ping
      noloss_failover yes            Enable lossless failover after ping failure
      num_retries     3              Times to retry ping before failing
      retry_time      1              Wait time (in seconds) between pings
      use_alt_addr    no             Enable Alternate EtherChannel Address
      use_jumbo_frame no             Enable Gigabit Ethernet Jumbo Frames

entstat output:

-------------------------------------------------------------
ETHERNET STATISTICS (ent3) :
Device Type: EtherChannel
Hardware Address: 00:14:5e:47:a3:10
Elapsed Time: 0 days 20 hours 10 minutes 25 seconds

Transmit Statistics:                          Receive Statistics:
--------------------                          -------------------
Packets: 1932476                              Packets: 2166445
Bytes: 144108349                              Bytes: 449085723
Interrupts: 0                                 Interrupts: 1518927
Transmit Errors: 0                            Receive Errors: 0
Packets Dropped: 0                            Packets Dropped: 0
                                              Bad Packets: 0
Max Packets on S/W Transmit Queue: 10
S/W Transmit Queue Overflow: 0
Current S/W+H/W Transmit Queue Length: 2

Elapsed Time: 0 days 0 hours 0 minutes 0 seconds
Broadcast Packets: 2464                       Broadcast Packets: 128669
Multicast Packets: 581                        Multicast Packets: 1162
No Carrier Sense: 0                           CRC Errors: 0
DMA Underrun: 0                               DMA Overrun: 0
Lost CTS Errors: 0                            Alignment Errors: 0
Max Collision Errors: 0                       No Resource Errors: 0
Late Collision Errors: 0                      Receive Collision Errors: 0
Deferred: 0                                   Packet Too Short Errors: 0
SQE Test: 0                                   Packet Too Long Errors: 0
Timeout Errors: 0                             Packets Discarded by Adapter: 0
Single Collision Count: 0                     Receiver Start Count: 0
Multiple Collision Count: 0
Current HW Transmit Queue Length: 2

General Statistics:
-------------------
No mbuf Errors: 0
Adapter Reset Count: 0
Adapter Data Rate: 2000
Driver Flags: Up Broadcast Running
        Simplex 64BitSupport ChecksumOffload
        PrivateSegment LargeSend DataRateSet

=============================================================
=============================================================

Statistics for every adapter in the EtherChannel:
-------------------------------------------------

Number of adapters: 2
Active channel: primary channel
Operating mode: Network interface backup mode

-------------------------------------------------------------

ETHERNET STATISTICS (ent0) :
Device Type: 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902)
Hardware Address: 00:14:5e:47:a3:10

Transmit Statistics:                          Receive Statistics:
--------------------                          -------------------
Packets: 1932512                              Packets: 2100051
Bytes: 144113833                              Bytes: 442340949
Interrupts: 0                                 Interrupts: 1453696
Transmit Errors: 0                            Receive Errors: 0
Packets Dropped: 0                            Packets Dropped: 0
                                              Bad Packets: 0
Max Packets on S/W Transmit Queue: 9
S/W Transmit Queue Overflow: 0
Current S/W+H/W Transmit Queue Length: 1

Broadcast Packets: 2464                       Broadcast Packets: 63103
Multicast Packets: 581                        Multicast Packets: 581
No Carrier Sense: 0                           CRC Errors: 0
DMA Underrun: 0                               DMA Overrun: 0
Lost CTS Errors: 0                            Alignment Errors: 0
Max Collision Errors: 0                       No Resource Errors: 0
Late Collision Errors: 0                      Receive Collision Errors: 0
Deferred: 0                                   Packet Too Short Errors: 0
SQE Test: 0                                   Packet Too Long Errors: 0
Timeout Errors: 0                             Packets Discarded by Adapter: 0
Single Collision Count: 0                     Receiver Start Count: 0
Multiple Collision Count: 0
Current HW Transmit Queue Length: 1

General Statistics:
-------------------
No mbuf Errors: 0
Adapter Reset Count: 1
Adapter Data Rate: 2000
Driver Flags: Up Broadcast Running
        Simplex 64BitSupport ChecksumOffload
        PrivateSegment LargeSend DataRateSet

2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902) Specific Statistics:
------------------------------------------------------------------------
Link Status : Up
Media Speed Selected: Auto negotiation
Media Speed Running: 1000 Mbps Full Duplex
PCI Mode: PCI-X (100-133)
PCI Bus Width: 64-bit
Latency Timer: 144
Cache Line Size: 128
Jumbo Frames: Enabled
TCP Segmentation Offload: Enabled
TCP Segmentation Offload Packets Transmitted: 390
TCP Segmentation Offload Packet Errors: 0
Transmit and Receive Flow Control Status: Enabled
XON Flow Control Packets Transmitted: 0
XON Flow Control Packets Received: 0
XOFF Flow Control Packets Transmitted: 0
XOFF Flow Control Packets Received: 0
Transmit and Receive Flow Control Threshold (High): 24576
Transmit and Receive Flow Control Threshold (Low): 16384
Transmit and Receive Storage Allocation (TX/RX): 24/40

-------------------------------------------------------------
Backup adapter - ent1:
======================

ETHERNET STATISTICS (ent1) :
Device Type: 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902)
Hardware Address: 00:14:5e:47:a3:10

Transmit Statistics:                          Receive Statistics:
--------------------                          -------------------
Packets: 0                                    Packets: 66397
Bytes: 0                                      Bytes: 6744954
Interrupts: 0                                 Interrupts: 65233
Transmit Errors: 0                            Receive Errors: 0
Packets Dropped: 0                            Packets Dropped: 0
                                              Bad Packets: 0
Max Packets on S/W Transmit Queue: 1
S/W Transmit Queue Overflow: 0
Current S/W+H/W Transmit Queue Length: 1

Broadcast Packets: 0                          Broadcast Packets: 65566
Multicast Packets: 0                          Multicast Packets: 581
No Carrier Sense: 0                           CRC Errors: 0
DMA Underrun: 0                               DMA Overrun: 0
Lost CTS Errors: 0                            Alignment Errors: 0
Max Collision Errors: 0                       No Resource Errors: 0
Late Collision Errors: 0                      Receive Collision Errors: 0
Deferred: 0                                   Packet Too Short Errors: 0
SQE Test: 0                                   Packet Too Long Errors: 0
Timeout Errors: 0                             Packets Discarded by Adapter: 0
Single Collision Count: 0                     Receiver Start Count: 0
Multiple Collision Count: 0
Current HW Transmit Queue Length: 1

General Statistics:
-------------------
No mbuf Errors: 0
Adapter Reset Count: 1
Adapter Data Rate: 2000
Driver Flags: Up Broadcast Running
        Simplex 64BitSupport ChecksumOffload
        PrivateSegment LargeSend DataRateSet

2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902) Specific Statistics:
------------------------------------------------------------------------
Link Status : Up
Media Speed Selected: Auto negotiation
Media Speed Running: 1000 Mbps Full Duplex
PCI Mode: PCI-X (100-133)
PCI Bus Width: 64-bit
Latency Timer: 144
Cache Line Size: 128
Jumbo Frames: Enabled
TCP Segmentation Offload: Enabled
TCP Segmentation Offload Packets Transmitted: 0
TCP Segmentation Offload Packet Errors: 0
Transmit and Receive Flow Control Status: Enabled
XON Flow Control Packets Transmitted: 0
XON Flow Control Packets Received: 0
XOFF Flow Control Packets Transmitted: 0
XOFF Flow Control Packets Received: 0
Transmit and Receive Flow Control Threshold (High): 24576
Transmit and Receive Flow Control Threshold (Low): 16384
Transmit and Receive Storage Allocation (TX/RX): 24/40

[Michael]

OK. First the easy part:
Jumbo Frame enabled on "lower level interfaces" (ent0 and ent1). This means, simplt: Jumbo Frames CAN be used.
ent2 has the attribute: USE Jumbo Frames: No. This means, even though they MIGHT be available do not use them - where YES means, If available then use them.
In short, in this case the underlying hardware supports them but the abstracted interface is instructed to NOT a feature even if it is available.

Extra: On the NIB Etherchannel (I would prefer to just call this a NIB rather than just Etherchannel - because, out of habit, my model of etherchannel is two adapters "actively" supporting an LAN aggregate, or two adapters being used in parallel, or round-robin, by a switch as if they were one adapter, whereas NIB is that the backup adapter (in your case ent1) only gets used when a failure of ent0 (or a regular "etherchannel of two interfaces" is detected.

Go back and give John's article HOWTO: Link Aggregation - As Essential As Disk Mirroring (http://www.rootvg.net/content/view/230/124/)

[Thiru]

Thanks, that is a good article.

I set these up as NIB (Backup) on seperate switches just for redundancy/failover.
I initially set them up with link-aggregation, but the network guy couldn't get it to work properly on the switches, which he picked up as "flapping".