Adapter ID normalization

[PIBE67]

Hello,

We plan to install 5 dual-VIOS systems (10 VIOS in total) on our brand new P795 system.
At this time, we set the adapter ID by using the VIOS order number and Lpar-id (example VIOS No 1 & LPAR-ID 55 ==> VFC Adapter 155).

With the new configuration, this could lead to very high adapter ID's.

Has anybody an experience with such a configuration ?
Which normalization did you define ?

Thanks in advance

Pierre

[Michael]

My recommendation is to make all VIOS similar to other VIOS and all VIOC similar to other VIOC (i.e. partitions that are clients of the VIOS) as similar as possible.

Do not misunderstand me - a VIOS and a VIOC are very different, so they will be different. However, all the VIOS should have the same layout including virtual slot numbers for what they serve. And VIOC should all have the nearly the same output when using the lsconf command.

The concept of using a formula that used the partion ID number to determine a slot number was popular in 2005 and 2006 due to a white paper and a education exercise that used that "method" as a way to explain the different roles of VIOS and VIOC. Unfortunately, too many people thought that this is the best practice.

The basic problem is one you have already implied - the virtual slot numbers get very extreme - and surprisingly - this has adversely affected performance.

If you are using 10 VIOS I would first try to develop a model such that any VIOS looks the same logically to clients. It's functional difference would be it's physical adapters, e.g.. Unless you are configuring all VIOS to be physically equivalent - then you need a model where the logical layout clarifies what function they are performing.

The methaphor I like to use, especially for clients, is putting a PC together on an assembly line. The ethernet is always at the same location, hard drive is always at the same location - so that I can just examine any assembled PC and know just by looking at it if it is a standard PC, or something specialized.

In short, with massive virtualization - and it sounds like you are going that way - standardization is your friend.

If you care to share more about what your goals are, I may be able to be more specific. In any case, what you are proposing is no longer considered "best practice".

[PIBE67]

Hello Michael,

Thank you for your quick answer.

We just defined a new normalization that leads to a highest adapter ID of 4999 (with 999 LPARs in the Power System !!!).

So we are far away from 65536 which is the highest LPAR-ID admitted.

What do you think about that value and the impacts on PowerVM memory consumption ?

Thanks in advance

Best Regards

Pierre

[Michael]

What do I think?  and 

In short, - impressive - and I wish I was visiting you to see how it all fits together. "1000" partitions is not something that happens everyday!

I am guessing that the highest numbers are in the VIOS - as they need the most slots, so I am curious to hear what the highest numbers are for the clients.

And I am wondering if you have created any space for both VSCSI and NPIV connections. There are reasons for having both.
If you have only NPIV right now then you will not be able to use either SSP (shared storage pools) nor PowerSC (Trusted Logging).
If you are planning to only use SSP - I can almost assure you that you will want NPIV for a few partitions.

Assuming this, and knowing you have 5 pairs of VIOS and 900 plus clients planned I would consider sufficient justification for a multiple models of VIOS (so, one model that is supporting only VSCSI, and one support only NPIV for storage plus limited VSCSI for security). I would think about how to have VSCSI in both (equivalent to equal) for supporting security (PowerSC trusted logging is the easiest example) but have the clients still "all the same".

re: VIOS support for PowerSC, another choice could be X (up to 5) additional VIOS, but only for security (no SEA, no NPIV, no SSP/VSCSI storage support - only security).

As far as the performance impact. I cannot honestly say. However, starting with the 730 firmware many changes were made in the firmware - one of them being supporting more than 254 partitions. So I would hope that many of the performance "impacts" of using large slot numbers have been addressed.

And I am sure that there are more "external" people who would love to come take a look.

Please excuse my drooling... 

[Michael]

p.s. I am SO! tempted to tweet this discussion. This is not an everyday configuration!