High Performance Computing Environment RFP

tangibleassistantSoftware and s/w Development

Dec 3, 2013 (3 years and 6 months ago)

340 views


UWYO HPC RFP


January 2012

Page
1






Request for Proposal






High Performance Computing Environment RFP









Procurement Services

University of Wyoming

Laramie, Wyoming 82071











PROPOSAL DUE:
February

29, 2012

2:00 P.M. MST



















UWYO HPC RFP


January 2012

Page
2



SECTION I


UNIVERSITY OVERVIEW



The University of Wyoming is the
only accredited four
-
year school in a state of nearly
one
-
half million people. The University is dedicated to providing quality instruction to
students, conducting valuable research programs, and serving Wyoming through a wide
variety of educational and cultural programs. The University has more than 2,800
benefited faculty and staff dedicated to educating more than 13,000 students.
Approximately

180 undergraduate, graduate, and professional programs of study are
offered with seven colleges: Agriculture, Arts and Sciences, Business, Education,
Engineering, Health Sciences, and Law. The main campus is located in Laramie, but
extends learning opport
unities through a variety of credit and non
-
credit outreach
programs, including the UW/Casper College Center and classes offered through
Wyoming's seven community colleges. Additional information regarding the University

of
Wyoming is available at
http://www.uwyo.edu
.


SECTION II


ADMINISTRATIVE INFORMATION


A.

ISSUING OFFICE:

This Request for Proposal (RFP) is issued for the University of
Wyoming by
Procurement Services
.


B.

INVITATION TO SUBMIT
PROPOSALS:

The University of Wyoming is hereby
contacting prospective Offerors whom the University deems potentially qualified to
meet its needs.


C.

PURPOSE:

This RFP provides prospective Offerors with sufficient information to
enable them to prepare and su
bmit proposals for consideration by the evaluation
team to satisfy the
University's needs for
a High Performance C
omputing
environment.


D.

SCOPE:

This RFP contains the instructions governing the proposal and the
material to be included herein; mandatory req
uirements which must be met to be
eligible for consideration; and other requirements to be met by each proposal.












UWYO HPC RFP


January 2012

Page
3




E.
SCHEDULE OF ACTIVITIES





Activity


Date


Time

(Mountain

Standard

Time)


1.



RFP e
-
mailed to

potential and
interested retail providers
.


01/26
/2012


5
:00 P.M.


2.



Last day to submit questions
concerning the RFP.


02/13/2012


2:00 P.M.


3.


Proposals due

(one electronic copy
and
two

hard
-
copies each in
a separate three
-
ring
binder)
.


02/29/2012


2:00 P.M.


F.

INQUIRIES:

Prospective Offerors may make inquiries concerning this RFP to
obtain clarification of requirements. Send inquiries to the attention of:


Greg Livingston

Assistant P
rocurement Services
Manager

Dept. 3605

1000 E. University Avenue

Laramie, WY 82071
-
3605


E
-
mail: libbie@uwyo.edu

Voice: 307
-
766
-
5718

Fax: 307
-
766
-
2800


Please include the following reference on the face of your envelope or as the subject of
your e
-
mail:



High Performance Computing Environment RFP


Response to any
Offeror's inquiries will be made in writing by
Procurement Services

in a
timely manner to all prospective Offerors. Any oral interpretations or clarifications of this
RFP shall not be relied upon. All changes to this RFP must be in writing to be valid.


G.

MODIFICATION OR WITHDRAWAL OF PROPOSALS:

Proposals may be modified
or withdrawn by the Offeror prior to the established due date and time.



UWYO HPC RFP


January 2012

Page
4



H.

PROPOSAL SUBMISSION:

Responses should be submitted via email in Microsoft
Word format
,
PDF

or other electronic medi
a agreeable by the University.
Electronic versions are mandatory. In addition to the mandatory electronic version,
a signed copy of the RFP must be sent to the below address. The e
-
mail and
two
(2)

original signed copy responses will be accepted up to
,

but no later than
,

2:00
P.M. MST, February 29, 2012
. Late responses will not be accepted. The e
-
mail
responses should be forwarded to the following e
-
mail address.

The proposals will
be opened and
the names of all proposers will be
read
aloud at

that time
.


libbie@uwyo.edu


The proposal package
,

if sent via the U.S. Postal Service, shall be sent to:


University of Wyoming

Procurement Services

1000 E. University,
Dept. 3605

Laramie, WY 820
71


If sent via
any other carrier, please send to:


University of Wyoming

Procurement Services

16 and Gibbon

Laramie, WY 82071





It is the responsibility of the Offeror to ensure that the proposal is received by the
University of Wyoming P
rocurement Services

on or before the proposal submission
deadline. Offerors mailing RFP copies are advised to allow sufficient mail delivery
time to ensure receipt by the time specified.

Proposals must be submitted and sealed in a package showing the following
information

on the envelope.


OFFEROR'S NAME

High Performance Computing Environment RFP


February 2
9
, 201
2
,

2:00 P.M. MST


The University of Wyoming Form of
Proposal

MUST be signed in ink by the Offeror
or an officer of the Offeror who is legally authorized to bind t
he Offeror to the
proposal. Proposals which are determined to be at a variance with this requirement
may not be accepted.


I.

ADDENDUM OR SUPPLEMENT TO REQUEST FOR PROPOSAL:

In the event
that it becomes necessary to revise any part of this RFP, an addendum
will be
mailed (and/or faxed/e
-
mailed if deemed appropriate) to each Offeror who received
or requested the original RFP. It is the responsibility of Offerors, prior to bid date,

UWYO HPC RFP


January 2012

Page
5



to inquire as to addenda issued and to ensure their bid reflects any and all
changes.
Procurement Services

will maintain a register of holders of this RFP.
Any party receiving this RFP in a fashion other
than by receipt from Procurement
Services

should inform P
rocurement Services

of its interest to ensure receipt of
any addenda.


All addenda must be acknowledged on the
Form of Proposal

page
and
copies of the signed addenda must be
included in the offeror’s proposal.


J.

ACCEPTANCE OF RFP TERMS:

A proposal submitted in response to this RFP
shall constitute a binding offer. Acknowledgment of this condition shall be
indicated by the signature of the Offeror or an officer of the Offeror legally
authorized to execute contractual obligations. A subm
ission in response to this
RFP acknowledges acceptance by the Offeror of all terms and conditions including
compensation, as set forth herein. An Offeror shall identify clearly and thoroughly
any variations between its proposal and the University's RFP.
Failure to do so
shall be deemed a waiver of any rights to subsequently modify the terms of
performance.


K.

CONFIDENTIAL/PROPRIETARY INFORMATION:

Any restrictions on the use or
inspection of material contained within the proposal shall be clearly stated in t
he
proposal itself. Written requests for confidentiality shall be submitted by the Offeror
with the proposal. The Offeror must state specifically what elements of the
proposal are to be considered confidential/proprietary. Confidential/proprietary
infor
mation must be readily identified, marked, and separately packaged from the
rest of the proposal. Co
-
mingling of confidential/proprietary and other information
is NOT acceptable. Neither a proposal in its entirety, nor proposal price
information will be
considered confidential/proprietary. Any information that will be
included in any resulting contract cannot be considered confidential.
Procurement
Services

will make a written determination as to the apparent validity of any
request for confidentiality.

The written decision of
Procurement Services

will be
sent to the Offeror as required.


L.

RFP RESPONSE MATERIAL OWNERSHIP:

All material submitted regarding this
RFP becomes the property of the University of Wyoming. Proposals may be
reviewed by any person
after
a contract has been issued
, subject to the terms of
law. The University of Wyoming has the right to use any or all information/material
presented in reply to the RFP, subject to limitations outlined in
P
roprietary/Confidential Information. Disquali
fication of an Offeror, or submission
of an unsuccessful proposal, does not eliminate this right.


M.

SELECTION OF PROPOSAL:

Upon review and approval of the evaluation
committee's recommendation for award,
Procurement Services

will issue a "Notice
of Intent t
o Make an Award" letter to the apparent successful Offeror

or Offerors
.


N.

ACCEPTANCE OF PROPOSAL CONTENT:

The contents of the proposal
(including persons specified to implement the project) of the successful Offeror

will
become contractual obligations if acquisition action ensues.

Failure of the

UWYO HPC RFP


January 2012

Page
6



successful Offeror to accept these obligations in a contract may result in
cancellation of the award and such Offeror may be removed from future
solicitations. The award wil
l be made to that Offeror whose proposal, conforming
to the RFP; will be the most advantageous to the University of Wyoming
.
The
University does not forfeit its sovereign immunity through the execution of the
contract.

Any contract(s) resulting from this

RFP will also incorporate the
requirements of the RFP. In the case of a conflict between the Offeror’s proposal
and the RFP, the RFP shall govern.


O.

STANDARD CONTRACT:

The University of Wyoming reserves the right to
incorporate standard UW contract provis
ions (Special Provisions) and other
provisions noted in this RFP into any contract resulting from this RFP
, such as, but
not limited to the following:

http://www.uwyo.edu/procurement/terms
-
and
-
conditions/index.html
.


P.

RFP CANCELLATION:

The University reserves the right to cancel this Request for
Proposal at any time, without penalty.


Q.

INCURRING COSTS:

The University of Wyoming is not liable for any cost i
ncurred
by Offerors prior to issuance of a legally executed contract. No property interest, of
any nature, shall accrue until a contract is awarded and signed by all concerned
parties.


R.

MINORITY PARTICIPATION:

It is the University's goal to maximize parti
cipation of
minorities in the procurement process. Accordingly, minority enterprises are to be
utilized when possible. By the submission of a proposal the Offeror shall agree to
utilize the maximum amount of minority business firms that the Offeror finds

to be
consistent with the efficient performance of any resulting contract.


S.

E
QUAL EMPLOYMENT OPPORTUNITY/DIVERSITY (Affirmative Action) Program

The University is committed to equal opportunity for all persons in all facets of the
University’s
operation. The University’s policy has been, and will continue to be,
one of nondiscrimination, offering equal opportunity to all employees and applicants
for employment on the basis of their demonstrated ability and competence without
regard to such matt
ers as race, color, religion, sex, national origin, disability, age,
veteran status, sexual orientation or political belief. It shall also be the policy of the
University to utilize an affirmative action approach in the recruiting, hiring and
promotion of

women, people of color and other persons from designated groups
covered by federal statutes, executive orders and implementing regulations in order
to achieve a diverse workforce and student body.


T.

REJECTION OF PROPOSALS:

The University of Wyoming reserv
es the right to
reject any or all proposals and to waive informalities and minor irregularities in
proposals received
, by the specific bid opening time and date,

and to accept any
portion of a proposal or all items proposed if deemed in the best interest
o
f the
University of Wyoming
.


UWYO HPC RFP


January 2012

Page
7




U.

PARENT COMPANY:

If an Offeror is owned or controlled by a parent company, the
parent company name, main office address, and parent company's tax
identification number shall be provided in the proposal.


V.

NEWS RELEASES:

News releases pertaining to this RFP shall NOT be made prior
to execution of the contract without prior written approval by the University.


W.

CERTIFICATION OF INDEPENDENT PRICE DETERMINATION:


1.


By submission of this proposal, each Offeror, and in the case

of a joint
proposal, each party thereto, certifies as to its own organization, that, in
connection with this procurement:


a.

The prices in this proposal have been arrived at independently, without
consultation, communication, or agreement, for the purpose o
f restricting
competition, as to any matter relating to such prices with any other
Offeror or with any competitor;


b.

Unless otherwise required by law, the prices which have been quoted in
this proposal have not been knowingly disclosed by the Offeror and wi
ll
not knowingly be disclosed by the Offeror prior to opening, directly or
indirectly, to any other Offeror or to any competitor; and


c.

No attempt has been made or will be made by the Offeror to induce any
other person or firm to submit or not to submit a p
roposal for the purpose
of restricting competition.


2.


Each person signing the Form of Bid form of this proposal certifies that:


a.

He/She is the person in the Offeror's organization responsible within that
organization for the decision as to the prices bein
g offered herein and that
he has not participated, and will not participate, in any action contrary to
(1 a) through (1 c) above; or


b.

He/She is not the person in the Offeror's

organization responsible within
that organization for the decision as to the prices being offered herein but
that he has been authorized in writing to act as agent for the persons
responsible for such decision in certifying that such persons have not
part
icipated, and will not participate, in any action contrary to (1 a)
through (1 c) above, and as their agent does hereby so certify; and he
has not participated, and will not participate, in any action contrary to (1 a)
through (1 c) above.


3.


A proposal

will not be considered for award where (1 a), (1 c), or (2) above
has been deleted or modified. Where (1 b) above has been deleted or

UWYO HPC RFP


January 2012

Page
8



modified, the proposal will not be considered for award unless the Offeror
furnishes with the proposal a signed statemen
t which sets forth in detail the
circumstances of the disclosure and the head of the agency, or his designee,
determines that such disclosure was not made for the purpose of restricting
competition.


Y.

CONFLICTS OF INTEREST:

The holding of public office or e
mployment is a public
trust. A public officer or employee whose conduct departs from his fiduciary duty is
liable for his or her actions.


Z.

TAXES:

The University of Wyoming, as purchaser, is exempt from all federal
excise taxes and from all
State of Wyomi
ng sales tax
.


AA.

ASSIGNMENT:

Neither party to any resulting contract may assign any portion of
the agreement without the prior written consent of the other party.


BB.

AVAILABILITY OF FUNDS:

Financial obligations of the University payable after the
current fisc
al year are contingent upon funds for that purpose being appropriated,
budgeted, and otherwise made available. In the event funds are not appropriated,
any resulting contract will become null and void, without penalty to the University of
Wyoming.


CC.

BID BO
NDS/SECURITY:

Not required for this proposal.


DD.

CONTRACTOR'S PERFORMANCE BOND:

Not required for this proposal.


EE.

INSURANCE:

$10 million minimum each claim and annual aggregate, covering all
acts, errors, omissions, negligence, infringement of intellectual property (except
patent and trade secret) and network and privacy risks (including coverage for
unauthorized access, failure

of security, breach of privacy perils, wrongful
disclosure of information, as well as notification costs and regulatory defense) in
the performance of services for the University of Wyoming. The policy shall
contain an affirmative coverage grant for cont
ingent bodily injury and property
damage emanating from the failure of the technology services or an error or
omission in the content/information provided. Such insurance shall be maintained
in force at all times during the term of the agreement and for a

period of five (5)
years thereafter for services completed during the terms of the agreement.


FF.

INDEMNIFICATION:

To the extent authorized by law, the contractor shall
indemnify, save and hold harmless the University, its employees and agents,
against any and all claims, damages, liability, and court awards including costs,
expenses, and attorney fees incurred as a
result of any act or omission by the
contractor or its employees, agents, subcontractors, or assignees pursuant to the
terms of the contract resulting from this RFP.



UWYO HPC RFP


January 2012

Page
9



GG.

WYOMING STATE PREFERENCE:

Preference shall be allowed in purchase of
materials, supplies,

equipment, and machinery provided by bona fide Wyoming
residents when such materials, supplies, equipment, machinery or provisions are of
quality equal to those of any other state enforcing or having a differential for “out
-
of
-
state” materials, supplies,
equipment, machinery, or provisions.


Such
preference shall be five percent (5%) and shall apply to materials, supplies,
equipment, machinery or provisions produced, manufactured or grown in Wyoming,
and to materials, supplies, equipment, and machinery sup
plied by a Wyoming
resident.


(UW Regulation 3
-
105, Purchases)


However, preferences shall not be
given in violation of any Federal law, rule, or regulation.


Whenever Federal laws
are applicable, Federal laws shall supersede any State laws.


HH.

EVALUATION CR
ITERIA
:

Proposals will be considered from firms with a
demonstrated history of successfully providing service similar to what we are
requesting. The contract will be awarded to the proposer who most completely
meets the University’s needs. University pers
onnel will evaluate all proposals
submitted in response to this RFP. Proposals will be evaluated using the following
criteria:


1.


Responsiveness of the proposal in stating a clear understanding of the
requirements.

Failure to respond to all items may be

grounds for rejecting the
proposal.

Proposals that successfully address the overall design goals meet
the effective use of the UWRC as a community resource, flexibility in the
incorporation of satellite clusters, and allow for easy and cost
-
effective
ex
pansion will be more favorable reviewed.

2.


Granularity (i.e. the smallest increment of expansion allowed) and cost of
expansion will greatly influence the decision of the HPC system vendor. Priority
will be given to proposals that give the University the m
ost flexible growth over
a
three
-
year

period.

3.


Regarding the HPS system, priority will be given to the Offeror who offers the
best performance per cost and has the most granular and cost effective
capacity growth proposal.


4.


Completeness of a step
-
by
-
ste
p plan for implementing service.


5.


The amount bid to carry out the work will be taken into consideration once it is
determined which firms are qualified
.


6.


Ability to complete the Scope of Work in a timely manner
.


II.

TERMINATION PROVISIONS
:

The contract
may be terminated as follows:

a.

The University and the contractor, by mutual written agreement, may

terminate the contract at any time.


UWYO HPC RFP


January 2012

Page
10



b.

The University, in its sole discretion, may terminate the contract for any
reason upon 90 days written notice to the contr
actor. Such termination
shall not relieve the contractor of any obligations to provide services
during the notification.

c.

The University may terminate the contract, in whole or in part, if the
contractor fails to

perform its obligations under
the contract i
n a timely and
proper manner. The University may, by providing written notice of default
to the contractor, allow the contractor to cure a failure or breach of
contract within a period of ten (10) days (or longer at the University’s
discretion considering

the gravity and nature of the default). Said notice
shall be delivered by Certified Mail, Return Receipt Requested or in
person with proof of delivery. Allowing the contractor time to cure a failure
or breach of contract does not waive the University’s r
ight to immediately
terminate the contract for the same or different contract breach which may
occur at a different time. In case of default of the contractor, the
University may contract the service from other sources and hold the
contractor liable for a
ny excess cost occasioned thereby.

d.

The University may terminate the contract immediately for the following
reasons:

i.

Contractor has made an assignment for the benefit of creditors, has
admitted in writing its inability to pay debts as they mature, has
cease
d operating in the normal course of business, the contractor
has voluntarily or involuntarily entered into bankruptcy proceedings,
or the contractor is otherwise deemed insolvent

ii.

A trustee or receiver of the contractor or of any substantial part of
the con
tractor’s assets has been appointed by a court

iii.

Fraud, misappropriation, embezzlement, malfeasance,
misfeasance, or illegal conduct pertaining to performance under the
contract by the contractor, its employees, officers, directors of
shareholders

iv.

Failure to

uphold any fiduciary obligation to the University



JJ.

SUSTAINABILITY PRACTICES
: The University

of Wyoming strongly encourages
the campus community to adhere to sustainable practices. For product categories
that have ENERGY STAR rated products available, the university will focus its
procurement efforts on products with this rating, consistent with
the needs of the
UW campus community.


KK.

GOVERNMENTAL CLAIMS
:
Any actions or claims against the University under
this Agreement must be in accordance with and are controlled by the Wyoming
Governmental Claims Act, W.S. 1
-
39
-
101 et seq. (1977) as amended.


LL.

INTERPRETATION
:
The Parties hereto agree that (i) the laws of Wyoming shall
govern this Agreement, and (ii) any questions arising hereunder shall be

UWYO HPC RFP


January 2012

Page
11



construed according to such laws, (iii) this Agreement has been negotiated and
executed in the State of Wyo
ming and is enforceable in the courts of Wyoming.


MM.

SOVEREIGN IMMUNITY
: The University does not waive its sovereign immunity
or its governmental immunity by entering into this Agreement and fully retains all
immunities and defenses provided by law with rega
rd to any action based on this
Agreement.





























UWYO HPC RFP


January 2012

Page
12



Section
III. Technical Specifications

Contents


1

Executive Summary

................................
................................
................................
.............

13

2

Overview [Info]

................................
................................
................................
......................

14

3

Abbreviations [Info]

................................
................................
................................
..............

17

4

RFP Terminology [Info]

................................
................................
................................
.......

17

4.1

RFP Conventions

................................
................................
................................
.......................

17

5

General Requirements and Specifications [Req]

................................
............................

18

5.1

Environmental Infrastructure [Info]
................................
................................
...........................

18

5.2

Environmental Requirements [Req]
................................
................................
.........................

19

5.3

Warranty/Maintenance Requirements for HPC and HPS [Req]

................................
..........

19

5.4

Delivery Requirements [Req]

................................
................................
................................
....

20

5.5

Acceptance Testing [Req]

................................
................................
................................
.........

20

5.6

UWRC Reliability, Maintenance, and Support Requirements [Imp]

................................
...

21

5.7

UWRC Software Support [Imp]

................................
................................
................................

22

5.8

UWRC Training [Req]

................................
................................
................................
................

23

6

UWRC Environment

................................
................................
................................
............

24

6.1

UWRC Architecture [Req]

................................
................................
................................
.........

24

6.2

Software

................................
................................
................................
................................
.......

25

6.2.1

Operating System [Imp]

................................
................................
........................

26

6.2.2

Resource Manager and Job Scheduler [Req]

................................
..................

26

6.2.3

Compilers [Imp]

................................
................................
................................
......

26

6.2.4

User Environment Support [Req]

................................
................................
........

27

6.2.5

Cluster Monitoring
[Imp]

................................
................................
.......................

27

6.2.6

Parallel Computing Environment [Imp]

................................
..............................

27

6.2.7

Debuggi
ng Tools, Performance Measurement, Analysis and Tuning Tools
[Imp]

27

6.2.8

BIOS Management Tools [Req]

................................
................................
..........

27

6.3

Technical Requirements for HPC, HPS and Network [Req]

................................
................

27

6.3.1

Network Infrastructure [Req]

................................
................................
................

28

6.3.2

HPS Storage System Requirements [Req]

................................
.......................

32

6.3.3

HPC Condo Resources and Requirements [Info]

................................
............

35

7

Documentation

................................
................................
................................
.....................

42

8

Expandability and the UW Condo Model

................................
................................
.........

42





UWYO HPC RFP


January 2012

Page
13




1

Executive Summary

The University of Wyoming (hereafter referred to as The University) has a diverse and growing
group of scientists with intense computational research. We have over 50 computational
scientists, including faculty in scientific and engineering disciplines sp
anning Geology and
Geophysics, Energy Research, Environmental Research, Computational Fluid Dynamics, and
Bioinformatics.

In order to support this community, the University of Wyoming desires to purchase an enterprise
class research computing cluster and
tiered storage system. These systems will support
general high
-
performance computing research, as well as provide an environment for University
Earth System scientists to hone and test their software prior to use on the NCAR
-
Wyoming
Supercomputing Center
resources.

The High Performance Cluster (HPC) and Storage (HPS) will be deployed using a Condo
-
model. The University will purchase the core equipment and software and will provide central
system management, maintenance and user support. The University
will then grow the
computing and storage facility with individual project, department and researcher contributions
as needs and funding allow.

The initial HPC and HPS configuration and size will depend on available funding, the best
capability and performa
nce per cost. As funding and individual contributions allow, The
University will grow the facility by adding compute nodes and storage. Granularity (i.e. the
smallest increment of expansion allowed) and cost of expansion will greatly influence the
decisi
on of the cluster vendor and initial configuration.

The current plan is to update hardware annually and refresh on a 3 to 4 year cycle. Older
hardware will either be refreshed, retired or remain active in a semi
-
supported mode.

The following are approxima
te current funding plans for the initial purchase and expansion.
These plans may be accelerated should additional funding materialize.

Initial System Funding


$930K

CY 2012




$500K

CY 2013




$660K

CY 2014 +




$350K


Funding may be substantially larger a
s individual researchers and project contribute to the
expansion of the University of Wyoming Research Computing (UWRC) facility.

Because of the diverse nature of the research at the University the HPC will be required to
contain various types of compute n
odes including the following:



Thin (or low memory) nodes



Fat (or high memory) nodes



Multi
-
core (more than 8) Fat nodes



Thin Nodes with one or more GPUs



Fat Nodes with one or more GPUs


UWYO HPC RFP


January 2012

Page
14




Funding numbers listed above are the current commitments. Based on add
itional researcher
contributions the University envisions growing the cluster substantially over the next 3 years.
Priority will be given to proposals that give the University the most flexible growth over this
period. Section 6.3.3 describes the nodes a
nd lists the estimated mix of node types. It is
envisioned that the mix of nodes will change as the University researchers begin to use the
cluster and determine optimal performance for their scientific applications.

Current plans are to grow the storage

facility to a multi
-
Petabyte facility within 3 years. Section
6.3.2 describes the desired storage technology and estimated mix of storage types. The mix of
storage technology may change as the researchers become familiar with the facility and their
perf
ormance requirements.

The HPC and HPS should be interconnected through a robust high
-
speed Infiniband network
and the HPS will be accessible from desktop computers across campus running a variety of
Operating Systems.

This document describes the Universit
y’s vision of an acceptable HPC and HPS condo system.
It should be considered as a set of guidelines for the Offer’s proposed solution. Offerors may
bid different node and storage mixtures and even different architectures than those described in
this RFP

but shall articulate the benefit of their proposed solution to the University.

2

Overview [Info]

The University of Wyoming seeks to build a scalable Research Computing environment
(UWRC) consisting of a High Performance Computing system and a tiered High Performance
Storage system that are interconnected by a robust Infiniband backbone network as desc
ribed
below. This Request for Proposals (RFP) states the instructions for submitting proposals, the
procedure and criteria by which an Offeror may be selected, and the contractual terms that shall
govern the relationship between the selected Offeror(s) and

the University.

The University's cyber
-
infrastructure plan mandates the creation of a robust networking
infrastructure and the building out of the computational and storage capability of UWRC over a
three
-
year period, after which annual University and res
earcher investments will allow for
regular upgrades to the latest HPC, HPS and networking technologies.

The diversity of stakeholders in the UWRC endeavor, their growing set of research interests and
needs require a flexible and expandable solution.

Anot
her driving force for the UWRC is a joint Utah
-
Wyoming, NSF
-
funded project devoted to

acquiring and developing hardware and software cyber
-
infrastructure to support the
development and use of large
-
scale, high
-
resolution computational water resources model
s to
enable comprehensive examination of integrated system behavior through physically
-
based,
data
-
driven simulation. Through its partnership with NCAR, University of Wyoming researchers
in Earth System Sciences will have access to the NCAR
-
Wyoming Superco
mputing Center
(NWSC), a
p
etascale system that will come online in the summer 2012. Thus, the UWRC will
also serve as a resource for UW faculty and collaborators to perform smaller runs for
development, testing, and benchmarking codes before NWSC is used f
or large
-
scale production
runs.



UWYO HPC RFP


January 2012

Page
15



Top
-
level requirements for the UWRC are:



A High Performance Computing system that provides the core community computing
resource;



A tiered High Performance Storage system that acts as a central data file system,
and provid
es data backup and archiving;



A robust network backbone.


Proposals that successfully address the overall design goals outlined below, the effective use of
the UWRC as a community resource, flexibility in the incorporation of satellite clusters, and
allow
for easy and cost
-
effective expansion will be more favorable reviewed.

For the purposes of this RFP and development of the UW Research Computing environment
these components will be treated as separate modules. This modular approach eases
component upgrade
s, provides flexibility in purchasing, and allows the University to integrate
existing resources such as smaller specialized clusters into a unified HPC environment.

Sections 5 and 6 provide descriptions of the individual components. Offerors may bid on a
ny or
all parts including Software, Network Infrastructure, High Performance Storage system, High
Performance Computing system. All Offerors must bid on the appropriate Network
Infrastructure. If an Offeror bids on the just the HPC or the HPS system, the
n selected Offerors
must be willing to work together to complete the installation. If an Offeror bids on the HPC
system, the Offeror should also include information on Section 5.2, to complete their bid.




UWYO HPC RFP


January 2012

Page
16



Figure 1 is a diagram of the envisioned layout of

the HPC environment. Note that this is a
conceptual diagram that does not necessarily represent actual purchased hardware or specific
implementations. The exact configuration will depend on the chosen Offeror's solution.




High Speed Interconnect
(
e
.
g
.
InfiniBand
)
HSI
Campus
Gigabit
Ethernet
Regional and Wide Area
(
GigE
)
NWSC
FRGP
High Performance Storage
(
e
.
g
.
Lustre
,
GPFS
)
General Purpose Storage
And Archive
Master Node
(
s
)
Configuration Management Software
(
CMS
)
Job Scheduler
Fat Nodes
Compute Nodes
Satellite Cluster
(
s
)
Virtualization Lab
Satellite Storage
Network Gateway

Figure 1




UWYO HPC RFP


January 2012

Page
17



3

Abbreviations [Info]

Condo

A shared computing resource to support the scientific computing needs
of the University

GPU

Graphics Processing Unit

HA

High Availability System Configuration, i.e. master/slave

HPC

High Performance Computing

HPS

High Performance Storage

IBA

Infiniband Network Architecture

IPMI

Intelligent Platform Management Interface 2.0

MPI

Message Passing Interface

NCAR

The Nation Center for Atmospheric Research

NWSC

The NCAR Wyoming Supercomputing Center

PXE

Pre
-
boot
Execution Environment

Satellite Cluster

A cluster that currently exists or may exist at the University that is
physically separate from the HPC condo cluster. A satellite cluster may
be interconnected to the HPS via Ethernet or Infiniband. These
cluster
s may be housed externally to the University’s Datacenter.

University

The University of Wyoming

UWRC

The University of Wyoming Research Computing environment

4

RFP Terminology [Info]

4.1

RFP Conventions

Offerors

should review the abbreviations section for the definitions of terms and acronyms used
in the following specifications.

Each of the technical resource sections below are categorized by their relative importance to the
University using the following defini
tions:

Specification Priority

Description

Information [Info]

These items are provided as background or supporting information to
aid the Offerors in determining the most suitable system configuration
in responding to the following priorities.

Required [
Req]

These specifications are the most important to the University.
Offerors are required to meet or exceed these specifications in their
response(s).

Important [Imp]

These specifications are of sufficient importance to the University that
the overall ef
fectiveness of the HPC may suffer if not provided.
Offerors are encouraged to include as many of these features as
possible.

Enhance [Enh]

These specifications enhance the function of the HPC environment or
are highly desirable.


UWYO HPC RFP


January 2012

Page
18



5

General Requirements and

Specifications [Req]

5.1

Environmental Infrastructure [Info]

The IT Data Center serves the academic and administrative computing needs for the University
of Wyoming. This data center provides for 6000 ft
2

of

raised floor computing space with an
initial cooling capacity of 50W/ft
2
.

The data center operates at approximately 7300 feet above sea level and this must be factored
into all electrical and cooling designs.


All equipment numbers provided in this secti
on are for
informational purposes only.

The primary electrical supply is from Rocky Mountain Power’s grid and is backed up by a single
Cummins 1.2MW Diesel Generator. The UPS system uses redundant 9315 Powerware units
with three (3) battery strings on each

UPS.

The UPS units are currently running at
approximately 60% capacity and provide approximately 70
-
90 minutes of stand
-
by power.
Power to the equipment cabinets is provided overhead via redundant Starline Busway Rails.



Primary cooling for the IT Data
Center is handled by the campus chilled water supply (CWS)
with an on
-
site 125
-
ton Trane Chiller for backup.


The computer room air handlers (CRAH’s)
reside in a mechanical gallery adjacent to the data center.


Five (5) 60
-
ton down
-
flow Stultz
units use va
riable speed fans to provide forced air cooling into the raised floor plenum.

The data center equipment cabinets are standardized on Chatsworth (TS1004171) F series
TerraFrame Cabinets (700mm x 975mm x 2121mm). The University wishes to maintain the
look
and aesthetic feel of these cabinets.

Cabinets are arranged in a hot aisle/cold configuration with a raised ceiling open
-
air return
plenum.


Hot air is returned to the mechanical gallery via large elevated grates in the data
center/mechanical gallery wal
l. Currently the data center does not use hot aisle/cold aisle
containment. The design capacity of the data center (with upgrades) is for 100W/ft
2
.

Chilled water supply piping is stubbed out in key locations around the data center perimeter for
two additi
onal 60
-
ton CRAH units.


These CWS taps can also be used for specialized cooling
configurations.

Rear door heat exchangers and enclosed in
-
rack cooling will be considered for
any high density cooling requirements above 5KW per cabinet and less than 20kW p
er
cabinet.


A space of approximately 440 ft
2

has been allocated for the placement of the UWRC
that will service the research computing community.

Proposals should not include cooling densities that exceed 20kw per cabinet. All proposed
equipment should f
it in a standard generic equipment cabinet that remains the choice of the
University of Wyoming.

Any necessary high density cooling and computing environments that will exceed 10KW of heat
load per equipment cabinet shall have the capability to use closed

loop in
-
rack cooling for heat
dissipation or rear door heat exchangers. The preferred solution for in
-
rack cooling will be
compatible with industry leading Offerors such as Rittal or APC. Proprietary configurations will
be considered but information sha
ll be provided on the merits of the solution and all associated
cooling dependencies such as airflow requirements, chilled water loops, air handlers, etc. shall
be included in the solution.


UWYO HPC RFP


January 2012

Page
19



5.2

Environmental Requirements [Req]

Offerors shall provide the follo
wing information in electronic form:



A rack elevation diagram showing the layout of all equipment.



Electrical requirements for each cabinet and individual components including amperage
for each, number and type of outlets required, redundant vs. non
-
redund
ant, single
phase or 3
-
phase power, voltage.



Cooling requirements including total heat load in kW per equipment cabinet, operational
intake temperature, delta T of the heat rise across the equipment.


5.3

Warranty/Maintenance Requirements for HPC and HPS [Req
]

The Offeror shall provide a minimum three (3) year warranty, 5 x 8 next business day on all
hardware and software supplied by the Offeror. Hardware and software provided by the Offeror
for the UWRC must be covered and include maintenance, support, and upg
rades.

The Offer, in order to meet the warranty and maintenance requirements, may propose different
support models that would best serve the University. Models may include maintaining onsite
spares that the University may use to correct minor failures. O
nsite spares should include one
of every critical board, disk, subassembly, or power supply. If the Offer chooses this model it
will be the Offeror's responsibility to maintain the proper spare parts onsite.

Alternatively, the Offeror may provide a sys
tem where the University can perform direct ordering
of failed parts, perform the repair and return the failed part.

The Offer as part of their RFP response shall provide information on the following items:



warranty length and coverage and length of
warranty



A guarantee that each system will run as designed at an elevation of 7300 ft.



Details about any components that are not covered by the warranty.



Three years hardware maintenance on
-
site, 5 x 8 next business day repair of all
hardware.



Option for s
ixty (60) days of Offeror onsite personnel support after acceptance.



Warranty repair procedure beginning with support call initiation.



Maintenance repair procedure beginning with support call initiation.



Contact information for the company who will be perf
orming repair and maintenance for
each component. If some components have different service providers, supply their
contact information as well.

All hardware and software will be maintained by University staff via direct access to the original
equipment ma
nufacturer, i.e. University staff should not have to contact the Offeror for support
on items that are not of the Offeror’s manufacture.

All warranties and licenses for software, hardware shall be transferred to the University at the
time of acceptance of

the system.


UWYO HPC RFP


January 2012

Page
20



5.4

Delivery Requirements [Req]



The Offeror has the following racking options:

o

Hardware will be delivered to the site and racked in University supplied racks,

o

Request the University to provide the racks, rack the equipment and then return
the pre
-
built racks back to the University,

o

The Offeror purchases University approved racks and delivers hardware pre
-
racked.

The Offeror shall indicate which option they prefer.



The Offeror shall supply all power cables, network and Infiniband

cables to connect
equipment.



The Offeror shall deliver all documentation in electronic form detailing the cluster
configuration including MAC address, Node configuration, Part Numbers, and serial
numbers.



On
-
site training of administrators shall be provid
ed by the Offeror. Training should
include familiarization with the hardware and any specialized software.

Schedule



Hardware shall be delivered within 30 calendar days of receipt of purchase order unless
an alternate time frame is agreed upon by the Offero
r and the University.



All hardware and software shall be installed and operational within fifteen (15) working
days of delivery to customer site.



The acceptance test period start date shall begin after the fifteen (15) day installation
period. The syste
ms shall run for twenty (20) consecutive calendar days. Failure of
acceptance testing will restart the clock and shall be complete within ninety (90) days of
the initial start date.



Full payment shall not be made until acceptance testing is complete and a
greed upon by
the Offeror and the University.

Any changes to this schedule shall be agreed upon in writing by the Offeror and the
University.

5.5

Acceptance Testing [Req]

Each system comprising the UWRC HPC and HPS resources shall be subject to Acceptance
Tes
ting. This testing and the criteria for acceptance of each system shall be subject to the
terms and conditions of the awarded contract between The University of Wyoming and the
chosen Offeror.

Activities to be conducted during Acceptance Testing of each s
ystem will include, but not be
limited to, running and or performing the following
:

For the UWRC HPC System:

a.

Subcontractor
-
supplied system diagnostic tests to verify that all components of the system
are functioning within specifications.

b.

The system’s capa
city and capability measured using the UWRC Benchmark Suit
e.


UWYO HPC RFP


January 2012

Page
21



c.

A workload composed of components of the UWRC Benchmark Suite.

d.

Testing of the remote management, LOM and other operational and administrative attributes
of the system, including placing simulated

maintenance and support requests to
demonstrate that service entitlement is in effect.

e.

Specific timed and untimed activities to evaluate particular aspects of the system, including
but not limited to: measuring system shutdown and cold
-
start time, removal

and addition of
Node(s) from/to an operational system, time required to perform a complete backup and
subsequent full bare
-
metal restore of a Node’s Software and configuration.

f.

The University of Wyoming may choose to run one or more applications that are
not in the
UWRC Benchmark Suite but which will be run by users of the system after acceptance. The
University of Wyoming may choose to run independent I/O tests to corroborate the
performance demonstrated.


For the UWRC HPS System:

g.

Contractor
-
supplied
diagnostic tests to verify that all components of the system are
functioning within specifications.

h.

The creation, expansion, contraction and deletion of one or more parallel file

system(s).

i.

Testing of file

system failover capabilities and evaluation of the

total failover time interval
and the impact on I/O performance during failover processing.

j.

The UWRC I/O Benchmark tests run from external system(s).

k.

Testing of the remote management, LOM and other operational and administrative attributes
of the system, i
ncluding placing simulated maintenance and support requests to
demonstrate that service entitlement is in effect.

l.

Specific timed and untimed activities to evaluate particular aspects of the system, including
but not limited to: measuring system shutdown an
d cold
-
start time, removal and addition of
Node(s) from/to an operational system, time required to perform a complete backup and
subsequent full bare
-
metal restore of a node’s software and configuration.

m.

A workload on attached HPC system(s) composed of com
ponents of the UWRC Benchmark
Suite.

n.

The University of Wyoming may choose to run one or more applications that are not in the
UWRC Benchmark Suite but which will be run by users of the system after acceptance.

o.

The University of Wyoming may choose to run in
dependent I/O tests to corroborate the
performance demonstrated.


Instructions for accessing the

UWRC Benchmark suite may be obtained by contacting the
University of Wyoming Procurement Services.

5.6

UWRC Reliability, Maintenance, and Support Requirements [Imp
]

Each system comprising the UWRC HPC resource shall exhibit availability in excess of 98.0%
and an MTBSF exceeding 384 hours (16 Days).

Each system comprising the UWRC HPS resource shall exhibit availability in excess of 99.0%
and an MTBSF exceeding 576
hours (24 Days).

The following table provides target hardware maintenance and Software support for the UWRC
HPC Production resource. In the table,


UWYO HPC RFP


January 2012

Page
22



1.

“critical components” and “critical system problems” are those which, upon failure or
occurrence, cause the
system to either be unusable or sufficiently degraded as to be unable
to run 85% of its normal workload or cause the normal workload to exhibit less than 90% of
normal performance, and

2.

“non
-
critical” components include, but are not limited to individual ba
tch and individual login
Nodes of the HPC resource, portions of Node memory (provided the node continues to
operate), RAID disks, and redundant external network interfaces.

Hardware

Software

Critical components: 24x7, 4
-
hour response
on
-
site maintenance

Critical system problems: 24x7, 4
-
hour
response, remote telephone and remote login
support

Non
-
critical components: 8x5, next business
day, on
-
site Hardware maintenance

Non
-
critical system problems: 8x5, next
business day, remote telephone and/or on
-
lin
e problem reporting and tracking

On
-
site spare parts inventory for commonly
failing components with next business day
replacement of parts used from inventory.

All software shall be provided with 4
-
year,
unrestricted multi
-
user licenses.


The Offeror sha
ll describe the level of maintenance and support proposed for the UWRC HPC
and HPS resources, and explain how any differences from the target levels are designed to
meet the above reliability specifications and are advantageous to University. The Offeror s
hall
provide location information for the parts depots that would serve Critical Component response.


5.7

UWRC Software Support [Imp]


The UWRC HPC and HPS resources shall be supplied with software support services that
include the above
-
specified 24x7, 4
-
hour

response, remote support for critical system incidents
and critical software subsystems (such as the file system software products, job scheduler and
resource management subsystem) and 8x5, next business day response remote support for
non
-
critical incide
nts. In addition, software support services shall provide the following to UW
system administrators, user support and operations staff:

1.

Access to a support website, and telephone support personnel, for the communication of
software problems, including the

submission of problem reports, querying of problem status
and resolution.

2.

Access to a support website, or functional equivalent, containing bug reports and software
patch packages, available 24x7.

3.

Electronic access to images of current and previous versio
ns of system installation media,
available 24x7.

4.

Electronic mail notification of the availability of new software patches.

5.

Electronic mail notification of the availability of patches for University of Wyoming
-
reported
problems.


UWYO HPC RFP


January 2012

Page
23




The Offeror shall provide s
ecurity
-
related software patches expeditiously.



The Offeror’s proposal shall describe the Offeror’s software support processes and procedures,
and identify the Offeror’s timeliness targets for providing security
-
related patches to customers.


5.8

UWRC Train
ing [Req]


The UWRC HPC and HPS resources shall be provided with sufficient training for UW engineers,
system administrators, user support staff and operators so that the systems can be installed,
administered, maintained, supported, and interoperate. Training shall

include system hardware
maintenance (if customer
-
assisted maintenance is proposed), software maintenance, system
administration (including bare
-
metal restore procedures), compiler suite usage, application
performance tools usage, system monitoring and ope
ration, and Offeror
-
supported hardware
and software problem reporting procedures.

Training may be composed of both on
-
site (i.e. at the University of Wyoming) and remote (e.g.,
at the Offeror’s facility) venues and may be composed of custom training target
ed to the
delivered systems and or standard classroom or web
-
based and self
-
directed training. Initial
training shall be provided within nine (9) months of the subcontract award, and training shall be
provisioned annually throughout the subcontract lifeti
me, as mutually determined by UW and the
Offeror. Initial training shall accommodate up to six (6) engineers, administrators, user support
staff and/or operators, whether conducted on
-
site or remotely.


Initial training shall include Offeror recommended
procedures for system administration,
maintenance and operation that is sufficient for UW to conduct the Acceptance Testing activities
on each system after installation and prior to the acceptance test period. After the initial training
is complete, annua
l training should accommodate up to forty (40) hours for each of up to three
(3) trainees for at least two (2) years after the completion of all initial training.

Additional training, as appropriate, should be provided in the event that Offeror's hardwar
e,
software products, or administrative and/or operational procedures for UWRC systems change
significantly, or if new systems(s) are installed which would warrant additional training due to
changes in architecture, administrative and/or operational proced
ures.

The Offeror’s proposal shall describe the initial training that will be supplied with the UWRC
HPC and HPS resources and what provisions will be made by the Offeror for ongoing training.
In addition, the technical and price volumes of the Offeror’s
proposal shall list any additional,
optional training, beyond that proposed, that is available for the UWRC HPC and HPS
resources
.




UWYO HPC RFP


January 2012

Page
24



6

UWRC Environment

The general attributes of the hardware and software comprising the HPC environment is
provided in the following sections. For each of the following sections the offeror's proposal shall
include a:



Technical section describing how the Offeror's system conf
orms to the hardware
specifications. As part of the proposal the Offeror shall describe any and all exceptions
to, or non
-
compliance with the hardware specification.



Pricing section including pricing information for all equipment being offered as part of t
he
Offeror's proposal.


6.1

UWRC Architecture [Req]


Offerors shall provide a comprehensive description of the architecture of their proposed UWRC
HPC and HPS systems(s), including, but not limited to the following:



How it may differ from the composition descr
ibed in this RFP,



Number and type of nodes for each UWRC resource,



Number and type of cores per node, the memory hierarchy,



Memory bandwidth and latency within a node, per processor chip and per core,



Inter
-
node communications technology and its topolo
gy,



Theoretical and MPI
-
measureable bandwidth and latency,



Disk storage technology,



Interconnect technology between physical disk subsystems and servers, their respective
interconnect technologies and data transfer speeds,



Power and cooling
requirements.


For the purpose of these technical specifications, the UWRC HPC resources, and the server
portions of the UWRC HPS resources, shall be one or more clusters of nodes, where a node is
defined as a basic building block of a possibly larger system containing o
ne or more processor
chips and sharing a common memory, with uniform or near
-
uniform memory access by all
processors on the node. These nodes are aggregated to form a system, interconnected with
some form of inter
-
node communications switch, network or ot
her Infiniband fabric.

The UWRC HPC resources will be the primary computational environment for user activities.
The HPC shall be composed of login, management, and batch computation nodes and sufficient
I/O aggregator nodes with peripheral interfaces to
the UWRC HPS interconnect for sustaining
high
-
performance I/O to and from the UWRC resources. The UWRC HPC resources shall have
sufficient resource
-
local disk storage as required for their operation and system administration.

The UWRC HPS resources shall
provide the disk
-
based data storage managed by high
-
performance, parallel file systems. They shall be composed of the following:


UWYO HPC RFP


January 2012

Page
25





Server systems that support and serve high
-
performance file systems;



Disk storage systems;



Interconnect Infiniband fabric to

the UWRC HPC system(s);



Local and wide
-
area networks.


The UWRC HPS resources shall have sufficient resource
-
local disk storage as required for their
operation and system administration.

While the UWRC HPC and HPS resources shall all be operational for

normal user activity, any
of these UWRC resources shall be administrable and capable of being shut down and started
independently of the other UWRC resources.

Should the Offeror’s proposed UWRC solution substantively differ from the above (e.g., if the
HPC resources share a common high
-
performance interconnect, or if the Offeror’s solution does
not require I/O aggregator nodes), the Offeror’s technical proposal shall describe the differences
and how they are advantageous to University of Wyoming.


6.2

Softwa
re

The general software to be provided to the University is described in the following sections. The
Offeror shall provide a full
-
featured software stack, including:



operating system,



resource manager and job scheduler,



standard compliant optimizing compilers,



runtime libraries,



monitoring tools,



debuggers,



performance analysis tools and



support tools for hardware and system management.


The Offeror, collaborating with the University, will install the initial softwar
e stack.

The Offeror will ensure that all licenses are properly transferred to the University.

The technical section in the Offeror’s response shall describe what software outlined below will
be provided.

The University may optionally choose to purchase an
d install portions of the software stack.
The Offeror shall provide configurations and pricing for necessary and proprietary software as
well as the complete software stack.



UWYO HPC RFP


January 2012

Page
26



6.2.1

Operating System [Imp]

Due to the amount of Linux expertise at the University,

the University prefers that the HPC use
one of the following Linux distributions, RedHat Enterprise Linux (RHEL), CentOS or Scientific
Linux. The distribution release will be version 6.X or later and support a 64
-
bit application
environment. The Univers
ity welcomes solutions with multiple operating systems, specifically
ones that propose open source operating systems on the compute nodes.

6.2.1.1

Kernel [Req]

The Offeror’s hardware solution must not require a modified kernel, except to provide driver
support as
outlined in Section
6.2.1.3
.

6.2.1.2

Infiniband Stack [Req]

OpenFabrics Enterprise Distribution (OFED) stack as distributed by the Operating System
provider. The OpenSM Infiniband subnet manager will be used as part of the Infiniband stack.

6.2.1.3

Device Drivers [Req]

Any device driver required to support the Offeror’s hardware, which is not available in the
operating systems mentioned in Section
6.2.1
's standard distribution, will be provided by the
Offeror. This additional or modified software shall be provided in so
urce form.

In the case of the GPU drivers, the Offeror may provide the driver in a binary only format.

6.2.2

Resource Manager and Job Scheduler [Req]

The University prefers the Moab Cluster Suite as a professional cluster manager solution that
integrates schedul
ing, management, monitoring and reporting of cluster workloads.

Moab Cluster Suite (
http://www.clusterresources.com
)

For cluster provisioning the University prefers the Bright Cluster Manager which provides
simple
installation of bare metal systems.

Bright Cluster Manager (
http://www.brightcomputing.com
)

If the Offeror proposes a different Resource and Job Manager solution, then it should be fully
described in th
e proposal and include a justification as to why it presents the University a better
solution. Note that the Offeror’s solution should be capable of being installed on existing
University clusters.

6.2.3

Compilers [Imp]

The Offeror shall include a compiler suit
e that includes FORTRAN, C/C++, as provided by Intel,
AMD , Pathscale, or Portland Group.

The Offeror shall include a standard set of optimized libraries that include BLAS levels 1, 2, and
3, LaPack and ScaLAPACK.


UWYO HPC RFP


January 2012

Page
27



The HPC shall support multiple versions
of all compilers, libraries and tools concurrently, see
section
6.2.4.

6.2.4

User Environment Support [Req]

The Offeror shall provide a software package capable of allowing the user to manage their work
environment. This software should allow a user to manage a
nd dynamically change his or her
compiler, libraries, debuggers, runtime features and any package where multiple versions are
found within the HPC cluster. The Offeror shall provide a setup of pre
-
canned templates for any
supplied software.

6.2.5

Cluster Monito
ring [Imp]

The Offeror shall provide and help implement a set of cluster monitoring and performance
measuring tools such as Ganglia and Nagios. This monitoring shall include CPU, memory, I/O,
and process use. System environmental data (CPU temperature, fa
ns, and system voltages)
shall be collected using the IPMI/BMC interface via out of band management.

6.2.6

Parallel Computing Environment [Imp]

The Offeror shall provide a set of tools to create parallel applications that meet the OpenMP and
MPI standards. The
Offeror shall include OpenMPI, MPICH, MPICH2, MVAPICH and
MVAPICH2 as part of their parallel environment
.

6.2.7

Debugging Tools, Performance Measurement, Analysis and Tuning Tools
[Imp]

The Offeror shall provide a comprehensive suite of debugging, performance me
asurement,
analysis and tuning tools. This suite of tools should include tools to:



Monitor and determine parallel applications behavior and performance



Tune commands, subprograms and code sections



Monitor and measure an applications effect on the overall
system



Debugging, monitoring and timing user codes, from a subroutine to a whole application



Monitor File system and I/O performance.

6.2.8

BIOS Management Tools [Req]

The Offeror

shall include BIOS management tools required to support, distribute BIOS
configuration and upgrade the Offeror’s hardware. All tools must run from the host Operating
system command line.

6.3

Technical Requirements for HPC, HPS and Network [Req]

The planned r
esult after 3 years of University procurement is a HPC of highly integrated, well
-
balanced, CPU and CPU/GPU compute nodes, an HPS providing data storage to researchers
and an Infiniband network.


UWYO HPC RFP


January 2012

Page
28



The HPC cluster shall consist of Login, management and compu
te nodes; all interconnected via
the high speed Infiniband Interconnect, and multiple Gig networks. The University anticipates
needing a large number of coupled Linux systems that integrate with the HPS file system
through the Infiniband network. Our requ
irement is to build a cluster from enterprise AMD x86
-
64 or Intel EM64T nodes containing at least two (2) or more microprocessor sockets. These
nodes shall have at least one Infiniband interface and two (2) GIG interfaces.

The envisioned HPS shall consis
t of backend storage nodes with frontend nodes providing
access to the HPS from the different campus clusters and campus. Some form of high speed
interconnect (possibly FDR Infiniband) shall be between the frontend and backend nodes.

Finally, a high speed

FDR Infiniband network, shall connect the HPS campus clusters together.

The Offeror shall be responsible for building and performing pre
-
ship testing of all the individual
components. The Offeror, with the University, shall integrate the HPC and HPS into

the
University datacenter and perform any acceptance tests.

This RFP is for the initial delivery of a HPC Cluster and HPS Storage System. It is planned that
over a three (3) year period additional equipment will be added to the HPC and HPS, and that
fu
ture regular investments in the UWRC will enable regular upgrades to the HPC and HPS
systems. The University encourages design of HPC systems that are designed for future
upgrades, expansion and integration of next
-
generation HPC technologies. The system
(s)
should be able to offer easy upgrade or expansion of CPU, GPU, memory, I/O or network
capabilities. It must be possible for the University to adjust system configurations to meet the
current and changing requirements easily and cost
-
effectively.


6.3.1

Netw
ork Infrastructure [Req]

Solid network infrastructure is vital to the success of the University HPC solution. Networking for
the UWRC environment needs to maximize speed and expandability while minimizing cost.

The network design will consists of four (4)

network components:



A one (1) Gb network for low
-
level interactions such as ssh traffic, job submission,
monitoring via IMPI, and setup within the cluster,



A one (1) Gb network to interface to the campus network.



A ten (10) Gb interface between the HPS a
nd the campus network,



An Infiniband (IBA) network to handle the MPI traffic within the HPC cluster and connect
the HPC to the HPS
.


In order to minimize support and maintenance issues, all non
-
Infiniband

network switches shall
be supplied by one of the following four approved Offerors:



Cisco



Juniper



Force10



Arista



UWYO HPC RFP


January 2012

Page
29



Figure 2 illustrates the four networks envisioned and how they will integrate into the existing
network infrastructure.



Figure 2


While we recognize that Offerors may wish to include networking components specific to a
particular RFP section, we ask that Offerors provide a separate networking section in their bid
responses that enumerates all networking components proposed, including

cost, vital statistics,
and to which RFP section the networking component is associated.

Any proposed network equipment shall include redundant power supplies.

The Offeror's response shall include a growth capacity of 40% in number of connections.

All O
fferor supplied switches should be capable of link aggregation using the IEEE 802.1ax
-
2008 standard protocol.


UWYO HPC RFP


January 2012

Page
30



All interconnections between the Offeror provided network structure and the campus core switch
shall be based on 50 micron fiber.

Offerors shall
bid all switches within the HPC Condo or HPS sections as needed.

UW would prefer to support the non
-
Infiniband network infrastructure of the solution itself.

All warranty network hardware maintenance shall be in the University name not that of the
offero
r. The University desires to have direct access to network TACs and software downloads
without going through the offeror's support process.

6.3.1.1

Gig Management Network [Req]

This one (1) Gb network will be used for low
-
level cluster traffic such as ssh

sessions, scheduler
communications, monitoring via IMPI 2.0, and cluster management setup. In addition, this
network will be tied to the VSN that will host license and database servers, web servers, and
possibly net
-
boot servers. Should the Offeror use no
n
-
blade chassis (i.e. 1u or multi
-
u) in their
solution, the Offeror shall configure with top of the rack switches and uplink to an aggregate
switch.

6.3.1.2

Gig Campus Connect Network [Req]

This one (1) Gb network will be connected to the campus network and will p
rovide users access
to the HPC environment. Offerors should bid all switches within the HPC Condo or HPS
sections as needed to connect this network to the campus network core switch. This switch
should be capable of connecting via 10 Gb fiber back to the
core switch. Should the Offeror use
non
-
blade chassis (i.e. 1u or multi
-
u) in their solution, the Offeror shall configure with top of the
rack switches and uplink to an aggregate switch, which is then uplinked to the core switch.

6.3.1.3

Ten (10) Gig Interface [Re
q]

The HPS front
-
end servers are to connect to a ten (10) Gb/s aggregate switch, which is then
connected to the campus network core switch, using the ten (10) GIG interface. The Offeror's
HPS solution should be capable of link aggregation of these connecti
ons for reliability and
performance.

6.3.1.4

Infiniband [Req]

The RWSC environment shall be built with an FDR Infiniband network (IBA) interconnect. The
IBA will be used to connect the HPC condo and the HPS storage subsystems, and possibly
existing University clus
ters. All nodes shall have at least one Infiniband HBA of FDR or faster
speed, with PCIe 8x bus interface. The IBA HBA should be placed on a private PCIe bus.

The IBA network shall deliver a minimum MPI ping/pong latency (round trip divided by two) of
no
more than 0.7µs as measured between any and all two MPI task pairs in the cluster. As part
of the University acceptance, it will run the LLNL “com” benchmark from the ASC Purple Presta
MPI Stress Test Suite, available at


https://asc.llnl.gov/computing_resources/purple/archive/benchmarks/presta/
”.

The Offeror

will provide a fully functional IBA consisting of a FDR fabric, all HBA’s, IBA
switches, and cables to allow the University to construct a full fat
-
tree topology full bandwidth,

UWYO HPC RFP


January 2012

Page
31



non
-
blocking within a block of nodes, Infiniband fabric. A node
-
block should
consist of a
maximum twenty eight (28) HPC nodes which maintain a non
-
blocking infrastructure. Going off
a node
-
block may allow for a minimal blocking infrastructure. It is left to the Offeror to propose
the most efficient and cost effective Infiniband i
nfrastructure. The overall infrastructure should
not exceed and fifty (50) percent blocking factor.

All Infiniband hardware shall be compatible with the OFED software stack as provided with the
Linux Operating System. In addition, the Offeror’s solution
shall allow for the PXE booting of
compute nodes over the Infiniband network.

As part of the overall IBA fabric, the Offeror’s design will accommodate a connection to existing
University clusters at full bandwidth and non
-
blocking speed. Existing clusters
may consist of
HPC nodes linked by SDR,QDR Infinibands connected via Cisco, Mellanox switches.

In the case that the Offeror proposes either 1U or 2U cluster hardware, the Offeror should
configure the IBA network using top of rack switch and provide aggreg
ate switches to connect
all racks. The aggregate switch should have additional capacity to allow additional existing
clusters to be connected to the fabric, approximately sixteen (16) ports.

6.3.1.5

Subnet Manager [Req]

As part of the Infiniband fabric, the Offero
r shall provide two nodes to perform the functions of a
subnet manager. The subnet manager (OpenSM) assigns Local Identifiers (LIDs) to each port
connected to the Infiniband fabric, and develops a routing table based off of the assigned LIDs.

The systems s
hall run the Linux (OpenSM) subnet management software and shall be
compatible with the OFED software stack. These systems should be configured in a high
available environment in case of a primary system failure. This HA environment should be
configured i
n a master/slave configuration, so that if the master should fail the slave would take
over managing the Infiniband network. The HA configuration should provide shutdown and
automatic restart of an unresponsive master, and migration to the backup when fai
lures on the
master are detected. The cluster backup node assumes the cluster name and IP address and
automatically takes over system processing until such time as the failure is corrected on the
master and a failback is initiated (manually returning proce
ssing to the master system).

Minimum specifications for these nodes are found in section
6.3.3.6

of the RFP.

Neither a vendor nor hardware switch specific subnet manager is acceptable.

6.3.1.6

Jumbo Frame Support [Req]

The Offeror's networking shall support Jumbo
Frames, whether implemented in hardware or
software, of frame size of nine thousand (9000) byte minimum transmission units (MTU). The
Offeror’s proposal shall state the support to be provided for Jumbo Frames in the proposed
hardware and/or software.

6.3.1.7

Lin
k Aggregation [Req]

The Offeror's networking solution shall support Link Aggregation per IEEE 802.1ax
-
2008. The
technical section of the Offeror’s proposal shall state what support is provided for Link
Aggregation for the Offeror's proposed hardware and s
oftware.


UWYO HPC RFP


January 2012

Page
32



6.3.2


HPS Storage System Requirements [Req]

The University HPS shall be engineered from one or more systems, containing front end
storage servers, management system(s) and all associated disk storage, network fabric(s) for
data transfer and control communication(s) for internal operations, as well a
s data transfer and
control from the University HPC and campus community.

Figure 2 shows a conceptual diagram of the University HPS and how it shall integrate with the
HPC and University environment. It should be composed of a large pool of disks, possib
ly SSD,
and front
-
end server hardware to connect to the University Campus network via 10GigE
aggregated connections, and a high
-
speed Infiniband connection that provides I/O connectivity
to the HPC and existing campus clusters.

Since the HPS will be used
for storage of all research data, it is expected that there will not be a
consistent file size and type hosted on the HPS. The Offeror’s proposed solution should
therefore take these issues into consideration and provide a solution that performs well unde
r all
transaction types.

For this section and its subsections of the RFP, the Offeror’s proposal shall include the following
data:



A technical section, with a complete description of the proposed HPS environment,
description of all HPS hardware, their comp
osition, component type and quantity and
any other technical attributes.



A technical section, containing a list of the exceptions and changes to these
specifications and how the exceptions are advantageous to the University.



A pricing section, including pr
icing information for all equipment being offered as part of
the Offeror’s proposal.

6.3.2.1

HPS Capacity [Req]

The Offeror's proposed HPS solution shall be composed of an extensible set of equipment for
hosting and serving a high
-
performance file system to the Un
iversity community, HPC
environment, and interfacing to other computational systems on campus.

The Offeror shall propose a HPS solution with an initial usable storage capacity in the range of
several hundred TB, with a minimum expansion to multi
-
petabytes
over a three (3) year period.
Budget allowing, an initial configuration between 500 and 750 TB is envisioned. All drives
proposed shall be generally available for a minimum of 2 months.

The Offeror as part of the HPS proposal should state the usable ca
pacity of the HPS, and
whether the solution meets or exceeds these requirements.

The University shall decide one final size of the storage system based on price per TB. Priority
will be given to the Offeror who offers the best performance per cost and has
the most granular
and cost effective capacity growth proposal.

6.3.2.2

HPS Performance [Req]

The Offeror should meet an overall performance throughput requirement of two hundred (200)
Gb/s using a four (4) K block size over both the Infiniband and the 10Gb/s fro
nt end interfaces.


UWYO HPC RFP


January 2012

Page
33



The Offeror's solution shall be provisioned and engineered for resiliency, fault
-
tolerance, and
serviceability and most of all to assure data integrity with zero (0) data corruption. It is therefore
assumed that RAID
-
6, or similar techn
ology, shall be used to enhance data storage, equipment
reliability, and performance.

6.3.2.3

HPS Interface Protocols [Req]

6.3.2.3.1

File Sharing Protocols [Req]

The Offeror's

proposed solution shall include support for the following operating systems:
Linux, Windows, and MacOS; and use the following file sharing protocols: NFS, CIFS. The
Offeror’s HPS shall be robust enough to support all protocols without file system corru
ption. The
Offeror should state as part of their technical section how they meet this requirement and if not,
should state why the Offeror's solution provides a better solution to the University.

6.3.2.3.2

Authentication Protocols [Req]

The Offeror's solution shall
support Active Directory and NIS authentication at the same time.
The Offeror should state as part of their technical section how they meet this requirement and if
not, state why the Offeror's solution provides a benefit to the University.

6.3.2.4

HPS Proposed
Storage Tiers [Req]

The University anticipates that the storage should be configured into the tiers as outlined in the
following sections. The Offeror's solution should allow for dynamic volume expansion and
hardware expansion without interruption to the u
ser community. The Offeror's solution shall
provide software or a means to perform automatic migration of data among the tiers of storage
via administrator defined migration rules

6.3.2.4.1

Tier One [Req
]

This tier of the Offeror's solution should be composed of ve
ry fast disks (15000 RPM drives) or
SSD drives. The initial size of tier capacity should be approximately ten (10) percent of the
HPS.

6.3.2.4.2

Tier Two [Req]

This tier of the Offeror's proposed solution should be composed of medium speed disk (7200 or
10000 RPM
drives). The initial size of this tier will be approximately ninety (90) percent of the
initial storage capacity of the HPS.

6.3.2.4.3

Tier Three [Req
]

The third tier of the HPS solution will be used for long
-
term storage and archival of files housed
in the HPS. T
his tier shall be integrated as a functioning tier in the file system/storage
architecture. Once a file is moved to this tier, it will still be seen in the file system, and can be
restored to either a Tier One or a Tier Two location without the user being

aware where the file
is actually located.


UWYO HPC RFP


January 2012

Page
34



It is left to the Offeror to propose an archival system to the University. Suggested technologies
are: slow online disks, tape library, or a VTL solution. The solution should be a complete
archive/backup solutio
n that employs retention periods and is capable of keeping multiple
versions of the same file. Preference will be given to solutions that employ cost saving features
such as data de
-
duplication. The Offeror's solution should be totally integrated into the

overall
HPS environment. As part of the proposal, the Offeror should describe in detail their proposed
solution
and how it is integrated into the HPS.

6.3.2.5

HPS Proposed Configuration [Info]


The Offeror's solution should consist of management nodes, file syst
em nodes, front end
servers as well as all required metadata and database server nodes. The Offeror's design
should allow for simple expansion of all Tiers of the storage system.

The file system nodes will be used by the Offeror's file system(s) to manage

all data, metadata,
and databases associated with the Offeror's proposed high performance parallel file system(s).

Management nodes will be used solely by the system administrators and will be used
exclusively for managing, maintaining and operating the

HPS, and HPS services. The Offeror
shall propose a number of nodes to provide a resilient, fault
-
tolerant, and assured data integrity.

The front end, server nodes, will provide general access to the HPS from the HPC and campus
users. The Offerors

solution should provide link aggregation and redundant nodes in case of
failure.

Should the Offeror's solution differ from the proposed solution outlined above, the Offeror's
technical proposal shall describe differences and how the differences will benef
it the University.

The node type and quantities shall be determined and described by the Offeror in the proposal.

6.3.2.6

HPS External Networking [Req]

6.3.2.6.1

Infiniband Support [Req]


The Offeror's solution shall provide support for Infiniband connected Clusters as sh
own in
Figure 2.

6.3.2.6.2

10GigE Support [Req]

The Offeror's solution shall provide support for multiple 10 Gb/s Ethernet connections to the
University Campus core switch. It is left to the Offeror to engineer a solution that meets these
University needs, i.e. mu
lti
-
front end servers with aggregate ten (10) Gb/s connections directly
to the core switch and/or an aggregate switch(s) tied to the campus core switch. It is left to the
Offeror to purpose the best solution.


UWYO HPC RFP


January 2012

Page
35



6.3.3

HPC Condo Resources and Requirements [Info]

The

University HPC resource shall be composed of the following systems:

Node Type

Minimum Quantity

Description

Login

Offeror

shall provide a
minimum of two (2) to
meet the University’s one
桵湤r敤 (㄰MF 捯c捵cre湴n
畳ur猠 慮搠 潮攠 t桯畳u湤
(㄰〰F 灲p捥獳敳e
req畩r敭敮t.

q桥獥s 湯摥猠 will 扥 u獥s for
畳ur logi測n i.e. 慳at桥 条t敷慹a
t漠 t桥 emCI 獵smi獳i潮 潦o
扡t捨c j潢sI 獯ftw慲a
m慮ag
敭敮tI 敤iti湧 慮d
捯浰cl慴楯湳⸠ q桥獥 湯摥s
獨s畬搠畳u t桥 獡m攠pr潣o獳潲
捨c瀠 慳a t桥 捯浰cte 湯摥献
p敥 p散瑩e渠S.㌮P.ㄠf潲 f畲t桥r
湯摥 獰s捩fi捡瑩c湳.

M慮慧敭敮e

㈠乯摥s

q桥獥 湯摥猠will 扥 畳u搠潮l礠
批 t桥 獹獴敭e 慤mi湩獴rat潲s
慮搠畳u搠獯s敬y f潲

ma湡gin本
m慩湴慩渠 慮搠 潰er慴a湧 t桥
emC 獥牶i捥献c q桥獥 湯摥s
獨s畬搠畳u t桥 獡m攠pr潣o獳潲
捨c瀠 慳a t桥 捯浰cte 湯摥献
p敥 p散瑩e渠S.㌮P.㈠f潲 f畲t桥r
req畩r敭敮ts.

C潭灵oe

lff敲er 獨sll 灲潶i摥 a
mi湩m畭u of tw漠 (㈩O t漠
m敥t t桥 r湩v敲獩ty
req畩r敭敮ts.

r獥s t漠 r畮 扡tc栠 j潢猬 慮d
m慹 扥 畳u搠 ex捬畳uvel礠 (i.攮
j潢 摯敳e 湯t 獨sr攠 th攠 湯摥
wit栠潴桥o j潢猩. All 湯d敳e獨sll
桡v攠t桥 獡s攠灲p捥獳潲o 捨cp
wit栠i摥湴楣nl 捯牥c捯cnt献 p敥
p散瑩e渠 㔮R.㈮㌠ f潲 f畲t桥r
req畩r敭敮ts.

sirt畡l p敲e敲

ㄠ乯摥

q桩猠獥r
v敲ewill 扥 畳u搠批 t桥
獹獴敭e 慤mi湩獴rat潲猠 to
灲潶i摥 獰s捩慬iz敤 virt畡l
獯ftw慲攠敮vir潮m敮tsI 獵捨 慳
li捥c獥s 獥牶敲猬 摡t慢慳a
獥牶敲e etc.

p畢湥t M慮慧敭敮t

㈠乯摥s

q桥獥 獥牶敲e 慲a 畳ud
數捬畳uv敬y 慳a t桥 f湦楮i扡湤
獵s湥t m慮agem敮t 桯st献


l湥 of t桥 m潳t im灯rt慮t 摥獩g渠crit敲楡 i猠th慴a t桥 潶敲慬l rtoC 敮vir潮m敮t 扥 fl數i扬攠
敮潵gh to 慣捯mm潤ate t桥 f潬l潷i湧 t散e湯logy im灲潶敭敮t猺



Processor speed improvements within the same family of processors



New processor socket and/or chipset im
provements



New processors featuring new core design


UWYO HPC RFP


January 2012

Page
36





Larger and faster SATA, SAS or SSD disks



New memory speed and capacity improvements



Faster interconnect providing bandwidth and latency improvements.

For this section and subsections of the RFP, the Offer
or’s proposal shall include the following
data:



A technical section, with a complete description of the proposed University HPC
environment, description of all HPC hardware, their composition, component type and
quantity and any other technical attributes.



A technical section, containing a list of the exceptions and changes to these
specifications and how the exceptions are advantageous to the University.



A technical section that lists the date of first sale of all components, vendor’s expected
sales life t
ime for the component, length of support period for the component following
end of sales and for any chassis solution the expected support period for new
processors inside the proposed chassis.



A pricing section, including pricing information for all equip
ment being offered as part of
the Offeror's proposal. The pricing section should also include a granular breakdown for
expansion equipment. The Offeror should include a cost projection timeline.

6.3.3.1

General Requirements [Req]

6.3.3.1.1

Virtualized Memory Environment [En
h]


The University is interested in a shared memory or single address space cluster with features
that include, a shared memory interconnect that virtualizes I/O and memory, allowing
applications to use all memory in the HPC.

The Offeror, as part of their
technical response, should include as an option the details of what
they offer in this area.

6.3.3.1.2

Node Form Factor [Req]

The University prefers denser solutions (including blades), meeting the power, cooling and
facilities requirements in Section
5.1

are desire
d.

6.3.3.1.3

PCI Bus[Req]

PCI buses and all add
-
on HBA cards shall be Gen 3 if possible.

6.3.3.1.4

Networking [Req]

6.3.3.1.5

GIG Connections [Req]

Each node shall include two (2) GIG Ethernet ports, one to be connected to the management
network, the other to the Campus network for
user access.


UWYO HPC RFP


January 2012

Page
37



6.3.3.1.5.1

HP Interconnect [Req]

Each node shall include a high
-
performance FDR Infiniband interface that connects the node to
the high performance backbone network.

6.3.3.1.6

Node Initialization and Booting [Req]

6.3.3.1.6.1

Initialization [Req]

The node BIOS initialization
process shall complete without human intervention (e.g. pressing
any key on a keyboard to continue) or an error message written to the console. The time
required for a node’s BIOS to take the node from a power off state (or reset state) to the start of
lo
ading the Linux kernel shall be less than thirty (30) seconds. Note that this includes the POST
step settings. Shorter times are clearly desirable.

6.3.3.1.6.2

System Booting [Req]

The BIOS should be capable of booting the node from different boot sources, including
CD/DVD, USB Key or network booting via PXE over Ethernet or Infiniband.

6.3.3.1.7

Node I/O Configuration [Req]

The Offeror should configure each node with a minimum of three (3) PCIe Gen3 8x or faster
busses and two (2) PCIe 8x slots. One bus to be used exclusively
by the Infiniband HBA, the
other bus to be used by the GPU equipped compute nodes and the third bus to be used by
other interfaces, i.e. the GIG network interfaces.

6.3.3.1.8

Remote Management (IPMI 2.0/iKvm) [Req]

All nodes shall be 100% capable of being remotely m
anaged and all routine administration tasks
automatable in an efficient manner. The Offeror’s solution for remote management shall be
compatible with the IPMI 2.0 and BMC style remote management. The Offeror’s solution
should be compatible with Serial ov
er LAN (SOL) and iKVM features that allows for remote
console access. In particular, all service operations, under any circumstance on a node shall be
accomplished without the attachment of a keyboard, video monitor and mouse locally or via a
KVM.

In the

event that the IPMI/BMC is not integral to the base motherboard, the IPMI/BMC daughter
card (or equivalent) shall be provided. The Offeror shall provide a fully compliant IPMI/BMC 2.0
implementation, and any security relevant features within the IPMI/BMC

specification shall be
supported and configurable. All IPMI/BMC functions shall be available to and accessible from
the host Linux System via network or in
-
band. The University is planning on using the Linux
IPMI tools to manage these features. In addi
tion, the University is considering using the
Powerman and Conman tools for remote management. The Offeror should be able to provide
software and support to help integrate with these tools. The Offeror may recommend an
alternate solution to Powerman and Co
nman. If the Offeror provides an alternate solution please
provide details in the technical section and to why the Offeror's solution would be beneficial to
the University.


UWYO HPC RFP


January 2012

Page
38



The Offeror shall provide tools that allow the IPMI/BMC firmware to be upgraded an
d allow
verification of the upgraded BMC firmware. These tools shall be able to perform the firmware
upgrade in
-
band; an out
-
of
-
band solution is not acceptable.

6.3.3.1.9

System Monitoring [Req]

As part of the Offeror's remote management solution (Section
6.3.3.1.8
), the Offeror’s solution
shall include capabilities to remotely monitor all hardware sensors and provide sensor codes as
listed in (Table 42
-
3 of the IPMI 2.0 specification). The Offeror may not provide their own
sensor values and interpretation.

The
IPMI/BMC and motherboard hardware shall provide the following sensor data through the
IPMI 2.0 protocol:



Each and Every Fan speed within the node,



Temperature of every processor ,



All motherboard temperature sensors,



Voltage supplies, to each socket, batte
ry, and power supply,



Processor state,



Power supply state.

Temperature sensors should be designed to be sensitive to manufacturing tolerances while
remaining highly accurate. Offeror shall provide documentation on any OEM specific sensors so
that the Unive
rsity can accurately and correctly interpret the sensor. An individual sensor shall
be provided for each power supply and processor that exists in the system. A single sensor that
represents multiple devices, power supplies, or processor/cores is not acc
eptable.

6.3.3.1.10

Firmware Management [Req]

The Offeror shall provide a set of software tools that will allow the University to perform BIOS
firmware updates. The Offeror shall also provide a written plan to be submitted with the bid
proposal outlining processes t
hat provide BIOS updates to address problems or deficiencies in
the area of functionality, performance and security. The plan shall outline a process that the
University personnel can follow to identify, prioritize and implements BIOS updates in general
a
nd for addressing any specific University related issues and concerns that may be raised by
the Offeror or University. The plan shall be finalized after contract award and include as an early
deliverable the Statement of Work for the subcontract.

6.3.3.1.11

System Po
wer [Req]

The proposed login nodes should use less than two hundred (200) Watts per socket plus one
hundred fifty (150) Watts when running a copy of the LinPACK on each core. Power supplies
shall have a minimum efficiency of 80% when tested at each of the

following load conditions:
20%, 50% and 100% of rated power supply output. All power shall be configured in a redundant
configuration. Power supplies should comply with green computing standards.


UWYO HPC RFP


January 2012

Page
39



6.3.3.1.12

Memory Error Detection [Req]

The Offeror’s systems shall
include a hardware mechanism to detect correctable and
uncorrectable memory errors from Linux. The hardware shall be capable of sending a non
-
maskable or machine check exception when an uncorrectable error occurs, so that Linux
system may take appropriate
actions. When a correctable or uncorrectable memory error
occurs, the hardware shall provide sufficient information to the Linux system to identify the
affected failed or failing DIMM FRU (i.e. the exact DIMM location on the motherboard).

6.3.3.1.13

System Spares [R
eq]

If the Offeror chooses the onsite support option, then the Offeror shall provide to the University
a reasonable quantity and breadth of spares so that University personnel can make simple
hardware repairs. These parts should include disks, memory, pow
er supplies. The spare parts
should be determined by the Offeror, and be based on industry best practices and Offeror's
hardware failure experience.

6.3.3.2

Login Nodes

The Offeror shall provide at a minimum two (2) Login nodes for redundancy.

6.3.3.2.1

CPUs [Req]

Each L
ogin node shall be configured with a minimum of four (4) AMD64 (Interlagos) or Intel
EM64T (Sandy Bridge) processors, with not less than twenty four (24) total cores. The total
login node resources shall, in aggregate, support in excess of one hundred (100
) active
concurrent users and over one thousand (1000) active concurrent processes.

6.3.3.2.2

Memory [Req]

Each Login node shall be configured with not less than 128 gigabytes of memory, with each
processor supporting a minimum four (4) memory channels.


6.3.3.2.3

GPUs [Info]

No GPU’s are required.

6.3.3.2.4

Disks [Req]

Each node shall be equipped with two (2) 2TB disk drives configured in a Raid 1 for
redundancy; fast SAS or SATA disks are acceptable with a minimum of a 6Gb/s host interface.

6.3.3.3

Management Nodes

6.3.3.3.1

CPUs [Req]

Each Management
node shall be configured with at least two (2) AMD64 (Interlagos) or Intel
EM64T (Sandy Bridge) processors, with not less than twenty four (24) total cores.


UWYO HPC RFP


January 2012

Page
40



6.3.3.3.2

Memory [Req]

Each Management node shall be configured with not less than forty eight (48) gigabytes

of
memory, with each processor supporting a minimum four (4) memory channels.

6.3.3.3.3

GPUs [Info]

No GPU’s are required.

6.3.3.3.4

Disks [Req]

Each node shall be equipped with two (2) 1TB OS disk drives configured in a RAID 1 for
redundancy, fast SAS or SATA disk are accep
table, with a minimum of a 6Gb/s host interface.
In addition, the Offeror should provide in a Raid
-
6 configuration disk space to host the disk
image used for diskless booting. There should be 50% free space once all nodes are
configured.

6.3.3.4

Compute Nodes

The University envisions growing the HPC substantially during the first 3 years. Priority will be
given to the Offeror who offers the best performance per cost and has the most granular and
cost effective capacity growth proposal.

In addition, the Offer
or shall provide these nodes with the following distribution:



10% CPU/GPU Fat nodes with at least one GPU per node



5% CPU/GPU Thin nodes with at least one GPU per node



5% fat nodes



5% Fat nodes with many cores, i.e. greater than eight (8) cores



The rest as

thin nodes.

6.3.3.4.1

CPUs [Req]

Each Compute Node shall be configured with state
-
of
-
the
-
art enterprise class Intel processors
to support scientific applications. Offer’s may also provide pricing for AMD processors.

6.3.3.4.2

Memory [Req]

The compute nodes shall be config
ured with a minimal amount of memory to support academic
research programs. The Offer shall specify the amount of memory per node and articulate the
benefits of their recommendation

6.3.3.4.3

GPUs [Req]

The Offeror shall provide a number the nodes as per section
6.3
.3.4
, which will contain NVIDIA
GPU’s.


UWYO HPC RFP


January 2012

Page
41



6.3.3.4.4

Disks [Req]

Each node shall be equipped with one (1) 1TB disk drives, fast (10000 or 15000 rpm) SATA or
SAS disks are acceptable with a minimum 6Gb/s host interface. These disks will be used as
Swap, temporary disk
space (/tmp and /usr/tmp) and user scratch space. The Offer shall also
provide pricing for equipping nodes with one (1) 2TB drives.

6.3.3.5

Virtual Server (VS) Node

6.3.3.5.1

CPUs [Req]

The Offeror shall configure the Virtual Server node with a minimum of four (4) AMD64
(
Interlagos) or Intel EM64T (Sandy Bridge) processors, with not less than forty eight (48) total
cores.

6.3.3.5.2

Memory [Req]

The Offeror shall configure the Virtual Server node with not less than eight (8) gigabytes of
memory, with each processor supporting at a mi
nimum four (4) memory channels.

6.3.3.5.3

GPUs [Info]

No GPU’s are required for the Virtual Server.

6.3.3.5.4

Disks

Each node shall be equipped with two (2) 2TB disk drives for the operating system configured in
a RAID
-
1 configuration. Fast SATA or SAS disks are acceptable
with a minimum 6Gb/s host
interface.

The Virtual Server node shall be configured with at least one (1) highly reliable hardware RAID
-
6 in a (D+2P+S) configuration utilizing 1TB SAS or SATA drives with a minimum 6Gb/s
interface. The number of data (D) drive
s shall be at least eight (8), two (2) parity (P) drives and
one (1) spare drive. The RAID
-
6 array(s) shall deliver at least a five hundred (500) MB/s large
block (4096B) read/write bandwidth to the Linux server. The RAID
-
6 array(s) shall deliver at
leas
t one thousand (1000) IOPS for an I/O workload randomly reading/writing 4096B blocks
from a Linux EXT4 file system mount on a partition. The RAID
-
6 proposed solution shall be
compatible with the Linux LVM subsystem and support Linux multi
-
path. The RAID
-
6
device
shall be capable of notifying system administrators when a drive has failed. With the failure of a
drive, the array should be configurable to automatically rebuild the RAID
-
6 array using the spare
drive. All disks in the array should be hot swappa
ble and directly accessible from the exterior of
the system.

6.3.3.6

Subnet Management Node

6.3.3.6.1

CPUs

The Offeror shall configure the subnet management node with a minimum of two (2) AMD64
(Interlagos) or Intel EM64T (Sandy Bridge) processors, with not less than 8 tota
l cores.


UWYO HPC RFP


January 2012

Page
42



6.3.3.6.2

Memory

The Offeror shall configure the subnet management node with not less than twenty four (24)
gigabytes of memory, with each processor supporting at a minimum four (4) memory channels.

6.3.3.6.3

GPUs

No GPU’s are required for the Subnet Management Node.

6.3.3.6.4

Disks

The node shall be equipped with two (2) 1TB disk drives configured in a Raid 1 for redundancy,
fast SAS or SATA disk are acceptable, with a minimum of a 6Gb/s host interface.

7

Documentation

The UWRC HPC and HPS systems shall be provided with a full
suite of documentation for all
software and hardware products delivered with the system(s). This documentation suite shall
be in electronic form (e.g., PDF or HTML files) as well as in command
-
line “man pages” and
“info” for all software. Documentation s
hall include user
-
level manuals for all user products
(including compilers, libraries, performance tools.), and system administrator
-
level manuals for
management software, hardware and subsystem description documents, system interconnect
and MPI implementa
tion documents.

The Offeror shall also grant to The University of Wyoming the right to publish all such
documentation on a the University of Wyoming website and grant to the University of Wyoming
and UWRC HPC and HPS system(s) users the right to print copi
es of such documentation for
official, personal use.

The Offeror’s proposal shall describe the documentation to be supplied with the UWRC
system(s) and state compliance with the above specified publication and use rights, or
restrictions thereon, if any.

8

E
xpandability and the UW Condo Model

The University of Wyoming will be using the HPC and HPS systems in a Condo Model.
Departments and individual researchers will be contributing resources to expand the HPC and
HPS of the UWRC. The Offeror shall describe
the growth options for the HPC and HPC and
provide pricing for this expansion over three years following the initial installation. Pricing needs
to be specified by node type and storage tier.






UWYO HPC RFP


January 2012

Page
43





FORM OF PROPOSAL

THE UNIVERSITY OF WYOMING



Procuremen
t Services

The University of Wyoming

Dept. 3605, 1000 E. University

16th and Gibbon

Laramie, WY 82071


To Mr. Greg Livingston:


The undersigned offeror, having read the terms and conditions, together with all addenda and being
acquainted with and fully
understanding the requirements, submit our proposal to be opened at 2:00 P.M.,
MST, February 2
9
, 201
2
, in room 115, Merica Hall, Laramie, Wyoming.


The offeror acknowledges receipt of the following addenda:


_____________________ __________________ ____
________________



The offeror has identified clearly any variations between its proposal and the RFP and has broken down
the costs per function. The University prefers to purchase all functions from the same offeror, therefore,
special consideration will

be given to joint proposals that minimize integration problems, but reserves the
right to award the contract in the best interest of the University. The offeror understands that the
University reserves the right to consider the capabilities of the organi
zation, as well as the amounts of the
various proposals. The University further reserves the right to accept or reject any and all proposals and
to waive any irregularities or informalities and to award the contract in the best interest of the University.


Sincerely,



________________________________________________________

_____________________

Authorized Signature








Date


______________________________________________________


Company Name


_____________________________________________


____________________________










Telephone Number

_____________________________________________


____________________________










Fax Number

_____________________________________________


____________________________

Company Address







E
-
mail
address