NCAR-Wyoming Supercomputing Center


The NCAR-Wyoming Supercomputing Center is a high-performance computing and data archival facility located in Cheyenne, Wyoming that provides advanced computing services to researchers in the Earth system sciences.
NWSC provides researchers’ needs for computing, data analysis, and scientific visualization resources combined with powerful data management capabilities to support finer Earth system model resolution, increased model complexity, better statistics, more predictive power, and longer simulation times. The data storage and archival facility at NWSC holds unique historical climate records and a wealth of other scientific data. Scientists at U.S. universities and research institutions access NWSC resources remotely via the Internet from desktop or laptop computers.

History

The NWSC data center is funded by the National Science Foundation and the State of Wyoming, and is operated by the National Center for Atmospheric Research. It was created through a partnership of the University Corporation for Atmospheric Research, the State of Wyoming, the University of Wyoming, Cheyenne LEADS, the Wyoming Business Council, and Cheyenne Light Fuel and Power Company. Consistent with NCAR’s mission, this supercomputing center is a leader in energy efficiency, incorporating the newest and most efficient designs and technologies available. Planning for this data center began in 2003, groundbreaking at the North Range Business Park in Cheyenne took place in June 2010, and computing operations began in October 2012.

Sustainability and energy efficiency

The facility design is based on modular and expandable spaces that can be adapted for computing system upgrades. Its sustainable design makes it 89% more efficient than a typical data center and up to 10% more efficient than state-of-the-art data centers operating in 2010. Almost 92% of the energy it uses goes directly to its core purpose of powering supercomputers to enable scientific discovery. Part of its efficiency comes from the regionally integrated design that uses Wyoming’s climate to provide natural cooling during 96% of the year and local wind energy that supplies at least 10% of its power. The NWSC achieved LEED Gold certification for its sustainable design and construction. In 2013 it won first place for Facility Design Implementation in the Uptime Institute’s Green Enterprise IT awards. This award recognizes pioneering projects and innovations that significantly improve energy productivity and resource use in information technology. In June 2013, the NWSC won the Datacenter Dynamics North American ‘Green’ Data Center award for demonstrated sustainability in the design and operation of facilities.
The center currently has a total of 153,000 square feet with 24,000 square feet of raised floor modules for supercomputing systems in its expandable design. It incorporates numerous resource conservation features to reduce its environmental impact. Water consumption at the NWSC is reduced by about 40% compared to traditional designs by using innovative technologies, specialized cooling tower equipment, and low-flow plumbing fixtures. Waste heat from the systems is recycled to pre-heat components in the power plant, to heat the office spaces, and to melt snow and ice on outdoor walkways and rooftops. Windows supply natural light; combined with room occupancy sensors, the building saves 20-30% in lighting and electricity compared to typical office buildings. A building automation system saves energy by continuously optimizing pumps, fans, and controls that heat or cool only occupied areas of the facility. During construction, sustainable practices were used with emphasis on recycled and locally sourced materials.

Supercomputing resources

for computing, data, and scientific visualization resources are allocated to researchers who investigate the Earth system through simulation. The current HPC environment includes two petascale supercomputers, data analysis and visualization servers, an operational weather forecasting system, an experimental supercomputing architecture platform, a centralized file system, a data storage resource, and an archive of historical research data. All computing and support systems required for scientific workflows are attached to the shared, high-speed, central file system to improve scientific productivity and reduce costs by analyzing and visualizing their data files in place at the NWSC.

Supercomputer: Yellowstone

In 2012, the Yellowstone supercomputer was installed in the NWSC as its inaugural HPC resource. Yellowstone is an IBM iDataPlex cluster consisted of 72,288 Intel Sandy Bridge EP processor cores in 4,518 16-core nodes, each with 32 gigabytes of memory. All nodes are interconnected with a full fat tree Mellanox FDR InfiniBand network. Yellowstone has a peak performance of 1.504 petaflops and has demonstrated a computational capability of 1.2576 petaflops as measured by the High-Performance LINPACK benchmark. It debuted as the world’s 13th fastest computer in the November 2012 ranking by the TOP500 organization. Also in November 2012, Yellowstone debuted as the 58th most energy efficient supercomputer in the world by operating at 875.34 megaflops per watt as ranked by the Green500 organization. Yellowstone is expected to remain in production operation through the end of 2017.

Supercomputer: Cheyenne

Becoming operational in 2017, the 5.34-petaflops Cheyenne supercomputer is currently providing more than three times the computational capacity of Yellowstone. Cheyenne is an SGI ICE XA system with 4,032 dual-socket scientific computation nodes running 145,152, 18-core 2.3-GHz Intel Xeon E5-2697v4 processing cores and has 315 terabytes of memory. Interconnecting these nodes is a Mellanox EDR InfiniBand network with 9-D enhanced hypercube topology that performs with a latency of only 0.5 microsecond. Cheyenne runs the SUSE Linux Enterprise Server 12 SP1 operating system. Similar to Yellowstone, Cheyenne’s design and configuration will provide balanced I/O and exceptional computational capacity for the data-intensive needs of its user community. Cheyenne debuted as the world's 20th most powerful computer in the November 2016 Top500 ranking.

Data analysis and visualization clusters: Geyser and Caldera

The Geyser and Caldera clusters are specialized data analysis and visualization resources within the data-centric Yellowstone environment. The Geyser data analysis server is a 640-core cluster of 16 nodes, each with 1 terabyte of memory. With its large per-node memory, Geyser is designed to facilitate large-scale data analysis and post-processing tasks, including 3D visualization, with applications that do not support distributed-memory parallelism.
The Caldera computational cluster has 256 cores in 16 nodes, each with 64 gigabytes of memory and two Graphics Processing Units for use as either computational processors or graphics accelerators. Caldera’s two NVIDIA Tesla GPUs per node support parallel processing, visualization activities, and development and testing of general-purpose GPU code.

Operational forecasting system for Antarctic weather: Erebus

The center also houses a separate, smaller IBM iDataPlex cluster named Erebus to support the operational forecasts of the NSF Office of Polar Programs’ Antarctic Mesoscale Prediction System. Erebus has 84 nodes similar to Yellowstone’s, an FDR-10 InfiniBand interconnect, and a dedicated 58-terabyte file system. If needed, Yellowstone will run Erebus’ daily weather forecasts for the Antarctic continent to ensure that the worldwide community of users receives these forecasts without interruption.

Experimental supercomputing architecture platform: Pronghorn

Pronghorn’s architecture has promise for meeting the Earth system sciences’ demanding requirements for data analysis, visualization, and GPU-assisted computation. As part of a partnership between Intel, IBM, and NCAR, this exploratory system is being used to evaluate the effectiveness of the Xeon Phi coprocessor’s Many Integrated Core architecture for running climate, weather, and other environmental applications. If these coprocessors prove beneficial to key NCAR applications, they can be easily added to the standard IBM iDataPlex nodes in Yellowstone as a cost-effective way to extend its capabilities.
Pronghorn has 16 dual-socket IBM x360 nodes featuring Intel’s Xeon Phi 5110P coprocessors and 2.6-gigahertz Intel Sandy Bridge cores. The system has 64 gigabytes of DDR3-1600 memory per node and is interconnected with a full fat tree Mellanox FDR InfiniBand network.

Centralized file system: GLADE

Geyser, Caldera, and Yellowstone all mount the central file system named GLobally Accessible Data Environment, which provides work spaces common to all HPC resources at NWSC for computation, analysis, and visualization. This allows users to analyze data files in place, without sending large amounts of data across a network or creating duplicate copies in multiple locations. GLADE provides centralized high-performance file systems spanning supercomputing, data post-processing, data analysis, visualization, and HPC-based data transfer services.
GLADE also hosts data from NCAR’s Research Data Archive, NCAR’s Community Data Portal, and the Earth System Grid that curates CMIP5/AR5 data. The GLADE central disk resource has a usable storage capacity of 36 petabytes as of February 2017. GLADE has a sustainable aggregate I/O bandwidth of more than 220 gigabits per second.

Data archival service: HPSS

Archival data storage at the NWSC is provided by a High Performance Storage System that consists of tape libraries with storage capacity of 320 petabytes. These scalable, robotic systems consist of six Oracle StorageTek SL8500 tape libraries using T10000C tape drives with an I/O rate of 240 megabits per second.

Climate data archive: RDA

NWSC’s data-intensive computing strategy includes a full suite of community data services. NCAR develops data products and services that address the future challenges of data growth, preservation, and management. The Research Data Archive contains a large collection of meteorological and oceanographic datasets that support scientific studies in climate, weather, hydrology, Earth system modeling, and other related sciences. It is an open resource; the global research community also uses it.

Educational projects

The NWSC also serves an educational role. Its public outreach program features the NWSC visitor center that explains the science goals and the technology of NCAR and the University of Wyoming.
NCAR's higher education internship program places two engineering interns at the NWSC each summer.
The facilities at NWSC are being used in a research collaboration with Colorado State University, Oak Ridge National Laboratory, Lagrange Systems, and NCAR to produce resilient resource management strategies for HPC environments, increase the number of researchers and scientific problems that can use HPC, and help achieve sustainable computing at extreme scales within realistic power budgets.