ACC and the Exacloud Cluster
Exacloud is a unique computational resource developed in close collaboration with OHSU corporate partner Intel to provide unique capabilities to OHSU researchers for supporting large-scale, computational and data intensive workflows. The primary Exacloud cluster includes over 7,600 Xeon cores with 70 TB of memory distributed across 250 compute nodes. The recent dedication of 8 compute nodes, hosting 32 advanced GPUs (NVIDIA P100s) and increased memory, has proven vital to the fostering the expansion of the machine learning research community at OHSU. The Exacloud architecture includes a Lustre distributed file system, currently with 1.3 Petabytes of usable storage. The cluster is housed at our newly constructed, state-of-the-art Data Center (DCW) at our West Campus. Internally, the cluster is interconnected with a combination of 10 Gb/s copper Ethernet and 56 Gb/s Infiniband. We have additional 56 Gb/s Infiniband to interconnect the Lustre distributed file system with the compute nodes and storage. The Lustre file system assures users of the cluster with high-speed data access even when cluster utilization is high.
More recently, Exacloud has been expanded to support nearly 10,000 computational cores and partitioned to enable an OpenStack virtualization environment (currently comprised of a total of 48 dedicated compute nodes) and a cluster to support clinical genomic workflows for the Knight Cancer Institution (currently made up of 24 dedicated compute nodes).
About 65% of the original system was provided by Intel, which donated all CPUs and the first solid-state disks. Other major donors include the Knight Cancer Institute (25%), HHMI (7%) and OCSSB (3%). The availability of the Exacloud system for research projects is governed by a faculty steering committee, whose membership includes Drs. Joe Gray, Heidi Feiler, Eric Gouaux, Adam Margolin, Paul Spellman, and Shannon McWeeney. No charges are currently associated with use of computational cycles on the cluster or short -term data storage on the Lustre storage.