Think HPC Shops Just Have Supercomputers? Think Again

From supporting AI workloads to providing access to hybrid cloud services, today’s HPC shops are expanding the definition of high performance computing.

For IT shops focused on delivering high performance computing services, the pace of evolution has accelerated in recent years. It’s in high gear as HPC shops have taken on roles and responsibilities that go far beyond the operation of supercomputers for limited numbers of scientists and engineers.

In this new data era, many HPC shops now function as multi-cluster, multi-cloud HPC and AI operations. Versatility is the road here, as HPC shops expand into the domain of the cloud service provider and the general-purpose enterprise IT shop.

Let’s look at some of the characteristics of these next-gen HPC shops — and do some rethinking along the way.

Think that HPC shops don’t virtualize? Think again.

In years past, HPC workloads have run primarily on bare-metal, unvirtualized servers. Today, these practices are changing, as IT leaders are recognizing the benefits of virtualization for even the most demanding HPC systems and applications.

Here’s a case in point: The Johns Hopkins University Applied Physics Laboratory has implemented a virtualized infrastructure for its weapons simulation program. As a VMware white paper explains, with the virtualization of compute-intensive applications on the VMware vSphere® platform, the lab was able to more than triple the average utilization of its hardware, reduce costs due to more effective resource sharing, and run a more diverse set of applications.

In another example, the HPC team at the University of Pisa virtualizes its Microsoft SQL Server environment with the Microsoft Hyper-V hypervisor. As a Dell Technologies case study notes, a virtualized software-defined storage environment makes it easier for the university’s IT Center to deploy, manage and scale storage for the SQL database.

Think that HPC shops don’t have containers? Think again.

Just as they have embraced virtualization, HPC shops are embracing the use of containers that bundle up software applications, virtualized operating systems, and all of the pieces and parts needed to deploy and run HPC and AI jobs. As Dell Technologies data scientist Dr. Lucas Wilson explains in an blog on the advantages of containers, the container approach simplifies the provisioning, distribution and management of the software environments that run on top of the virtualized hardware layer.

Here’s a real-life use case: Data science teams in the Dell Technologies HPC & AI Innovation Lab are leveraging Kubernetes containers to streamline and accelerate the development of deep learning solutions. As data science systems engineering specialist John Lockman explains in blog on the power of Kubernetes, the lab uses Kubernetes containerization to speed up and streamline the production and distribution of deep learning training workloads to thousands of CPU and accelerator nodes in two supercomputing clusters.

Think that HPC shops aren’t cloud service providers? Think again.

The use of OpenStack, virtualization and containerization has helped HPC shops pave the road to hybrid cloud environments. In fact, many HPC shops now function as multi-cloud service providers that offer their users access to internal and external clouds, as well as centralized compute resources with multiple storage choices. Via self-service portals, these next-generation HPC shops streamline the path to infrastructure as a service (IaaS), platform as a service (PaaS), software as a service (SaaS) and managed services.

The Cloud and Interactive Computing (CIC) group at the Texas Advanced Computing Center, for example, operates multiple national-scale production clouds that provide self-service academic cloud computing capabilities. And on top of the lower-level IaaS offerings, the CIC group develops, deploys and administers higher-level cloud and interactive computing platforms, applications and tools.

Elsewhere in the HPC world, the University of Liverpool provides public cloud bursting to  Amazon Web Services. And back in San Diego, the SDSC Research Data Services team offers its users access to both cloud storage and cloud compute options, along with a separate cloud storage system for data that needs to comply with PHI/PII or HIPAA regulations.

Think that HPC has nothing to do with AI. Think again.

HPC and artificial intelligence applications used to live in different domains. Not so anymore. Today’s AI workloads often require the compute performance and storage capacity of HPC clusters.

As I noted in an earlier blog on the convergence trend, HPC shops are in the business of running AI training and inferencing workloads, along with traditional HPC workloads like modeling and simulation. And that makes sense, because these workloads often have similar infrastructure and performance requirements.

Think HPC shops don’t function like a business? Think again.

Whether they are in an enterprise or academic space, many of today’s HPC operations now function like businesses that recover their costs from their users. This is the case for the University of Michigan, where supercomputing investment decisions for campus-wide machines must factor in 100-percent cost recovery, along with exceptional performance, usability and management characteristics. As noted in an article in The Next Platform, we’re talking about an academic supercomputing site that operates under constraints similar to those of any ROI-driven enterprise.

Let’s also consider the Triton Shared Computing Cluster at the San Diego Supercomputer Center (SDSC) at the University of California San Diego. This system is operated under a “condo cluster” model, in which faculty researchers use their contract and grant funding to buy compute nodes for the cluster. Researchers can also rent time on the cluster for temporary and shorter-term needs via a “hotel service” model. Sounds rather business-like, doesn’t it?

Key takeaways

For HPC shops, these are exciting times. We are in a new hybrid world that is blurring the lines between HPC, AI and more conventional IT services. As they navigate this changing world, HPC shops are opening their doors to a wider range of users and offering ever-larger menus of services to make sure people get what they need to do the things they need to do. From modeling complex systems to training machine learning algorithms, from delivering on-premises HPC clusters to enabling access to hybrid cloud services, HPC shops now do it all.

To learn more

For a deeper dive into the changing role of the HPC shop, check out the CIO.com blogs at Dell Technologies and Intel: Innovating to Transform. Learn more about Dell Technologies high performance computing.

About the Author: Janet Morss

Janet Morss previously worked at Dell Technologies, specializing in  machine learning (ML) and high performance computing (HPC) product marketing.