This is one of the most exciting posts I’ve written in my 15+ years of existence in the High Performance Computing (HPC) industry.
AMD EPYC Zen 3 “Milan” is integrated, tested and available on the HPCBOX platform on Microsoft Azure, on LAUNCH DAY! This is something which seldom happens in the HPC world, very few users ever get to start production on day one of the release of a new and very impressive processor generation upgrade. I will not be doing any technical comparison between Zen 3 and Zen 2 and neither will I present any specific application benchmark numbers in this post. I am sure there’ll be many posts by AMD, Microsoft and other ISVs who’ll be publicly sharing information on the performance boost they see for their codes with EPYC Zen 3 “Milan”. This post will be more about HPCBOX and how we could deliver this upgrade experience for our users, on launch day and without having to set our hands on the physical hardware or physically be present in a datacenter(s) (actually multiple regions)! Awesome work and support from the Azure team! Drizti was a launch partner for the new HBv3 instance size on Microsoft Azure and these instances are powered by the new EPYC 7xx3/Milan CPUs, HDR InfiniBand and sport very impressive dual NVMe drives which give a big performance boost to applications which use local scratch, specifically when they are striped. Read more about HBv3 here Unbelievably quick CPU generation upgrade
It all started with Drizti getting access to 1000+ cores of HBv3 instances for functional and compatibility testing, to make sure we are ready for GA availability of HBv3 on launch day of AMD EPYC ”Milan” CPUs. We went through testing, adding necessary support within the HPCBOX platform to make sure we are able to use the cool new features available on the instances, like automate striping of NVMe, testing out MPI compatibility, testing the workflow component of HPCBOX, auto-scaling/shutdown/start etc. and all other functions which are offered by the HPCBOX platform. In addition to this, we also did some performance tests with different ISV and open-source codes, mainly to make sure the HPCBOX workflow engine can correctly handle the new instances and automatically optimize application pipelines to take advantage of the new hardware.
Some of the applications we tested were ANSYS CFX, ANSYS Fluent, OpenFOAM. At a high level, we can share that we are seeing impressive performance benefits of using EPYC “Milan”, particularly for large runs. Also, for applications which are local scratch bound, we expect users to get a good boost due to the possibility of having striped NVMes in HBv3. We also did a test to see how easy it would be to upgrade HPC for our users and were really impressed that we could just switch our users from HBv2 to HBv3, meaning from “Rome” to “Milan” and to upgraded local scratch in under 30 minutes! This is mainly because of the design of the HPCBOX platform, it is a self-contained platform, and this gives us the ability to fine tune platform capabilities quickly without depending on external services to get upgraded first. Agile and Impressive! In Action
|
Overview This article discusses using ANSYS products on the HPCBOX™, HPC Cloud Platform and the benefits provided by the HPCBOX Platform for ANSYS users. |
|
Introduction
HPCBOX delivers an innovative desktop-centric workflow enabled platform for users who want to accelerate innovation using cutting edge technologies that can benefit from High Performance Computing Infrastructure. It aims to make supercomputing technology accessible and easier to use for scientists, engineers and other innovators who do not have access to in-house datacenters or skills to create, manage and maintain complex HPC infrastructure. To achieve a high level of simplicity in delivering a productive HPC platform, HPCBOX leverages unique technology that not only allows users to create workflows combining their applications and optimization of underlying infrastructure into the same pipeline, but, also provides integration points to help third-party applications directly hook into its infrastructure optimization capability. Some of the unique capabilities of HPCBOX are:
HPCBOX delivers an innovative desktop-centric workflow enabled platform for users who want to accelerate innovation using cutting edge technologies that can benefit from High Performance Computing Infrastructure. It aims to make supercomputing technology accessible and easier to use for scientists, engineers and other innovators who do not have access to in-house datacenters or skills to create, manage and maintain complex HPC infrastructure. To achieve a high level of simplicity in delivering a productive HPC platform, HPCBOX leverages unique technology that not only allows users to create workflows combining their applications and optimization of underlying infrastructure into the same pipeline, but, also provides integration points to help third-party applications directly hook into its infrastructure optimization capability. Some of the unique capabilities of HPCBOX are:
- Elastic scale-in and scale-out of HPC clusters
- Hardware accelerated remote 3D for visualization
- Rich desktop experience
- Secure access to resources
- Individual HPC clusters for every user
- Intelligent workflow capability
Configuring ANSYS licenses
HPCBOX provides various graphical tools for the end user to configure ANSYS licenses themselves. Easy to use tools let users point their HPC cluster to an existing license server within their organization, or, install a license provided by ANSYS on the cluster itself. All network configuration required for the license server on the cluster are automated and performed with the click of a button.
HPCBOX provides various graphical tools for the end user to configure ANSYS licenses themselves. Easy to use tools let users point their HPC cluster to an existing license server within their organization, or, install a license provided by ANSYS on the cluster itself. All network configuration required for the license server on the cluster are automated and performed with the click of a button.
Running ANSYS Fluent
ANSYS Fluent — is the most powerful computational fluid dynamics (CFD) software available — it’ll optimize your product’s performance faster. Fluent includes well-validated physical modeling capabilities that deliver reliable and accurate results across the widest range of CFD and multiphysics applications.
Fluent also offers highly scalable high-performance computing (HPC) to help solve complex, large-model CFD simulations quickly and cost-effectively.
Fluent can efficiently scale when run in distributed parallel mode and allows users to work on their models with its rich GUI interface along with providing batch execution capability.
HPCBOX Cluster for ANSYS is delivered pre-installed and integrated with the latest versions of ANSYS (19.0.0, 19.1.0 and 2019R1 at the time of writing this article). HPCBOX’s rich desktop experience allows users to execute Fluent simulations with both its native GUI and in batch mode, as preferred by the user. HPCBOX’s elastic scale-in/scale-out capability lets users control the underlying HPC infrastructure themselves, with no support required from HPC administrators. Furthermore, this can be done without writing a single line of code or executing a single console command.
ANSYS Fluent — is the most powerful computational fluid dynamics (CFD) software available — it’ll optimize your product’s performance faster. Fluent includes well-validated physical modeling capabilities that deliver reliable and accurate results across the widest range of CFD and multiphysics applications.
Fluent also offers highly scalable high-performance computing (HPC) to help solve complex, large-model CFD simulations quickly and cost-effectively.
Fluent can efficiently scale when run in distributed parallel mode and allows users to work on their models with its rich GUI interface along with providing batch execution capability.
HPCBOX Cluster for ANSYS is delivered pre-installed and integrated with the latest versions of ANSYS (19.0.0, 19.1.0 and 2019R1 at the time of writing this article). HPCBOX’s rich desktop experience allows users to execute Fluent simulations with both its native GUI and in batch mode, as preferred by the user. HPCBOX’s elastic scale-in/scale-out capability lets users control the underlying HPC infrastructure themselves, with no support required from HPC administrators. Furthermore, this can be done without writing a single line of code or executing a single console command.
Workflow with HPCBOX CLUSTER APP
HPCBOX provides a rich GUI application called hpcbox-cluster-app to create and execute workflows combining infrastructure control and application execution.
HPCBOX provides a rich GUI application called hpcbox-cluster-app to create and execute workflows combining infrastructure control and application execution.
In the first step of the workflow shown above, we scale out the cluster by adding 8 nodes with 16 cores each. Each node has the following properties:
Intel E5-2667 V3 3.2 GHz (with turbo up to 3.5 GHz)
112 GB DDR4 RAM
Interconnect: FDR Infiniband
Intel E5-2667 V3 3.2 GHz (with turbo up to 3.5 GHz)
112 GB DDR4 RAM
Interconnect: FDR Infiniband
In the second workflow step we start ANSYS Fluent on 128 cores.Once the workflow is executed, HPCBOX will start 8 nodes and once the nodes are ready, automatically start Fluent with its GUI.
In the final Workflow step above, once the Fluent application has completed execution, HPCBOX will scale-in the cluster by deleting the compute nodes that were added in Step-1.
HPCBOX Connector to ANSYS Workbench
ANSYS Workbench is ANSYS’s Integrated Engineering Simulation Platform. It combines the strength of ANSYS’s simulation tools with the tools necessary to manage user projects and create simulation workflows combining different ANSYS applications.
HPCBOX Connector
The HPCBOX Connector for ANSYS Workbench is a custom component system which can be loaded directly into ANSYS Workbench and delivers HPCBOX’s cluster optimization and elasticity technology directly in Workbench. The connector brings close integration between the underlying HPC infrastructure and the user workflows created in Workbench.
ANSYS Workbench is ANSYS’s Integrated Engineering Simulation Platform. It combines the strength of ANSYS’s simulation tools with the tools necessary to manage user projects and create simulation workflows combining different ANSYS applications.
HPCBOX Connector
The HPCBOX Connector for ANSYS Workbench is a custom component system which can be loaded directly into ANSYS Workbench and delivers HPCBOX’s cluster optimization and elasticity technology directly in Workbench. The connector brings close integration between the underlying HPC infrastructure and the user workflows created in Workbench.
In the above example workflow, we see an ANSYS Workbench workflow which scales-out the HPC cluster, runs the CFX solver and scales-in the cluster once the solver execution is complete.
Visualization on HPCBOX
HPCBOX delivers hardware accelerated remote 3D utilizing NVIDIA GPUs. Applications such as, DesignModeler, ANSYS Meshing, CFD-Post can offload 3D rendering to NVIDIA graphics cards and users can perform their pre/post processing tasks directly on the cluster, helping eliminate much of the data transfer operations.
HPCBOX delivers hardware accelerated remote 3D utilizing NVIDIA GPUs. Applications such as, DesignModeler, ANSYS Meshing, CFD-Post can offload 3D rendering to NVIDIA graphics cards and users can perform their pre/post processing tasks directly on the cluster, helping eliminate much of the data transfer operations.
Conclusion
HPCBOX delivers a highly personalized, rich HPC experience for users of ANSYS applications. The platform is suitable for both small and large enterprises alike; engineers from SMEs who do not have access to a HPC resource or skilled HPC personnel in-house can benefit from a self-serviced user experience for HPC and engage Drizti’s HPC support and skills to augment their innovation with ANSYS and HPC. Larger enterprises can make use of HPCBOX to tap into additional HPC capacity in times of peak need when time to delivery is critical and procuring, provisioning and setting up additional HPC resources would contribute to delays in project deliveries.
HPCBOX delivers a highly personalized, rich HPC experience for users of ANSYS applications. The platform is suitable for both small and large enterprises alike; engineers from SMEs who do not have access to a HPC resource or skilled HPC personnel in-house can benefit from a self-serviced user experience for HPC and engage Drizti’s HPC support and skills to augment their innovation with ANSYS and HPC. Larger enterprises can make use of HPCBOX to tap into additional HPC capacity in times of peak need when time to delivery is critical and procuring, provisioning and setting up additional HPC resources would contribute to delays in project deliveries.
Getting Access and Trial
Please contact Drizti to get a demo and trial of the HPCBOX Platform for ANSYS applications. We'd also like to get any feedback you might have for us.
Please contact Drizti to get a demo and trial of the HPCBOX Platform for ANSYS applications. We'd also like to get any feedback you might have for us.
All third-party product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation or endorsement by them.