Design and Implementation of Distributed Supercomputing platformfor ASTU to support scientific research with aggregated Gigaflops

Abstract

Today, distributed supercomputing is a buzzword for the universities and organizations and most of theparallel computing researchers are focused on harnessing the power of commodity processors and eveninternet computers to aggregate their computation powers to solve the computationally complex problems.The advantage of using flexible commodity cluster for supercomputing workloads over a dedicatedsupercomputer as well as expensive HPC infrastructure. The cost effective and scalable nature can betterutilize the available resources of an organization. It can benefit researchers who wish to carry out hugeamounts of repetitive calculations on large amounts of data and wish to obtain valid results in a reasonabletime. This research is a pilot research project which has been commenced with an objective to implementHPC based supercomputing facility in ASTU campus. In this research, the research team has providedtwo separate implementations for cluster based supercomputing- Hadoop and Spark based HPC clusterprimarily for data intensive jobs, while torque based cluster for MIMD workloads. The performance ofthese clusters was measured with an extensive experimentation. However with implementation of MPI theperformance of Spark and Torque clusters shall be compared in full scale implantation of this research.The team of researchers have concluded that the specific application or job can be chosen to run based oncomputation parameters on implemented clusters.