Abstract—Supercomputing has been indispensable in the unstoppable trend of high-speed computing evolution. This work aims at improving its running efficacy by introducing a new two-step scheduling approach. Based on the analysis of large historical data, we provide an accurate runtime estimation scheme using Instance-Based Learning (IBL) in the first step. Then a swarm intelligence based scheduling (SIBS) method is proposed to optimize the scheduling performance in terms of total runtime makespan and fair resource allocation. A method comparison on a dataset from the ALPS supercomputer, which consists of 804k workload data in 2016, shows that our proposed method outperforms the most commonly used strategy – Extensible Argonne Scheduling System (EASY).
Index Terms—Supercomputer, scheduling, swarm intelligence, instance-based learning, runtime estimation.
F. P. C. Lin was with the Graduate Institute of Communication Engineering, National Taiwan University, Taipei 10617, Taiwan (e-mail: frank555076@gmail.com).
F. K. H. Phoa is with the Institute of Statistical Science, Academia Sinica, Taipei 11529, Taiwan (e-mail: fredphoa@stat.sinica.edu.tw).
Cite: Frank Po-Chen Lin and Frederick Kin Hing Phoa, "Runtime Estimation and Scheduling on Parallel Processing Supercomputers via Instance-Based Learning and Swarm Intelligence," International Journal of Machine Learning and Computing vol. 9, no. 5, pp. 592-598, 2019.
Copyright © 2019 by the authors. This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0).