期刊
CONCURRENCY AND COMPUTATION-PRACTICE & EXPERIENCE
卷 27, 期 17, 页码 5037-5059出版社
WILEY
DOI: 10.1002/cpe.3505
关键词
scientific workflows; high-throughput computing; fault-tolerant computing
资金
- Materials Project (DOE Basic Energy Sciences) [EDCBEE]
- Joint Center for Energy Storage Research (JCESR), an Energy Innovation Hub - US Department of Energy, Office of Science, Basic Energy Sciences
- Universite catholique de Louvain through Fonds d'appui a l'internationalisation
- European Union Marie Curie Career Integration (CIG) grant HTforTCOs [PCIG11-GA-2012-321988]
- Office of Science of US Department of Energy [DE-AC02-05CH11231]
This paper introduces FireWorks, a workflow software for running high-throughput calculation workflows at supercomputing centers. FireWorks has been used to complete over 50 million CPU-hours worth of computational chemistry and materials science calculations at the National Energy Research Supercomputing Center. It has been designed to serve the demanding high-throughput computing needs of these applications, with extensive support for (i) concurrent execution through job packing, (ii) failure detection and correction, (iii) provenance and reporting for long-running projects, (iv) automated duplicate detection, and (v) dynamic workflows (i.e., modifying the workflow graph during runtime). We have found that these features are highly relevant to enabling modern data-driven and high-throughput science applications, and we discuss our implementation strategy that rests on Python and NoSQL databases (MongoDB). Finally, we present performance data and limitations of our approach along with planned future work. Copyright (c) 2015 John Wiley & Sons, Ltd.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据