Big data and machine learning framework for clouds and its usage for text classification

Reference architectures for big data and machine learning include not only interconnected building blocks but important considerations (among others) for scalability, manageability and usability issues as well. Leveraging on such reference architectures, the automated deployment of distributed toolsets and frameworks on various clouds is still challenging due to the diversity of technologies and protocols. The paper focuses particularly on the widespread Apache Spark cluster with Jupyter as the particularly addressed framework, and the Occopus cloud-agnostic orchestrator tool for automating its deployment and maintenance stages. The presented approach has been demonstrated and validated with a new, promising text classification application on the Hungarian academic research infrastructure, the OpenStack-based MTA Cloud. The paper explains the concept, the applied components, and illustrates their usage with real use-case measurements.

Automated summary

The paper discusses reference architectures for big data and machine learning, focusing on the application of Apache Spark cluster, Jupyter framework, and Occopus cloud-agnostic orchestrator tool. The approach has been demonstrated and validated through a text classification application on the Hungarian academic research infrastructure.