Alchemist is an interface between Apache Spark applications and MPI-based libraries for accelerating large-scale linear algebra and machine learning computations.

Performing communication-intense linear algebra computations in Spark can incur large overheads. Alchemist bypasses these overheads by sending the data from the Spark application to existing or custom MPI-based libraries, and then transmitting the results back to the application. This leads to significantly fewer overheads and to computations that are efficient and scalable.

For the current version of the code, see https://github.com/alexgittens/alchemist

A new version will be released soon!