Clustering of Binary Data in a Distributed Computing Environment

The scope of master thesis is to identify model investment portfolios within large datasets, where the dimensions of the datasets correspond to tens of millions of records and thousands of descriptive attributes. One of the most suitable tools for large-scale data processing is currently Apache Spark, which provides an interface for programming entire clusters with implicit data parallelism. The task of identifying investment model portfolios can be understood as the task of data clustering. To find a suitable algorithm, the research of accessible implementations of clustering algorithms for Apache Spark platform was concluded, where any of suitable implementation for this task was not found. This master thesis puts forward a proposal and practical implementation of methods for clustering large datasets, which are described by binary variables in the Scala language and that are applicable to parallel processing using Apache Spark platform. The most significant characteristic of this method is no need to specify the number of searched clusters. The proposed method of clustering was first evaluated based on a synthetic dataset and subsequently used with minor modifications to solve the real use-case of identifying investment model portfolios. The results of clustering, which was based on the proposed method were in line with expectations according to the domain knowledge of client. It is expected that the proposed method will be soon deployed for the production use in the client’s company.