When we hear of Data Analysts or people working in Business Intelligence, there is a veiled sense of awe associated with the complexity of these professions. However, the reality behind these roles is firmly grounded in the principal truth of any real-life Data Analysis pipeline i.e.

Data usually starts off as an unstructured, uncorrelated mountain of mishmash. The prime directive of any data analyst is to first make sense of their data, before any analysis is ventured.

The most powerful tool in the belt of any analyst worth their salt is of data clustering. Today we will take a broad look at the various kinds of clustering and how they can be used in real-world scenarios.

What is clustering?

Clustering is a Machine Learning algorithm and a popular technique for classifying data. It falls under the category of unsupervised machine learning algorithms as it’s useful in dealing with unlabeled and unstructured data.

Clustering is a great way to start making sense out of unstructured data.

In this algorithm we usually deal with only features within data and do not have any target labels or classes. These algorithms discover hidden patterns or data groupings without the need for human intervention. Its ability to discover similarities and differences in information make it the ideal solution for exploratory data analysis.

In other words, Clustering is a data mining technique which classifies datasets based on their similarities or differences. This will process raw data and unclassified data objects into groups represented by structures or patterns in the information. Clustering algorithms can be categorized into a few types, specifically exclusive, overlapping, hierarchical, and probabilistic.

Why Clustering?

To group items that might have same attributes together. It might be helpful to imagine that you have millions of chemical compounds that you cannot see and judge what they are trying to tell, what is similar among them. By clustering you will group those millions of clusters in lets say 5 or 10 clusters based on some similarity among them making it easier for you to analyze those 5 or 10 clusters rather than seeing each compound individually.

Types of Clustering

We can categorize data under various rules and parameters. From simple similarities in data values to comparing relationships between data points, there are a multitude of ways to go about the problem. One way to categorize all the techniques is in the format below.

1. Partition based Clustering

2. Hierarchical Clustering

3. Density-based Clustering

We’ll briefly explain these before moving on to applications.

Partition Based Clustering

Given a database of n objects or data tuples, a partitioning method constructs k partitions of the data, where each partition represents a cluster.

This clustering method classifies the information into multiple groups based on the characteristics and similarity of the data. Its the data analysts to specify the number of clusters that has to be generated for the clustering methods.

In the partitioning method when database(D) that contains multiple(N) objects then the partitioning method constructs user-specified(K) partitions of the data in which each partition represents a cluster and a particular region.

Hierarchical Clustering

Hierarchical clustering, also known as hierarchical cluster analysis, is an algorithm that groups similar objects into groups called clusters. The endpoint is a set of clusters, where each cluster is distinct from each other cluster, and the objects within each cluster are broadly similar to each other.

Hierarchical clustering starts by treating each observation as a separate cluster. Then, it repeatedly executes the following two steps: (1) identify the two clusters that are closest together, and (2) merge the two most similar clusters. This iterative process continues until all the clusters are merged together.

Density-based Clustering

Density-based spatial clustering of applications with noise (DBSCAN) is a well-known data clustering algorithm that is commonly used in data mining and machine learning. DBSCAN groups together points that are close to each other based on a distance measurement (usually Euclidean distance) and a minimum number of points. It also marks as outliers the points that are in low-density regions.

Case Study: K Means Clustering Algorithm

K means is an extremely popular iterative clustering algorithm. It aims to partition an input dataset into subgroups and in this each data point belongs to just one cluster that aims to seek out local maxima in each iteration. This algorithm works in broadly five steps:

Step 0: Find an adequate way to visualize your data. You can pick any 2 or 3 features that are relevant to plot on a graph. We will cluster(partition) our data by segmenting the data as seen in the plot.

Step 1: Choose the number of clusters (k) number of clusters K=3

An example dataset that charts the loan amounts sanctioned to people based on their respective incomes.

Step 2: Select a random centroid(starting value) for each cluster

Step 3: Assign all the points to the closest cluster centroid

Step 4: Keep iterating until there is no changes to the centroid.i.e,Assign each datapoint to the closest cluster

The points in red signify the respective cluster centroids we are iterating over. We have chosen K = 3 as we assume the data will be adequately categorized with 3 categories.

Step 5: Repeat step 3 and 4 iteratively till we reach a stable solution for each of the K cluster centers(the improvement in the calculation of K becomes sufficiently small).

When this difference is 0, we are stopping the training. Let’s now visualize the clusters we have received.

The final result for the location of each of the K(=3) means. The data points are color-coded based on which of the 3 categories they are classified into.

The K-Means Clustering algorithm is conceptually elegant in how our decision of K in any problem will influence how data is classified through this method. We will explore the theory & workings of this algorithm in a latter article.

Applications

Let’s take a look at some impactful ways that Clustering works in tandem with other techniques to improve our everyday lives.

1. Identifying Fake News

Fake news is not a new phenomenon, but it is one that is becoming prolific in our current day and age.

The problem: Fake news is being created and spread at a rapid rate due to technology innovations such as social media. The issue gained attention recently during the 2016 US presidential campaign. During this campaign, the term Fake News was referenced an unprecedented number of times.

How clustering helps: The way that the algorithm works is by taking in the content of the fake news article, the corpus, examining the words used and then clustering them. These clusters are what helps the algorithm determine which pieces are genuine and which are fake news. Certain words are found more commonly in sensationalized, click-bait articles. When you see a high percentage of specific terms in an article, it gives a higher probability of the material being fake news.

2. Identifying fraudulent or criminal activity

In this scenario, we are going to focus on fraudulent taxi driver behavior. However, the technique has been used in multiple scenarios.

The problem: You need to look into fraudulent driving activity. The challenge is how do you identify what is true and which is false?

How clustering helps: By analyzing the GPS logs, the algorithm is able to group similar behaviors. Based on the characteristics of the groups you ar

e then able to classify them into those that are real and which are fraudulent.

3. Marketing and Sales

Personalization and targeting in marketing is big business.

This is achieved by looking at specific characteristics of a person and sharing campaigns with them that have been successful with other similar people.

The problem: If you are a business trying to get the best return on your marketing investment, it is crucial that you target people in the right way. If you get it wrong, you risk not making any sales, or worse, damaging your Customer trust.

How clustering helps: Clustering algorithms are able to group together people with similar traits and likelihood to purchase. Once you have the groups, you can run tests on each group with different marketing copy that will help you better target your messaging to them in the future.

The world of Clustering techniques is vast and we will progressively explore effective methods in future articles.