Music Analytics Opportunities

Once upon a time the words music listening habits were private as their bedrooms, music lovers used to buy the CDs, recordings and other physical copies of music and never publicly shared, Record companies were aware which radio station played their songs and where their CDs were popular, but that information painted an incomplete picture at best. Who knew what music people were sharing on tapes and CDs burnt in the privacy of their own bedrooms?

A traditional business metrics like number of CDs were sold and nothing happened after that, who purchased what and whom to assist what to buy, all this was anonymous. Thats all changed after explosion of online music sources like torrenting, music streaming sites and social media platforms, are now playing a very key role for music industry to understand their fans, spot upcoming talents like never before and anyones personal music interest nowadays becoming a public. Music analytics is now worth around $24.35 billion per year.

At the same time that the internet is taking power away from record labels, it is also giving them the ability to predict future hits.

Recommendations with Apache Mahout

Recommendation?

Have you ever been recommended a friend on Facebook? Or visited a shopping portal where you can see the recommended items for you, Or an item you might be interested in on Amazon? If so then you've benefited from the value of recommendation systems.

for example, often see personalized recommendations phrased something like, “If you liked that item, you might like also like this one...” These sites use recommendations to help drive users  to other things they offer in an intelligent, meaningful way, tailored specifically to the user and the user’s preferences.

Recommendation systems apply knowledge discovery techniques to the problem of making recommendations that are personalized for each user. Recommendation systems are one way we can use algorithms to help us sort through the masses of information to find the “good stuff” in a very managed way.

From an algorithmic standpoint, the recommendation systems we’ll talk about today are considered in the k-nearest neighbor family of problems (another type would be a SVD-based recommender). We want to predict the estimated preference of a user towards an item they have never seen before. We also want to generate a ranked (by preference score) list of items the user might be most interested in. Two well-known styles of recommendation algorithms are item-based recommenders and user-based recommenders. Both types rely on the concept of a similarity function/metric (ex: Euclidean distance, log likelihood), whether it is for users or items.

Overview of a recommendation engine

The main purpose of a recommendation engine is to make inferences on existing data to show relationships between objects and entities. Objects can be many things, including users, items, products(in short user related data) and so on. Relationships provide a degree of likeness or belonging between objects. For example, relationships can represent ratings of how much a user likes an item, or indicate if a user bookmarked a particular page.

To make a recommendation, recommendation engines perform several steps to mine the data(Data mining). Initially, you begin with input data that represents the objects as well as their relationships. Input data consists of object identifiers and the relationships to other objects.

Consider the ratings users give to items. Using this input data, a recommendation engine computes a similarity between objects. Computing the similarity between objects(co-similarity) can take a great deal of time depending on the size of the data or the particular algorithm. Distributed algorithms such as Apache Hadoop using Mahout can be used to parallelize the computation of the similarities. There are different types of algorithms to compute similarities. Finally, using the similarity information, the recommendation engine can make recommendation requests based on the parameters requested.

For Example:

GroupLens Movie Data

The input data for this demo is based on 1M anonymous ratings of approximately 4000 movies made by 6,040 MovieLens users, which you can download from the www.grouplens.org site. The zip file contains four files:

movies.dat (movie ids with title and category)

ratings.dat (ratings of movies)

README

users.dat (user information)

The ratings file is most interesting to us since it’s the main input to our recommendation job. Each line has the format:

Ratings.dat description

UserID::MovieID::Rating::Timestamp

So let’s adjust our input file to match what we need to run our job. First download the file and unzip it locally from:

Next run the command:

        tr -s ':' ',' < ratings.dat | cut -f1-3 -d, > ratings.csv

This produces the csv output format we’ll use in the next section when we run our “Itembased Collaborative Filtering” job.

        hadoop fs -put [my_local_file] [user_file_location_in_hdfs]

this command put  input file on HDFS,

create user.txt file which stores the data(userID) of the users to which we want show recommendations.

put it on HDFS under users directory.

With our user list in hdfs we can now run the Mahout  recommendation job with a command in the form of:

     

       mahout recommenditembased --input [input-hdfs-path] --output [output-hdfs-path] --tempDir [tmp-hdfs-path] --usersFile [user_file_location_in_hdfs]

which will run for a while (a chain of 10 MapReduce jobs) and then write out the item recommendations into HDFS we can now take a look at.  If we tail the output from the RecommenderJob with the command:

         hadoop fs -cat [output-hdfs-path]/part-r-00000

The output will show the user(provided into user.txt) with the recommended items.

Friend Recommender In MapReduce

Hello Guys, today MapReduce is becoming a very popular framework for designing a data processing system for application has huge amount of data inshort #Bigdata. The main reason behind the popularity of MapReduce is the Scalability. You can easily carry out the very complex data processing through a huge amount of data in very short span of time(Nearly real time), unlike the traditional data processing systems takes hours to process it.

Here I wanna discuss a very popular use case of bigdata processing is the Friend Recommendations or you may name it as artifact recommendation

Here is the problem.
How to find out the Nth degree mutual friend from given list of friends like
A is direct friend of B and B is direct friend of C then C is the 2nd degree mutual friend of A.
below is the input(userid and their direct friends userid)

5101,5102
5102,5104
5102,5105
5103,5106
5101,5106
5106,5107
5105,5107
5104,5102

In the first phase MapReduce will findout the group of friends by user, in Map phase produces the Mapping of 2xN and reduce will reduce it to N with group of friends by user.

Mapper:

public static class Map extends Mapper<Longwritable,Text, Text, Text> {

  @Override
  public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
   String line[] = value.toString().split("\\t");
   String fromUser = line[0].trim();

   if (line.length == 2) {
    String toUser = line[1].trim();
    context.write(new Text(toUser), new Text(fromUser));
    context.write(new Text(fromUser),new Text(toUser));
   }else{
    context.write(new Text(fromUser),null);
   }
  }
 }

Reducer:

public static class Reduce extends Reducer<Text,Text, Text, Text> {
  @Override
  public void reduce(Text key, Iterable<text> values, Context context)
    throws IOException, InterruptedException {

   ArrayList<string> userEntryList = new ArrayList<>();
   Iterator<text> friends = values.iterator();

   while(friends.hasNext()){
    Text e = friends.next();
    if(e!=null){
     userEntryList.add(String.valueOf(e.toString()));
    }
   }
   context.write(key, new Text(userEntryList.toString()));
  }
 }

And the output will be generated

5101   [5102, 5106]
5102   [5104, 5105, 5101, 5104]
5103   [5106]
5104   [5102, 5102]
5105   [5102]
5106   [5107, 5103, 5101]
5107   [5106]

Now you need to find out the 2nd degree friends like friends of each friend
In Map Phase, Emit the <touser1, r=touser2,m=fromuser>, here touser1 is current user, r means recommended friend and m means mutual friend. Like A is friend of B and B of C, then we can recommend C to A though mutual friend B, means here  above formula becomes<touser1=A,r=touser2=C,m=fromuser=B>. It will emit n(n-1) records Totally there are n^2 records emitted though map phase. In reduce phase we just sum the how many friend will be there for current user and key.

As emitted value is not primitive type in hadoop, so we can create our own datatype

static public class FriendCount implements Writable {
  public Long user;
  public Long mutualFriend;

  public FriendCount(Long user, Long mutualFriend) {
   this.user = user;
   this.mutualFriend = mutualFriend;
  }

  public FriendCount() {
   this(-1L, -1L);
  }

  @Override
  public void write(DataOutput out) throws IOException {
   out.writeLong(user);
   out.writeLong(mutualFriend);
  }

  @Override
  public void readFields(DataInput in) throws IOException {
   user = in.readLong();
   mutualFriend = in.readLong();
  }

  @Override
  public String toString() {
   return " toUser: "
     + Long.toString(user) + " mutualFriend: " + Long.toString(mutualFriend);
  }
 }

Map and Reduce can be implemented by

public static class Map extends Mapper<LongWritable, Text, LongWritable, FriendCount> {
  private Text word = new Text();

  @Override
  public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
   String line[] = value.toString().split("\\t");
   Long fromUser = Long.parseLong(line[0]);
   List<Long> toUsers = new ArrayList<Long>();

   if (line.length == 2) {
    StringTokenizer tokenizer = new StringTokenizer(line[1], ",");
    while (tokenizer.hasMoreTokens()) {
     Long toUser = Long.parseLong(tokenizer.nextToken().replace("[", "").replace("]", "").trim());
     toUsers.add(toUser);
     context.write(new LongWritable(fromUser), new FriendCount(toUser, -1L));
    }

    for (int i = 0; i < toUsers.size(); i++) {
     for (int j = i + 1; j < toUsers.size(); j++) {
      context.write(new LongWritable(toUsers.get(i)), new FriendCount((toUsers.get(j)), fromUser));
      context.write(new LongWritable(toUsers.get(j)), new FriendCount((toUsers.get(i)), fromUser));
     }
    }
   }
  }
 }

 public static class Reduce extends Reducer<LongWritable, FriendCount, LongWritable, Text> {
  @Override
  public void reduce(LongWritable key, Iterable<FriendCount> values, Context context)
    throws IOException, InterruptedException {

   final java.util.Map<Long, Set<Long>> mutualFriends = new HashMap<Long, Set<Long>>();

   for (FriendCount val : values) {
    final Boolean isAlreadyFriend = (val.mutualFriend == -1);
    final Long toUser = val.user;
    final Long mutualFriend = val.mutualFriend;

    if (mutualFriends.containsKey(toUser)) {
     if (isAlreadyFriend) {
      mutualFriends.put(toUser, null);
     } else if (mutualFriends.get(toUser) != null) {
      mutualFriends.get(toUser).add(mutualFriend);
     }
    } else {
     if (!isAlreadyFriend) {
      mutualFriends.put(toUser, new HashSet<Long>() {
       {
        add(mutualFriend);
       }
      });
     } else {
      mutualFriends.put(toUser, null);
     }
    }
   }

   java.util.SortedMap<Long, Set<Long>> sortedMutualFriends = new TreeMap<Long, Set<Long>>(new Comparator<Long>() {
    @Override
    public int compare(Long key1, Long key2) {
     Integer v1 = mutualFriends.get(key1).size();
     Integer v2 = mutualFriends.get(key2).size();
     if (v1 > v2) {
      return -1;
     } else if (v1.equals(v2) && key1 < key2) {
      return -1;
     } else {
      return 1;
     }
    }
   });

   for (java.util.Map.Entry<Long, Set<Long>> entry : mutualFriends.entrySet()) {
    if (entry.getValue() != null) {
     sortedMutualFriends.put(entry.getKey(), entry.getValue());
    }
   }

   Integer i = 0;
         String output = "";
         Set<Long> entrySet = new HashSet<>();
   for (java.util.Map.Entry<Long, Set<Long>> entry : sortedMutualFriends.entrySet()) {
    entrySet.add(entry.getKey());
             entrySet.addAll(entry.getValue());            
   }
   Iterator<Long> setItr = entrySet.iterator();
   while(setItr.hasNext()){
    if(i==0)
     output+=setItr.next();
    else
     output+="\t"+setItr.next();
    
    ++i;
   }

  context.write(key, new Text(output));
 }

Final Output you can see like first is the current user id and against you can see the direct friends with recommended friends

[5101, 5102, 5106, 5104, 5105, 5107, 5103]
[5102, 5104, 5105, 5101, 5104, 5106]
[5103, 5106, 5107, 5101]
[5104, 5102, 5102, 5105, 5101]
[5105, 5102, 5104, 5101]
[5106, 5107, 5103, 5101, 5102]
[5107, 5106, 5103, 5101]

You can implement the same code in simple java programmer without using MapReduce framework, it works well but not much scalable as MapReduce, You can find below the Normal JAVA code to find out the recommended friends might help you to design MapReduce

package com.java.amolfasale;

import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;

@SuppressWarnings("serial")
public class FriendRecommendationWithoutMapReduce extends TreeMap<String, List<String>> {

 //Overriding put method to append friends of same user
 public void put(String key, String number) {
  List<String> current = get(key);
  if (current == null) {
   current = new ArrayList<String>();
   super.put(key, current);
  }
  current.add(number);
 }

 @SuppressWarnings("rawtypes")
 public static void main(String[] args) {
 
  FriendRecommendationWithoutMapReduce user = new FriendRecommendationWithoutMapReduce();
  //Putting all values in map
  user.put("5101", "5102");
  user.put("5102", "5104");
  user.put("5102", "5105");
  user.put("5103", "5106");
  user.put("5101", "5106");
  user.put("5106", "5107");
  user.put("5104", "5102");
  
  // Putting the same value in reverse
  user.put("5102","5101");
  user.put("5104", "5102");
  user.put("5105", "5102");
  user.put("5106", "5103");
  user.put("5106", "5101");
  user.put("5107", "5106");
  user.put("5102", "5104");
  
  System.out.println("\n___________________Group By Friends__________________________\n");
  
  ArrayList<String> userEntryList = new ArrayList<>();

  // For N=2
  for (Map.Entry e : user.entrySet()) {
   System.out.println(e.getKey() + "    " + e.getValue());
   userEntryList.add(String.valueOf(e.getKey()));
  }

  System.out.println("\n___________________Final Output__________________________\n");
  // For Rest Case
  for (int i = 0; i <= userEntryList.size() - 1; i++) {
   List<String> output = new ArrayList<>();
   output.add(userEntryList.get(i));

   // Get All 2nd degree Related Friend of User i
   List<String> friends = user.get(userEntryList.get(i));
   output.addAll(friends);
   
   for (int j = 0; j < friends.size(); j++) {
    List<String> aList = new ArrayList<>();
    aList.addAll(user.get(friends.get(j)));
    for (int k = 0; k < aList.size(); k++) {
     if(!output.contains(aList.get(k))){
      output.add(aList.get(k));
     }
    }
   }
   System.out.println(output.toString());
  }
  System.out.println("\n___________________End Final Output__________________________\n");
 }
}