pig tutorial - apache pig tutorial - Pig Latin Word Count ? - pig latin - apache pig - pig hadoop
Word Count in Pig Latin
Assume we have data in the file like below.
Here we want to generate output for count of each word like below
lets see in steps how to generate the same using Pig latin.
1.Load the data from HDFS
- We can use Load statement to load the data into a relation As keyword used to declare column names, as we dont have any columns, we declared only one column named line.
2. Convert the Sentence into words.
- The data we have is in sentences. So we have to convert that data into words using
If we have any delimeter like space we can specify as
Output will be like this:
but we have to convert it into multiple rows like below
3.Convert Column into Rows
- we have to convert every line of data into multiple rows ,for this we have function called FLATTEN in pig.
- Using FLATTEN function the bag is converted into tuple, means the array of strings converted into multiple rows.
4. Apply GROUP BY
- We have to count each word occurance, for that we have to group all the words.
5. Generate word count
- We can print the word count on console using Dump.
- DUMP wordcount;