[Solved-1 Solution] Generate multiple outputs with Hadoop Pig ?


How to generate Multiple Output for multiple files loaded in PIG ?

Solution 1:

Why we need split ?

  • Split operator can be used to partition the contents of a relation into two or more relations based on some expression. Based on the conditions provided in the expression either of the below two will be done:
    • A tuple may be assigned to more than one relation
    • A tuple may not be assigned to any relation

Multiple files in a directory which are used in pig to load, flatten and store:

[[email protected] ~]# ls /pigsamples/mfilesdata/
file1  file2  file3

Loading above directory:

grunt> input_data = LOAD '/pigsamples/mfilesdata' USING PigStorage (',') AS (f1:INT, f2:INT, f3:INT);
grunt> DUMP input_data;

Format the data based on your requirements

grunt> formatted_data = FOREACH input_data GENERATE FLATTEN(TOKENIZE($0));    //replace with your requirements

Use SPLIT operator to split the relation into multiple relations based on the conditions.

SPLIT formatted_data 
INTO split1 IF f1 <= 3, 
split2 IF (f1 > 3 AND f1 <= 6), 
split3 IF f1 > 6;       //split based on the column which is unique within all the files


grunt> DUMP split1;

grunt> DUMP split2;

Related Searches to Generate multiple outputs with Hadoop Pig