How do you force the number of reducers in a map reduce job - Wikitechy https://www.wikitechy.com/interview-questions/tag/how-do-you-force-the-number-of-reducers-in-a-map-reduce-job/ Interview Questions Tue, 14 Sep 2021 12:15:41 +0000 en-US hourly 1 https://wordpress.org/?v=6.9.4 https://www.wikitechy.com/interview-questions/wp-content/uploads/2025/10/cropped-wikitechy-icon-32x32.png How do you force the number of reducers in a map reduce job - Wikitechy https://www.wikitechy.com/interview-questions/tag/how-do-you-force-the-number-of-reducers-in-a-map-reduce-job/ 32 32 What is default numbers of reducers while executing a pig query ? https://www.wikitechy.com/interview-questions/apache-pig/what-is-default-numbers-of-reducers-while-executing-a-pig-query/ https://www.wikitechy.com/interview-questions/apache-pig/what-is-default-numbers-of-reducers-while-executing-a-pig-query/#respond Mon, 12 Jul 2021 07:14:15 +0000 https://www.wikitechy.com/interview-questions/?p=192

Set Number of Reducer in Pig:

SET default_parallel XXX
  • Where XXX is the number of reducer.
  • This command is used to set the number of reducers at the script level
  • The coder need to write this configuration at top/beginning of their pig script.
  • Alternatively, use the PARALLEL clause to set the number of reducers at the operator level.
  • We set the value using the PARALLEL clause will override any value we specify through (“SET default parallel.”) to reduce phase you can include the PARALLEL clause with any operator.
    • COGROUP
    • CROSS
    • DISTINCT
    • GROUP
    • JOIN (inner)
    • JOIN (outer) and
    • ORDER BY.

For Example:

In GROUP operator the PARALLEL class has been used.

  • A = LOAD ‘myfile’ AS (t, u, v);
  • B = GROUP A BY t PARALLEL 18

Here 18 is number of reducer.

  • If neither “set default parallel” nor the PARALLEL clause are used, using size of the input data Pig sets the number of reducers.

The properties values has been specified

  • pig.exec.reducers.bytes.per.reducer – Defines the number of input bytes per reduce; Pig reducer default value is 1000*1000*1000 (1GB).
  • pig.exec.reducers.max – Defines the upper bound on the number of reducers; Pig reducer default is 999.
]]> https://www.wikitechy.com/interview-questions/apache-pig/what-is-default-numbers-of-reducers-while-executing-a-pig-query/feed/ 0