apache pig architecture diagram - Wikitechy https://www.wikitechy.com/interview-questions/tag/apache-pig-architecture-diagram/ Interview Questions Wed, 15 Sep 2021 05:17:20 +0000 en-US hourly 1 https://wordpress.org/?v=6.9.4 https://www.wikitechy.com/interview-questions/wp-content/uploads/2025/10/cropped-wikitechy-icon-32x32.png apache pig architecture diagram - Wikitechy https://www.wikitechy.com/interview-questions/tag/apache-pig-architecture-diagram/ 32 32 What is the difference between Pig, Hive and MapReduce ? https://www.wikitechy.com/interview-questions/apache-pig/what-is-the-difference-between-pig-hive-and-mapreduce/ https://www.wikitechy.com/interview-questions/apache-pig/what-is-the-difference-between-pig-hive-and-mapreduce/#respond Mon, 12 Jul 2021 08:26:57 +0000 https://www.wikitechy.com/interview-questions/?p=207

Difference between Pig, Hive and MapReduce

Pig Hive Hadoop mapreduce
Pig is a scripting language SQL like query language It is a compiled language
Higher level of abstraction Higher level of abstraction Hadoop mapreduce used lower level of abstraction
Comparatively less line of codes
than mapreduce		 Comparatively less lines of code
than mapreduce and apache pig		 More lines of code
Development effort is less code efficiency
is relatively less		 Development effort is less code
efficiency is relatively less		 More development , effort is involved
Code efficiency is relatively less Code efficiency is relatively less Code efficiency is high when compared
to pig and hive
Pig is open source Hive open source Hadoop MapReduce were built so that hadoop developers could do the same
thing in Java in a less verbose way by writing only
fewer lines of code that is easy to understand
Disadvantage of pig is that commands
are not executed unless either you dump
or store an intermediate or final result.This
increases the iteration between debug and
resolving the issue.		 Disadvantages of hive is that No
real time access to data, Updating
data is complicated		 The only drawback that developers need to write several lines of basic java code
differences-between-hive-and-pig
]]> https://www.wikitechy.com/interview-questions/apache-pig/what-is-the-difference-between-pig-hive-and-mapreduce/feed/ 0 What is workflow when using Pig & R ? https://www.wikitechy.com/interview-questions/apache-pig/what-is-workflow-when-using-pig-r/ https://www.wikitechy.com/interview-questions/apache-pig/what-is-workflow-when-using-pig-r/#respond Mon, 12 Jul 2021 06:37:19 +0000 https://www.wikitechy.com/interview-questions/?p=185

Workflow when using Pig & R

  • Pig programming language is used for obtaining and manipulating data perhaps doing otherwise with UDFs.
  • Pig is used for filtering, cleaning, transforming and otherwise preparing large data sets.
  • Depending on the workflow, user might throw a Hive table on top of the Pig output and then use Hive to prepare subsets of data for R.
  • R is used for analysis and modeling and it’s also act as prototype models.
  • There are ways to connect directly to Hive from R using the Hive JDBC drivers.
  • We use R for early data exploration and examination of results.
  • In a normal workflow, Pig is used to perform the ETL(Extract, Transform, Load) functionality and transform it load the data into HDFS and analysts utilizes HIVE or extensions like RHIVE to perform the analysis on large volume of data available in HDFS.
  • The Comparison between Pig & R is invalid for ever because the pig is for large scale data and R definitely is not.
]]> https://www.wikitechy.com/interview-questions/apache-pig/what-is-workflow-when-using-pig-r/feed/ 0 What is the internal architecture of Apache Pig ? https://www.wikitechy.com/interview-questions/apache-pig/what-is-the-internal-architecture-of-apache-pig/ https://www.wikitechy.com/interview-questions/apache-pig/what-is-the-internal-architecture-of-apache-pig/#respond Mon, 12 Jul 2021 05:19:41 +0000 https://www.wikitechy.com/interview-questions/?p=155

Internal architecture of Apache Pig

  • Pig Latin consist of pig to analyze the data from Hadoop.
  • Its a highlevel data processing language it perform various operations like data types and operators.
  • To perform a particular task Pig Script is used and execution mechanisms like(Grunt Shell, UDFs, Embedded).
  • To produce the desired output continuous transformations applied in Pig Framework.
  • Programmer’s job become easier when converting these scripts into a series of MapReduce jobs.
Pig Architecture

Apache Pig Components

  • There are various components in the Apache Pig framework.

Parser

  • The Pig Scripts are handled by the Parser.It checks the syntax,type checking, and other miscellaneous checks.The output will be like a DAG (directed acyclic graph), which represents the Pig Latin statements and logical operators.
  • In the DAG, the logical operators of the script are represented as nodes and therefore the data flows are represented as edges.

Optimizer

  • The logical plan DAG is passed to the logical optimizer,it carries out the logical optimizations,projection and pushdown.

Compiler

  • It compiles the optimized logical plan into a series of MapReduce jobs.

Execution engine

  • Producing the desired results MapReduce jobs are executed on Hadoop.
Pig Latin

Pig Latin Data Model

  • It allows datatypes such as map and tuple and it representation of Pig Latin’s data model.

Atom

  • Any single value in Pig Latin, irrespective of their data, type is known as an Atom.
  • Values of Pig in Atom are number,int,long,float,double,chararray, and bytearray.
  • Field define as a piece of data or a simple atomic value. Example − ‘xxx’ or ‘30’

Tuple

  • A record formed by ordered set of fields is known as a tuple. A tuple is similar to a row in a table of RDBMS. Example − (xxx, 30)

Bag

  • Is an unordered set of tuples.
  • A collection of tuples (non-unique) is known as a bag.Tuple have any number of fields (flexible schema).
  • No need that every tuple contain the same number of fields and the same position (column) have the same type.
    • Example − {(xxx, 30), (yyyy, 45)}
  • A Field contains bags in context its called inner bags.
    • Example − {xxx, 30, {98xxxxx22, xxxx@gmail.com,}}

Map

  • Map is a set of key-value pairs the key define as chararray and its unique.
  • It is represented by ‘[]’ Example − [name#xxx, age#30]

Relation

  • A relation is a bag of tuples.Pig Latin relations are unordered (there is no guarantee that tuples are processed in any particular order).
]]> https://www.wikitechy.com/interview-questions/apache-pig/what-is-the-internal-architecture-of-apache-pig/feed/ 0 What is Apache Pig ? https://www.wikitechy.com/interview-questions/apache-pig/what-is-apache-pig/ https://www.wikitechy.com/interview-questions/apache-pig/what-is-apache-pig/#respond Mon, 12 Jul 2021 01:13:57 +0000 https://www.wikitechy.com/interview-questions/?p=148

Apache Pig

    • Apache Pig is used to creating programs that runs on Apache Hadoop.
    • Pig Latin is a language.
    • It execute in Hadoop jobs,MapReduce, Apache Tez, or Apache Spark
Big Data
The programming from the Java MapReduce idiom into a notation which makes MapReduce high level, similar to SQL for relational database management systems.
  • The user-defined functions (UDFs) called Pig Latin user can write in Java, Python, JavaScript, Ruby or Groovy.
Creating your first pig script

Advantages of Pig

  • Reusing the code
  • Quick development
  • Less lines of code
  • Type checking and Schema etc.

Simple program for pig:

  • Map: The data element consisting the data type chararray where element has pig data type include complex data type
  • Example- [city’#’bang’,’pin’#560001].
  • The city and pin are data element mapping the values here.
]]> https://www.wikitechy.com/interview-questions/apache-pig/what-is-apache-pig/feed/ 0