different ways of executing pig script - Wikitechy https://www.wikitechy.com/interview-questions/tag/different-ways-of-executing-pig-script/ Interview Questions Wed, 15 Sep 2021 05:17:20 +0000 en-US hourly 1 https://wordpress.org/?v=6.9.4 https://www.wikitechy.com/interview-questions/wp-content/uploads/2025/10/cropped-wikitechy-icon-32x32.png different ways of executing pig script - Wikitechy https://www.wikitechy.com/interview-questions/tag/different-ways-of-executing-pig-script/ 32 32 What is workflow when using Pig & R ? https://www.wikitechy.com/interview-questions/apache-pig/what-is-workflow-when-using-pig-r/ https://www.wikitechy.com/interview-questions/apache-pig/what-is-workflow-when-using-pig-r/#respond Mon, 12 Jul 2021 06:37:19 +0000 https://www.wikitechy.com/interview-questions/?p=185

Workflow when using Pig & R

  • Pig programming language is used for obtaining and manipulating data perhaps doing otherwise with UDFs.
  • Pig is used for filtering, cleaning, transforming and otherwise preparing large data sets.
  • Depending on the workflow, user might throw a Hive table on top of the Pig output and then use Hive to prepare subsets of data for R.
  • R is used for analysis and modeling and it’s also act as prototype models.
  • There are ways to connect directly to Hive from R using the Hive JDBC drivers.
  • We use R for early data exploration and examination of results.
  • In a normal workflow, Pig is used to perform the ETL(Extract, Transform, Load) functionality and transform it load the data into HDFS and analysts utilizes HIVE or extensions like RHIVE to perform the analysis on large volume of data available in HDFS.
  • The Comparison between Pig & R is invalid for ever because the pig is for large scale data and R definitely is not.
]]> https://www.wikitechy.com/interview-questions/apache-pig/what-is-workflow-when-using-pig-r/feed/ 0 What is the internal architecture of Apache Pig ? https://www.wikitechy.com/interview-questions/apache-pig/what-is-the-internal-architecture-of-apache-pig/ https://www.wikitechy.com/interview-questions/apache-pig/what-is-the-internal-architecture-of-apache-pig/#respond Mon, 12 Jul 2021 05:19:41 +0000 https://www.wikitechy.com/interview-questions/?p=155

Internal architecture of Apache Pig

  • Pig Latin consist of pig to analyze the data from Hadoop.
  • Its a highlevel data processing language it perform various operations like data types and operators.
  • To perform a particular task Pig Script is used and execution mechanisms like(Grunt Shell, UDFs, Embedded).
  • To produce the desired output continuous transformations applied in Pig Framework.
  • Programmer’s job become easier when converting these scripts into a series of MapReduce jobs.
Pig Architecture

Apache Pig Components

  • There are various components in the Apache Pig framework.

Parser

  • The Pig Scripts are handled by the Parser.It checks the syntax,type checking, and other miscellaneous checks.The output will be like a DAG (directed acyclic graph), which represents the Pig Latin statements and logical operators.
  • In the DAG, the logical operators of the script are represented as nodes and therefore the data flows are represented as edges.

Optimizer

  • The logical plan DAG is passed to the logical optimizer,it carries out the logical optimizations,projection and pushdown.

Compiler

  • It compiles the optimized logical plan into a series of MapReduce jobs.

Execution engine

  • Producing the desired results MapReduce jobs are executed on Hadoop.
Pig Latin

Pig Latin Data Model

  • It allows datatypes such as map and tuple and it representation of Pig Latin’s data model.

Atom

  • Any single value in Pig Latin, irrespective of their data, type is known as an Atom.
  • Values of Pig in Atom are number,int,long,float,double,chararray, and bytearray.
  • Field define as a piece of data or a simple atomic value. Example − ‘xxx’ or ‘30’

Tuple

  • A record formed by ordered set of fields is known as a tuple. A tuple is similar to a row in a table of RDBMS. Example − (xxx, 30)

Bag

  • Is an unordered set of tuples.
  • A collection of tuples (non-unique) is known as a bag.Tuple have any number of fields (flexible schema).
  • No need that every tuple contain the same number of fields and the same position (column) have the same type.
    • Example − {(xxx, 30), (yyyy, 45)}
  • A Field contains bags in context its called inner bags.
    • Example − {xxx, 30, {98xxxxx22, xxxx@gmail.com,}}

Map

  • Map is a set of key-value pairs the key define as chararray and its unique.
  • It is represented by ‘[]’ Example − [name#xxx, age#30]

Relation

  • A relation is a bag of tuples.Pig Latin relations are unordered (there is no guarantee that tuples are processed in any particular order).
]]> https://www.wikitechy.com/interview-questions/apache-pig/what-is-the-internal-architecture-of-apache-pig/feed/ 0 What is Apache Pig ? https://www.wikitechy.com/interview-questions/apache-pig/what-is-apache-pig/ https://www.wikitechy.com/interview-questions/apache-pig/what-is-apache-pig/#respond Mon, 12 Jul 2021 01:13:57 +0000 https://www.wikitechy.com/interview-questions/?p=148

Apache Pig

    • Apache Pig is used to creating programs that runs on Apache Hadoop.
    • Pig Latin is a language.
    • It execute in Hadoop jobs,MapReduce, Apache Tez, or Apache Spark
Big Data
The programming from the Java MapReduce idiom into a notation which makes MapReduce high level, similar to SQL for relational database management systems.
  • The user-defined functions (UDFs) called Pig Latin user can write in Java, Python, JavaScript, Ruby or Groovy.
Creating your first pig script

Advantages of Pig

  • Reusing the code
  • Quick development
  • Less lines of code
  • Type checking and Schema etc.

Simple program for pig:

  • Map: The data element consisting the data type chararray where element has pig data type include complex data type
  • Example- [city’#’bang’,’pin’#560001].
  • The city and pin are data element mapping the values here.
]]> https://www.wikitechy.com/interview-questions/apache-pig/what-is-apache-pig/feed/ 0