Best Apache Pig Interview Questions - Wikitechy

What is the difference between Pig, Hive and MapReduce ?

Editor — Mon, 12 Jul 2021 08:26:57 +0000

Difference between Pig, Hive and MapReduce

Pig	Hive	Hadoop mapreduce
Pig is a scripting language	SQL like query language	It is a compiled language
Higher level of abstraction	Higher level of abstraction	Hadoop mapreduce used lower level of abstraction
Comparatively less line of codes than mapreduce	Comparatively less lines of code than mapreduce and apache pig	More lines of code
Development effort is less code efficiency is relatively less	Development effort is less code efficiency is relatively less	More development , effort is involved
Code efficiency is relatively less	Code efficiency is relatively less	Code efficiency is high when compared to pig and hive
Pig is open source	Hive open source	Hadoop MapReduce were built so that hadoop developers could do the same thing in Java in a less verbose way by writing only fewer lines of code that is easy to understand
Disadvantage of pig is that commands are not executed unless either you dump or store an intermediate or final result.This increases the iteration between debug and resolving the issue.	Disadvantages of hive is that No real time access to data, Updating data is complicated	The only drawback that developers need to write several lines of basic java code

What is the difference between Pig and Hive ?

Editor — Mon, 12 Jul 2021 08:22:29 +0000

Difference between pig and Hive

Pig	Hive
Pig Hadoop Component is generally used by Researchers and Programmers.	Hive Hadoop Component is mainly used by data analysts.
Pig Hadoop Component is used for semi structured data.	Hive Hadoop Component is used for completely structured Data.
Pig Hadoop Component has a procedural data flow language (Pig Latin).	Hive Hadoop Component has a declarative Structured Query Language (HiveQL).
Pig can’t start reliance based server.	Hive can start reliance based server,send queries from any nook and corner directly to the Hive server which will execute them.
Pig is also SQL like but it varies from great extent and it will take some time efforts to master Pig.	Hive directly borrowed SQL expertise it learnt easily.
Avro supported for Pig.	Avro does not support for Hive.
Pig Hadoop Component operates on the client side of any group.	Hive Hadoop Component operates on the server side of any group.
Pig Hadoop Component is highly used for programming.	Hive Hadoop Component is mainly used for generating reports.
Pig Hadoop is a great ETL tool for big data because of its powerful transformation and processing capabilities.	Hive Hadoop Component is helpful for ETL(Extract, Transform and Load).
In Pig there is no dedicated metadata database and the schemas or data types will be defined in the script itself.	Hive makes use of exact variation of the SQL DLL language by defining the tables beforehand and storing the schema details in any local database.
Pig Hadoop component does not have any notion for partitions though might be one can achieve this through filters.	The Hive Hadoop component has a provision for partitions so that you can process the subset of data by date or in an alphabetical order.
Pig Hadoop Component renders users with sample data for each scenario and each step through its “Illustrate” function.	This feature is not incorporated with the Hive Hadoop Component.

What is the difference between Pig,Hive and Hadoop ?

Editor — Mon, 12 Jul 2021 08:18:28 +0000

Difference between Pig,Hive and Hadoop

Pig	Hive	Hadoop
Apache Pig is a high-level platform for creating programs it runs on Apache Hadoop.	Apache Hive is a data warehouse software project built on top of Apache Hadoop for providing data query, and analysis.	Open-source software framework for distributed storage and processing dataset of big data using the MapReduce programming model.
Pig Latin extracts the programming from the Java MapReduce phrase into a code which creates MapReduce programming high level.	Hive provides the SQL abstraction to integrate SQL-like queries (HiveQL). The underlying Java without we need to implement queries in the low-level Java API.	The Hadoop Common package for Java ARchive (JAR) files and scripts needed to start Hadoop.
Pig basically has 2 parts: the Pig Interpreter and the language, PigLatin	It provides us data warehousing facilities on top of an existing Hadoop cluster.	Hadoop is basically two things:Hadoop Distributed File System (HDFS) +a Computation or Processing framework (MapReduce).
Pig makes our life a lot easier, otherwise writing MapReduce is always not easy.	User can create tables in Hive and store data there. Along with that user can even map their existing HBase tables to Hive and operate on them.	Hadoop is a distributed, scalable,big data store, modelled after Google’s BigTable.It stores data as key/value pairs.

What is the difference between group and cogroup in Pig Latin ?

Editor — Mon, 12 Jul 2021 08:13:18 +0000

Difference between group and cogroup in Pig Latin

Group	Cogroup
For readability GROUP is used in statements involving one relation	Cogroup used as a statements involving two or more relations. Cogroup should no exceed 127 relations at a time.
The group statement collects together records with the same key	cogroup is a general statement of group. Instead of collecting records of 1 input it collects records of N inputs based on a key
Syntax: grunt> Group_data = GROUP Relation_name BY age;	Syntax: alias = CROSS alias, alias [, alias …] [PARTITION BY partitioner] [PARALLEL n];

What is the best tool to process web streaming data in Hadoop,PIG or HIVE ?

Editor — Mon, 12 Jul 2021 08:07:43 +0000

Best tool to process web streaming data in Hadoop or PIG or HIVE :

It contains easy programming.
It is little to achieve parallel execution of simple, “embarrassingly parallel” data analysis tasks.
For Complex tasks it comprised the multiple interrelated data transformations are explicitly encoded as data flow sequences, making them easy to write, understand, and maintain.
To upgrade their execution automatically, allowing the user to focus on semantics rather than efficiency and Extensibility.
A special-purpose processing to create own functions.
For efficient working pig and hive can be used together.
Pig is best tool for parsing (ETL) kind of job, and even pig supports UDF better than hive.
It can help you to develop you own framework using pig where you can call Hive ql and map-reduce job for better functionality.

What is the advantages of pig in Hadoop ?

Editor — Mon, 12 Jul 2021 08:04:01 +0000

Advantages of pig in Hadoop

- The development time is Decrease.
- It is a collection of Rich set of operators in order to perform operations such as join, filer, sort and many more.
- Pig Latin is similar to SQL, it becomes very easy for developers to write a Pig script.
- The task execution in Apache Pig it gets automatically optimized , hence the programmers need to focus on semantics language.

Using the existing operators, users can easily develop their own functions to process,read, and write data.
Pig provide the facility for creating User Defined Functions UDF’s, we can easily create a number of programming languages such as Java and invoke them in Pig Scripts.
All types of data handling are having both structured & unstructured is provided by Apache Pig and the results are stored inside HDFS.
You have UDFs which you want to parallellize and utilize for large amounts of data, then you are in luck. Use Pig as a base pipeline where it does the hard work and you just apply your UDF in the step that you want.

What is Hbase used for, and what are the benefits that it provides over Pig and Hive ?

Editor — Mon, 12 Jul 2021 07:58:18 +0000

Hbase uses

HBASE will not replace Map Reduce. It is scalable distributed database.
Hbase processing data will act as Map Reduce.
The Hadoop database is define Apache HBase.
HBase is a sub-project of the Apache Hadoop project and is used to provide real-time read and write access to big data.
Hbase is a Bigtable clone.
Hbase is a open source.
Hbase called as Cascading connector.

what are the Features that it provides over Pig and Hive?

Possible to create tables in Hive and store data,it can even map the user’s existing HBase tables to Hive and operate on them.
Hive should be used for analyzing query of data collected over a period of time.
Hbase has got its own set of very good Application Programming Interface(API) which could be used to push/pull the data.
It has a good community for the future.It developed on top if hadoop already used to integration for the Hadoop platform.
Hbase can be integrated with MapReduce that can do bulk operation, like indexing, analytics etc.
- Distributed storage
- Table-like in data structure
- multi-dimensional map
- High scalability
- High availability
- High performance

A distributed data store that can scale horizontally to 1,000s of commodity servers and petabytes of indexed storage.
Designed to operate on top of the Hadoop distributed file system (HDFS) or Kosmos File System (KFS, aka Cloudstore) for scalability, fault tolerance, and high availability.

What is default numbers of reducers while executing a pig query ?

Editor — Mon, 12 Jul 2021 07:14:15 +0000

Set Number of Reducer in Pig:

SET default_parallel XXX

Where XXX is the number of reducer.
This command is used to set the number of reducers at the script level
The coder need to write this configuration at top/beginning of their pig script.
Alternatively, use the PARALLEL clause to set the number of reducers at the operator level.
We set the value using the PARALLEL clause will override any value we specify through (“SET default parallel.”) to reduce phase you can include the PARALLEL clause with any operator.

- COGROUP
- CROSS
- DISTINCT
- GROUP
- JOIN (inner)
- JOIN (outer) and
- ORDER BY.

For Example:

In GROUP operator the PARALLEL class has been used.

A = LOAD ‘myfile’ AS (t, u, v);
B = GROUP A BY t PARALLEL 18

Here 18 is number of reducer.

If neither “set default parallel” nor the PARALLEL clause are used, using size of the input data Pig sets the number of reducers.

The properties values has been specified

pig.exec.reducers.bytes.per.reducer – Defines the number of input bytes per reduce; Pig reducer default value is 1000*1000*1000 (1GB).
pig.exec.reducers.max – Defines the upper bound on the number of reducers; Pig reducer default is 999.

What is Pig in Hadoop ?

Editor — Mon, 12 Jul 2021 06:43:27 +0000

Pig in Hadoop

User can perform all the data manipulation operations in Hadoop using Apache Pig.
Apache Pig is an abstraction over MapReduce.

Need Apache Pig:

Using Pig Latin, programmers can perform MapReduce tasks codes in Java.
Apache Pig uses multi-query approach, it reducing the length of codes.
For example, an operation that would require you to type 200 lines of code in Java can be easily done by typing just ten lines of codes in Apache Pig.
Ultimately Apache Pig reduces the development time by almost sixteen times.