• Hadoop is an open source software stack that runs on a cluster of machines.
• Hadoop provides distributed storage and distributed process for very big data sets.

• Hadoop Distributed file system or HDFS is a Java based distributed file system that enables us to store big data across multiple nodes in a Hadoop cluster.
• So, if you install Hadoop, you will get HDFS as an underlying storage system for storing the big data sets in the distributed environment.

• MapReduce is a programming framework for writing applications that method massive amounts of structured and unstructured data in parallel across a cluster of thousands of machines, in a reliable, fault-tolerant manner.

• Apache HBase is Hadoop database, a distributed, scalable, column oriented big data store.
• HBase is built on top of HDFS means that data you store in HBase is stored in HDFS itself.

• Hive is an important tool for Hadoop ecosystem it provides an SQL for querying data in HDFS, other file systems that integrate with Hadoop like MapR-FS and Amazon’s S3 and databases like HBase(the Hadoop database) and Cassandra.
• Hive too like HBase stores data into HDFS but it uses MapReduce too. It compiles queries into MapReduce jobs and runs them on the cluster. It was the primary abstraction engines to be built on top of MapReduce. Hive needs a metastore(JDBC compliant RDBMS) to store its metadata.

• Run a MR job to index data using solrj.
• Create Lucene index using mr job and duplicate to the appropriate shards.
• Use Hbase indexer to populate Solr.

• Understanding what to index typically requires deep business domain expertise on the data.
• This yields better indexing plan and increases accuracy for searching data.
• Not all data will be indexed but for an organization user have new data,Needs classification of all data untill it is understood what value it brings to the business.
• It implies is that data needs to be re-indexed so it is a good practice to store raw data somewhere low cost, often in HDFS or in the cloud object storage.

• By doing compression at various phases (i.e. on final output, intermediate data),we achieve performance improvement in Hive Queries.

1. Auto Map Join
2. Skew Joins
3. Enable Bucketed Map Joins

Auto Map Join:

• • Auto Map-Join is useful feature when joining a big table with a small table.
  • If we enable this feature, the small table will be saved in the local cache on each node, joined with the big table in the Map phase.
  • Enabling Auto Map Join provides 2 advantages.
  • Primary,it loads a small table into cache will save read time on each data node.
  • Secondary, it avoids skew joins in the Hive query, since the join operation has been already done in the Map phase for each block of data.

Skew joins:

• • We enable skew joins by setting hive.optimize.
  • Skew join property SET command in hive shell or hive-site.xml file.

Enable Bucketed Map Joins

• • The tables as specific column and tables used in joins to improve performance bucketed map join is used.

• Hive converts a query into more stages.The MapReduce stage, sampling stage, a mergestage and a limit stage.
• By default, Hive executes only one time for these satges.
• A particular job may consist of some stages that are not dependent on each other and could be executed in parallel, possibly allowing the overall job to complete more quickly.

• The single reducer used for multi operations, it combine multiple GROUP BY operations in a query into a single MapReduce job

• Vectorization introduced into hive for the first time in hive-0.13.1 release only
• It improve operations like scans, aggregations, filters and joins, batches of 1024 rows for each time.

• It provided the cost based optimization, based on query cost, resulting in different decisions: how to order joins, which type of join to perform and degree of parallelism.

• Hive is a component of Hortonworks Data Platform(HDP).
• Hive provides a SQL interface to data stored in HDP.
• Hive has 3 main functions:

• • Data Summarization
  • Query
  • Analysis.

• It supports queries expressed language called HiveQL, which automatically translates SQL like queries into MapReduce jobs executed on Hadoop.
• It also enables data serialization and increases flexibility in schema architecture including a system catalog called Hive Metastore.

• Different storage such as plain text, RCFile, ORC, HBase, and others.
• Pre-defined functions (UDFs) to manipulate dates, strings, and other data-mining tools.
• Hive supports extending the UDF set to handle use-cases not supported by built-in functions.

• Hive supports overwriting or hold data, but not updates and deletes.
• In Hive, sub queries are not supported

• • The development time is Decrease.
  • It is a collection of Rich set of operators in order to perform operations such as join, filer, sort and many more.
  • Pig Latin is similar to SQL, it becomes very easy for developers to write a Pig script.
  • The task execution in Apache Pig it gets automatically optimized , hence the programmers need to focus on semantics language.

• Using the existing operators, users can easily develop their own functions to process,read, and write data.
• Pig provide the facility for creating User Defined Functions UDF’s, we can easily create a number of programming languages such as Java and invoke them in Pig Scripts.
• All types of data handling are having both structured & unstructured is provided by Apache Pig and the results are stored inside HDFS.
• You have UDFs which you want to parallellize and utilize for large amounts of data, then you are in luck. Use Pig as a base pipeline where it does the hard work and you just apply your UDF in the step that you want.

• Where XXX is the number of reducer.
• This command is used to set the number of reducers at the script level
• The coder need to write this configuration at top/beginning of their pig script.
• Alternatively, use the PARALLEL clause to set the number of reducers at the operator level.
• We set the value using the PARALLEL clause will override any value we specify through (“SET default parallel.”) to reduce phase you can include the PARALLEL clause with any operator.

• • COGROUP
  • CROSS
  • DISTINCT
  • GROUP
  • JOIN (inner)
  • JOIN (outer) and
  • ORDER BY.

• A = LOAD ‘myfile’ AS (t, u, v);
• B = GROUP A BY t PARALLEL 18

• If neither “set default parallel” nor the PARALLEL clause are used, using size of the input data Pig sets the number of reducers.

• pig.exec.reducers.bytes.per.reducer – Defines the number of input bytes per reduce; Pig reducer default value is 1000*1000*1000 (1GB).
• pig.exec.reducers.max – Defines the upper bound on the number of reducers; Pig reducer default is 999.

• Pig Latin is not a language but its a language game that all use to speak in code.
• Altering words in English Pig Latin is formed.
• Pig Latin can be extended using user-defined functions which the user can write these languages Java, Python, JavaScript, Ruby or Groovy and then call directly.
• It execute Hadoop jobs in MapReduce, Apache Tez, or Apache Spark
• Pig Latin abstracts the programming from the Java MapReduce idiom which makes MapReduce programming as a high level, its similar to that SQL for relational database management systems.

• User can perform all the data manipulation operations in Hadoop using Apache Pig.
• Apache Pig is an abstraction over MapReduce.

• Using Pig Latin, programmers can perform MapReduce tasks codes in Java.
• Apache Pig uses multi-query approach, it reducing the length of codes.
• For example, an operation that would require you to type 200 lines of code in Java can be easily done by typing just ten lines of codes in Apache Pig.
• Ultimately Apache Pig reduces the development time by almost sixteen times.

• Pig programming language is used for obtaining and manipulating data perhaps doing otherwise with UDFs.
• Pig is used for filtering, cleaning, transforming and otherwise preparing large data sets.
• Depending on the workflow, user might throw a Hive table on top of the Pig output and then use Hive to prepare subsets of data for R.
• R is used for analysis and modeling and it’s also act as prototype models.
• There are ways to connect directly to Hive from R using the Hive JDBC drivers.
• We use R for early data exploration and examination of results.
• In a normal workflow, Pig is used to perform the ETL(Extract, Transform, Load) functionality and transform it load the data into HDFS and analysts utilizes HIVE or extensions like RHIVE to perform the analysis on large volume of data available in HDFS.
• The Comparison between Pig & R is invalid for ever because the pig is for large scale data and R definitely is not.

• The FOREACH operator is used to generate specified data transformations based on the column data.

1. Simple FOREACH…GENERATE
2. Nested FOREACH {…GENERATE }

• This is simple to use the FOREACH…GENERATE in a single line of pig statement to generate the columns that we need.
• Relation_Name = FOREACH <Previous Relation Name> GENERATE column1, column2…..;

• Use operators such as DISTINCT, FILTER, LIMIT, ORDER and SAMPLE.

• • It should be taken set of expressions and it applied to every record in the data pipeline, hence define as FOREACH.
  • The pipeline to next operator.For those database, it is Pig’s projection operator.

For example, the code loads an entire record, then it removes all user and id fields from each record:

• • A = load ‘input’ as (user:chararray, id:long, address:chararray, phone:chararray, preferences:map[]);
  • B = foreach A generate user, id;

• It allows datatypes such as map and tuple and it representation of Pig Latin’s data model.

• Any single value in Pig Latin, irrespective of their data, type is known as an Atom.
• Values of Pig in Atom are number,int,long,float,double,chararray, and bytearray.
• Field define as a piece of data or a simple atomic value. Example − ‘xxx’ or ‘30’

• A record formed by ordered set of fields is known as a tuple. A tuple is similar to a row in a table of RDBMS. Example − (xxx, 30)

• Is an unordered set of tuples.
• A collection of tuples (non-unique) is known as a bag.Tuple have any number of fields (flexible schema).
• No need that every tuple contain the same number of fields and the same position (column) have the same type.
  • Example − {(xxx, 30), (yyyy, 45)}
• A Field contains bags in context its called inner bags.
  • Example − {xxx, 30, {98xxxxx22, xxxx@gmail.com,}}

• Map is a set of key-value pairs the key define as chararray and its unique.
• It is represented by ‘[]’ Example − [name#xxx, age#30]

• A relation is a bag of tuples.Pig Latin relations are unordered (there is no guarantee that tuples are processed in any particular order).

• In 2006 Pig was developed by Yahoo Research for particular way of creating and executing MapReduce jobs on very large data sets.
• Apache Software Foundation was moved into 2007.
• Pig started a project in Yahoo! Research, where Yahoo! scientists designed it and produced an implementation.
• The MapReduce paradigm presented by Hadoop “is too low-level and rigid, and leads to a great deal of custom user code that is hard to maintain and reuse.”
• Users were not comfortable with MapReduce languages such as SQL.
• Sort the problem a new language called Pig Latin that we have designed to fit between the declarative style of SQL, and the low-level, procedural style of MapReduce.”
• Hadoop users adopt Pig. So, a team of development engineers was assembled to take the research prototype and build into a production-quality product. About this same time, in fall 2007, Pig was open sourced via the Apache Incubator.
• The first Pig release in September 2008, were Pig graduated from the Incubator and became a subproject of Apache Hadoop.