• Generally, in a transistor some amount of voltage is required for turning it on.
• This voltage is the cut in voltage. Up to this voltage it exists a nonlinearity in its characteristics. Beyond this cut in voltage the characteristics of transistor is linear.
• This nonlinear region lays between 0-4mA, beyond 4ma it linear. That’s why 4-20 mA range is used.

• 4-20mA has an inherent ‘live zero’. Zero engineering unit is 4mA not 0.0mA indicates an open circuit, fault in the line which is not valid.
• 4-20 mA has little, if any, signal loss in its circuit; current regulation compensates for minor resistances in terminal boxes, whereas voltage drop across resistances for voltage signals creates an error.
• S 4-20 mA signal is less susceptible to noise pickup than a voltage circuit. So it is free from interference from other signals.
• 4-20 mA works at a relatively long distance(>1 Km ) boosting the power supply voltage will drive the signal further.

• f_v is the frequency of vortex shedding.
• D is the diameter of the pipe.
• w is the width of bluff body over diameter of pipe.
• K is the a constant for all fluids given design of meter. A k factor is introduced to compensate for the non-uniform profile of pipe flow.
• S is the Strouhal number, determined experimentally.

• K-factor is a dividing factor.
• The term is usually encountered when dealing with pulse signals although analog K-factors are sometimes used.

There are several types of Query Optimization Techniques we can use in Hive in order to Optimize Hive Performance.

• • Tez-Execution Engine in Hive – Tez is a new application framework built on Hadoop Yarn that can execute complex directed acyclic graphs of general data processing tasks.
  • Usage of Suitable File Format in Hive.
  • Hive Partitioning

• • Bucketing in Hive

• • Vectorization In Hive

• Cost-Based Optimization in Hive (CBO)

• Table in Hive is stored as a directory in the HDFS.
• Using select from table the Hive query processor simply goes directory that have one or more files in table schema.
• You may do this if you have very small data like less than a Gigabyte.
• In real clusters if you hit ‘select * from table’, it may have data in Terabytes and displaying that will run for long time.
• Hive achieved sequence of map reduce programs that reads data from table stored on Hadoop Distributed File System.
• Any data processing you do in Hive is achieved through sequence of map reduce programs that reads data from table stored on HDFS.
• Hive map reduce based on query processing engine.
• Tables have wide number of columns that representing different values.To perform select column the map reduce program will scan all rows and extract a column.

• A view is a query which is defined as a table.

• CREATE VIEW x AS SELECT * FROM y;

• When the user queries table x the query planner combines the queries and executes a single one.

• An external table is something totally different. An external TABLE is a table that when DROPPED will NOT remove the physical data.
• This is a alternative that affects how data is loaded, controlled, and managed.

• The data is also used outside of Hive.
• For example, the data files are browse and processed by an existing program that doesn’t lock the files.
• Data needs to stay within the underlying location even after a DROP TABLE.
• Pointing multiple patterns at a single data it sets repeats via possible patterns.User can use custom location like ASV.
• Hive does not contain own data and control settings, dirs, etc.In Hive existing table (i.e) not modify.

• Temporary data needs Hive to manage the table and data.

• Sql is a database however hive is not a database.
• Sql is used for OLTP hive is used for OLAP.
• Sql is used for insert updates and deletes however hive is efficient for batch processing.

• Hadoop is an open source software stack that runs on a cluster of machines.
• Hadoop provides distributed storage and distributed process for very big data sets.

• Hadoop Distributed file system or HDFS is a Java based distributed file system that enables us to store big data across multiple nodes in a Hadoop cluster.
• So, if you install Hadoop, you will get HDFS as an underlying storage system for storing the big data sets in the distributed environment.

• MapReduce is a programming framework for writing applications that method massive amounts of structured and unstructured data in parallel across a cluster of thousands of machines, in a reliable, fault-tolerant manner.

• Apache HBase is Hadoop database, a distributed, scalable, column oriented big data store.
• HBase is built on top of HDFS means that data you store in HBase is stored in HDFS itself.

• Hive is an important tool for Hadoop ecosystem it provides an SQL for querying data in HDFS, other file systems that integrate with Hadoop like MapR-FS and Amazon’s S3 and databases like HBase(the Hadoop database) and Cassandra.
• Hive too like HBase stores data into HDFS but it uses MapReduce too. It compiles queries into MapReduce jobs and runs them on the cluster. It was the primary abstraction engines to be built on top of MapReduce. Hive needs a metastore(JDBC compliant RDBMS) to store its metadata.

• The default location of the hive table is: /user/hive/warehouse/databasename
• Go to in browser, it will take to a page with a Browse the filesystem link.
• In the $HIVE_HOME/conf directory there is the hive-default.xml and/or hive-site.xml which has the hive.metastore.warehouse.dir property.
• That value is where the user want to navigate to after clicking the Browse the filesystem link.
• Once the user navigate to that location, they will see the names of their tables.
• Clicking on a table name (which is simply a folder) can then expose the partitions of the table.
• If you wish access to data, you can use a Hive query and output the result to a file.These will have the exact same structure (divider between columns, ect) because the files on the HDFS.

• Partitioning is the optimization technique in Hive which improves the performance significantly.
• Apache Hive is the data warehouse on the top of Hadoop, which enables ad-hoc analysis over structured and semi-structured data
• Apache Hive organizes tables into partitions.Partitioning is dividing a table into connected components based on the values of specific columns like date, city, and department.
• Each table in hive can have one or additional partition keys to identify a particular partition. using partition it is easy to do queries on slices of the data.

Additionally, you can use Hive to copy data from a table to local directory.

• Hue has a Web SQL Editor with autocomplete, display of table/data samples, light graphing, query download

• SQL Query Tool and Database Client

• RazorSQL is an SQL query tool, database editor, database browser, and database administration tool for Windows, macOS / Mac OS X, Linux, and Solaris.

• Edit SQL scripts.
• Run SQL queries.
• Auto column and auto table lookup.
• Multi-tabular query display.
• Supports over 20 programming languages including SQL, PL/SQL, T-SQL, PHP, HTML, XML, Java, and more.

• Edit database tables in a spreadsheet format with support for finding and replacing data, automatic generation of SQL, and support for editing multi-line data.

• Easily build select, insert, delete, and update SQL queries and statements. Build multi-table joins.

• Includes tools for creating tables, views, indexes, procedures, functions, triggers, databases, users and tools for altering and dropping tables and views.
• Also includes tools for visually editing procedures, functions, and triggers.

• Run a MR job to index data using solrj.
• Create Lucene index using mr job and duplicate to the appropriate shards.
• Use Hbase indexer to populate Solr.

• Understanding what to index typically requires deep business domain expertise on the data.
• This yields better indexing plan and increases accuracy for searching data.
• Not all data will be indexed but for an organization user have new data,Needs classification of all data untill it is understood what value it brings to the business.
• It implies is that data needs to be re-indexed so it is a good practice to store raw data somewhere low cost, often in HDFS or in the cloud object storage.

• Partitioning data is used for distributing load horizontally, helps to organizing data in a very logical fashion.
• Example:Assume user managing a large employee table and often run queries with where clauses that restrict the results to a particular country or department.
• For a quicker query response Hive table can be partitioned BY (country STRING, DEPT STRING). Partitioning tables changes however Hive structures the data storage and Hive can now create subdirectories reflecting the partitioning structure like…/employees/country=ABC/DEPT=XYZ.
• If query limits for employee from country=ABC, it will only scan the contents of 1 directory country=ABC.
• This can dramatically improve query performance, but providing the partitioning scheme reflects common filtering.
• Hive used to design too several partitions could optimize some queries, but be damaging for other important queries.
• Disadvantage several partitions having huge number of Hadoop files and file directories and Name Node should keep all data for the file system in memory.

• Decomposing data sets into more manageable parts using bucketing technique.
• For example, table using date as the top-level partition and employee_id as the second-level partition leads to too many small partitions.
• Employee_id contains the column it will be hashed by a user-defined number into buckets.
• Records with the same employee_id kept within the same bucket.
• Assuming the number of employee_id is much larger than the quantity of buckets, every bucket can have several employee_id.
• While creating table you can specify like CLUSTERED BY (employee_id) INTO XX BUCKETS; wherever XX is the number of buckets.
• Bucketing has many benefits.The number of buckets is fastened so it does not alternate with data.
• If a pair of tables are bucketed by employee_id, Hive will produce a logically correct sampling. Bucketing additionally aids in doing efficient map-side joins etc.

• Functionality to write & run multiple queries concurrently for the same user in the same session it used Hive, Impala, etc.,
• Hue has a Web SQL Editor with autocomplete, display of table samples, light graphing.Toad for Hadoop is used.
• HiveSQL Client is a free graphical standalone Java program.
• It allow user to connect Kerberised and Hive objects maintained by Hive Cluster.
• The client applications require MIT Kerberos it needs an additional step to log in and rebuild to obtain a new Kerberos ticket.

• By doing compression at various phases (i.e. on final output, intermediate data),we achieve performance improvement in Hive Queries.

1. Auto Map Join
2. Skew Joins
3. Enable Bucketed Map Joins

Auto Map Join:

• • Auto Map-Join is useful feature when joining a big table with a small table.
  • If we enable this feature, the small table will be saved in the local cache on each node, joined with the big table in the Map phase.
  • Enabling Auto Map Join provides 2 advantages.
  • Primary,it loads a small table into cache will save read time on each data node.
  • Secondary, it avoids skew joins in the Hive query, since the join operation has been already done in the Map phase for each block of data.

Skew joins:

• • We enable skew joins by setting hive.optimize.
  • Skew join property SET command in hive shell or hive-site.xml file.

Enable Bucketed Map Joins

• • The tables as specific column and tables used in joins to improve performance bucketed map join is used.

• Hive converts a query into more stages.The MapReduce stage, sampling stage, a mergestage and a limit stage.
• By default, Hive executes only one time for these satges.
• A particular job may consist of some stages that are not dependent on each other and could be executed in parallel, possibly allowing the overall job to complete more quickly.

• The single reducer used for multi operations, it combine multiple GROUP BY operations in a query into a single MapReduce job

• Vectorization introduced into hive for the first time in hive-0.13.1 release only
• It improve operations like scans, aggregations, filters and joins, batches of 1024 rows for each time.

• It provided the cost based optimization, based on query cost, resulting in different decisions: how to order joins, which type of join to perform and degree of parallelism.

• Map join may be a little-known feature of Hive.
• It allows a table to be loaded into memory so that a (very fast) join could be performed entirely within a mapper while not having to use a Map/Reduce step.
• It directs Hive to load aliasname (which may be a table or alias of the query) into memory.

• A bucket map join is used when the tables are large and all the tables used in the join are bucketed on the join columns. during this type of join, one table should have buckets in multiples of the number of buckets in another table.
• For example, if one table has two buckets then the other table must have either 2 buckets or a multiple of two buckets 2, 4, 6, and so on.
• The preceding condition satisfied joining can done mapper side only, otherwise a normal inner join is performed.
• It required buckets are fetched on the mapper side and not the complete table.
• That is, only the matching buckets of all small tables are duplicated onto each mapper.
• The efficiency of the query is improved to solve problem.In a bucket map join, data is not sorted.