• VB.NET offers code execution that runs under the Common Language Runtime (CLR), resulting in robust, stable and secure applications. Both features of the .NET framework are eagerly obtainable in VB.NET.
• VB.NET is object oriented. This is a major addition that VB6 and other earlier releases didn’t have.
• The .NET framework comes with ADO.NET, which follows the disconnected paradigm, i.e. once the required records are fetched the connection no longer exists.
• It also retrieves the records that are expected to be accessed in the immediate future.
• This enhances Scalability of the application to a great extent.
• VB.NET uses XML to transfer data between the various layers in the DNA Architecture i.e. data are passed as simple text strings.
• Error handling has changed in VB.NET. A new Try-Catch-Finally block has been introduced to handle errors and exceptions as a unit, allowing appropriate action to be taken at the place the error occurred thus discouraging the use of ON ERROR GOTO statement. This again credits to the maintainability of the code.

• Pig is a platform for managing large data.
• It uses high level programming to analyse the data.
• The programming is done using the language pig latin.
• It is simple and highly specific.
• The pig Metastore stores all info about the tables. And we can execute spark sql queries because spark can interact with pig Metastore

• Sqoop is a simple tool used for transferring large amount of data.
• It is used for transferring data from hadoop to relational database.
• sqoop can import data from Oracle,MYSQL,SQl.
• Sqoop metastore is a tool for using hosts in a shared metadata repository. Multiple users and remote users execute saved jobs in metastore.

• Staging table is used to stage the exported data.
• Sqoop breaks down method into multiple transactions. A failing job may create partial knowledge.
• This could result in job failing because it could cause duplication of data or even collision while insertion.
• Staging tables are simple database tables that contain business details.
• Staging table acts as an auxiliary table that’s used to transferred data.
• Sqoop breaks down export method into multiple transactions, it’s possible that a failing export job may lead to partial data being committed to the database

• Sqoop is a simple tool that is used for moving data.
• It is generally used move data from non hadoop data stores.
• These non hadoop data stores are the relational databases and data warehouses .
• Sqoop allows the users to specify the target (i.e) the place where the data is to be moved inside of hadoop from relational databases like Oracle,Teredata.
• Data ingestion is important in any big data project because the volume of data is generally in petabytes or exabytes.
• Hadoop Sqoop and Hadoop Flume are the two tools in Hadoop which is used to gather data from different sources and load them into HDFS.

• Sqoop metastore is an internally implemented database that can hold the definition of a given job.
• It includes details like which database,which table,which approach,what query to extract from,what target to write to and so on.
• Consider an array of SQL databases that have the same structure, but are partitioned by for performance reasons.
• Thus we have the same sqoop job running against multiple database instances at the same time, all unified at a single table in Hadoop.
• In order to get parallelization, we generate a job that manages the Sqoop sub-workflows. This results in 500 distinct sqoop jobs.
• Metastore is used to handle these complexities.

• Apache Sqoop is efficiently transferring data between Relational databases and Apache Hadoop.
• It is used to import data from relational databases such as MySQL, Oracle to Hadoop HDFS, and export from Hadoop file system to relational databases.
• It improves the efficiency of data analysis by combining structured data and unstructured data.
• Sqoop stands for SQL for Hadoop.SQL + HADOOP =SQOOP

There are several types of Query Optimization Techniques we can use in Hive in order to Optimize Hive Performance.

• • Tez-Execution Engine in Hive – Tez is a new application framework built on Hadoop Yarn that can execute complex directed acyclic graphs of general data processing tasks.
  • Usage of Suitable File Format in Hive.
  • Hive Partitioning

• • Bucketing in Hive

• • Vectorization In Hive

• Cost-Based Optimization in Hive (CBO)

• Table in Hive is stored as a directory in the HDFS.
• Using select from table the Hive query processor simply goes directory that have one or more files in table schema.
• You may do this if you have very small data like less than a Gigabyte.
• In real clusters if you hit ‘select * from table’, it may have data in Terabytes and displaying that will run for long time.
• Hive achieved sequence of map reduce programs that reads data from table stored on Hadoop Distributed File System.
• Any data processing you do in Hive is achieved through sequence of map reduce programs that reads data from table stored on HDFS.
• Hive map reduce based on query processing engine.
• Tables have wide number of columns that representing different values.To perform select column the map reduce program will scan all rows and extract a column.

• A view is a query which is defined as a table.

• CREATE VIEW x AS SELECT * FROM y;

• When the user queries table x the query planner combines the queries and executes a single one.

• An external table is something totally different. An external TABLE is a table that when DROPPED will NOT remove the physical data.
• This is a alternative that affects how data is loaded, controlled, and managed.

• The data is also used outside of Hive.
• For example, the data files are browse and processed by an existing program that doesn’t lock the files.
• Data needs to stay within the underlying location even after a DROP TABLE.
• Pointing multiple patterns at a single data it sets repeats via possible patterns.User can use custom location like ASV.
• Hive does not contain own data and control settings, dirs, etc.In Hive existing table (i.e) not modify.

• Temporary data needs Hive to manage the table and data.

• Sql is a database however hive is not a database.
• Sql is used for OLTP hive is used for OLAP.
• Sql is used for insert updates and deletes however hive is efficient for batch processing.

• Hadoop is an open source software stack that runs on a cluster of machines.
• Hadoop provides distributed storage and distributed process for very big data sets.

• Hadoop Distributed file system or HDFS is a Java based distributed file system that enables us to store big data across multiple nodes in a Hadoop cluster.
• So, if you install Hadoop, you will get HDFS as an underlying storage system for storing the big data sets in the distributed environment.

• MapReduce is a programming framework for writing applications that method massive amounts of structured and unstructured data in parallel across a cluster of thousands of machines, in a reliable, fault-tolerant manner.

• Apache HBase is Hadoop database, a distributed, scalable, column oriented big data store.
• HBase is built on top of HDFS means that data you store in HBase is stored in HDFS itself.

• Hive is an important tool for Hadoop ecosystem it provides an SQL for querying data in HDFS, other file systems that integrate with Hadoop like MapR-FS and Amazon’s S3 and databases like HBase(the Hadoop database) and Cassandra.
• Hive too like HBase stores data into HDFS but it uses MapReduce too. It compiles queries into MapReduce jobs and runs them on the cluster. It was the primary abstraction engines to be built on top of MapReduce. Hive needs a metastore(JDBC compliant RDBMS) to store its metadata.

• The default location of the hive table is: /user/hive/warehouse/databasename
• Go to in browser, it will take to a page with a Browse the filesystem link.
• In the $HIVE_HOME/conf directory there is the hive-default.xml and/or hive-site.xml which has the hive.metastore.warehouse.dir property.
• That value is where the user want to navigate to after clicking the Browse the filesystem link.
• Once the user navigate to that location, they will see the names of their tables.
• Clicking on a table name (which is simply a folder) can then expose the partitions of the table.
• If you wish access to data, you can use a Hive query and output the result to a file.These will have the exact same structure (divider between columns, ect) because the files on the HDFS.

• Partitioning is the optimization technique in Hive which improves the performance significantly.
• Apache Hive is the data warehouse on the top of Hadoop, which enables ad-hoc analysis over structured and semi-structured data
• Apache Hive organizes tables into partitions.Partitioning is dividing a table into connected components based on the values of specific columns like date, city, and department.
• Each table in hive can have one or additional partition keys to identify a particular partition. using partition it is easy to do queries on slices of the data.

Additionally, you can use Hive to copy data from a table to local directory.