sqoop - sqoop2 - sqoop2 tutorials - apache sqoop - sqoop tutorial - sqoop hadoop

Apache Top-Level Project

SQl to hadOOP

Tool to transfer data from relational databases

• Teradata, MySQL, PostgreSQL, Oracle, Netezza

To Hadoop ecosystem

• HDFS (text, sequence file), Hive, HBase, Avro And vice versa

sqoop-hadoop-hive :

Sqoop1 Architecture :

Sqoop1 Challenges

• CrypAc, contextual command line arguments
• Tight coupling between data transfer and output format
• Security concerns with openly shared credentials
• Not easy to manage installation/Configuration
• Connectors are forced to follow JDBC model

Sqoop2 Architecture :

---

Sqoop1: Client side Tool

Client side installation+ configuration –

• - Connectors are installed/configured locally
• – Local requires root privileges
• – JDBC drivers are needed locally
• – Database connecAvity is needed locally

Sqoop2: Sqoop as a Service - client side tools :

Server side installation + configuration

• – Connectors are installed/configured in one place
• – Managed by administrator and run by operator
• – JDBC drivers are needed in one place
• – Database connectivity is needed on the server

---

Client Interface

Sqoop1 client interface:

• – Command line interface (CLI) based
• – Can be automated via scripting

Sqoop 2 client interface:

• – CLI based (in either interactive or script mode)
• – Web based (remotely accessible)
• – REST API is exposed for external tool integration

Sqoop 2: Connection vs Job metadata :

There are two distinct sets of options to pass in to Sqoop:

• Connection (distinct per database)
• Job (distinct per table)

Working Process of connecting sqoop database

• Connectors Register Metadata
• Metadata enables creation of Connections and Jobs
• Connections and Jobs stored in Metadata Repository
• Operator runs Jobs that use appropriate connections
• Admins set policy for connection use

Sqoop 2: Security

Support for secure access to external systems via role-based access to connection objects

• Administrators create/edit/delete connections
• Operators use connections

Sqoop 2: Usability & Extensibility

Connections and Jobs use domain specific inputs (Tables, Operations, etc.)

Domain Isolation and thus easy to understand and use

Connectors work with Intermediate Data Format

Any downstream functionality needed is provided by Sqoop Framework