[Solved-1 Solution] A way to read table data from Mysql to Pig ?
What is Mysql:
- MySQL is an open source relational database management system (RDBMS) based on Structured Query Language (SQL).
- MySQL runs on virtually all platforms, including Linux, UNIX, and Windows. Although it can be used in a wide range of applications, MySQL is most often associated with web-based applications.
What is read table data
- Reads a file in table format and creates a data frame from it, with cases corresponding to lines and variables to fields in the file.
Problem :
How to read table data from mysql to pig ?
Solution 1:
-
DBStorageonly supports saving results to a database. - To load data from MySQL we could look into a project called sqoop.
- Sqoop helps to copy a data from a database to HDFS, or we could perform a mysql dump and then copy the file into HDFS.
- Both ways required some interaction and cannot be directly used from inside Pig.
- A third option look into writing a Pig LoadFunc.
- Its not much difficult, we need to pass much the same options as DBStorage (driver, connection credentials and a SQL query to execute), and we can probably use some result set metadata inspection to auto generate a schema too.