sqoop - Sqoop saved Job - apache sqoop - sqoop tutorial - sqoop hadoop



What is Sqoop saved job? - Saved Jobs in Sqoop

  • The Saved Sqoop Job remembers the parameters used by a job so they can be re-executed by invoking the job several times.
  • Following command creates saved jobs:
    sqoop job --create job_name --import --connect <connect-string>/dbname \ --table table_name
    
    Click "Copy code" button to copy into clipboard - By wikitechy - sqoop tutorial - team
  • The command above just creates a job with the job name you specify.
  • It means that the job you created is now available in your saved jobs list which can be executed later.
  • Following command executes a saved job :
    sqoop job --exec job_name --username uname –P
    
    Click "Copy code" button to copy into clipboard - By wikitechy - sqoop tutorial - team
  • Sample Saved Job

    sqoop job --create JOB1 -- import --connect jdbc:mysql://192.168.56.1:3306/adventureworks 
    -username XXX
    -password XXX
    --table transactionhistory 
    --target-dir /user/cloudera/datasets/trans 
    -m 1  
    --columns "TransactionID,ProductId,TransactionDate" 
    --check-column TransactionDate 
    --incremental  lastmodified 
    --last-value "2004-09-01 00:00:00"; 
    Click "Copy code" button to copy into clipboard - By wikitechy - sqoop tutorial - team
    learn sqoop - sqoop tutorial - sqoop2 tutorial - data ingestion tool - sqoop job - sqoop code - sqoop programming - sqoop download - sqoop examples

    Sqoop Metastore

  • A Sqoop metastore keeps track of all jobs.
  • By default, the metastore is contained in your home directory under .sqoop and is only used for your own jobs. If you want to share jobs, you would need to install a JDBC-compliant database and use the --meta-connect argument to specify its location when issuing job commands.
  • Important Sqoop commands:
    • $ sqoop job –list – Lists all jobs available in metastore
    • sqoop job --exec JOB1 – Executes JOB1
    • sqoop job --show JOB1 – Displays metadata of JOB1
  • Sqoop Option File :

  • Certain arguments in import, export commands and saved jobs are to be written every time you execute them.
  • For instance following arguments are used repetitively in import and export commands as well as saved jobs :
  • learn sqoop - sqoop tutorial - sqoop2 tutorial - sqoop option text - sqoop job - sqoop code - sqoop programming - sqoop download - sqoop examples

    learn sqoop - sqoop tutorial - sqoop2 tutorial - sqoop option text - sqoop job - sqoop code - sqoop programming - sqoop download - sqoop examples

  • So these arguments can be saved in a single text file say option.txt.
  • While executing the command just include this file for the argument --options-file.
  • Following command shows the use of –options-file argument:
  • sqoop --options-file <path_to_option_file>/db_name --table table_name
    
    Click "Copy code" button to copy into clipboard - By wikitechy - sqoop tutorial - team

    Key points to note

  • Each argument in the option file should be on a new line.
  • -connect in option file cannot be written as --connect.
  • Same is the case for other arguments too.
  • Option file is generally used when large number of Sqoop jobs use a common set of parameters such as:
    • Source RDBMS ID, Password
    • Source database URL
    • Field Separator
    • Compression type

  • Related Searches to Sqoop saved Job