[Solved-2 Solutions] Copying file from s3:// to local file system ?



Problem:

How to Copying file from s3:// to local file system ?

Solution 1:

  • We should use the s3n protocol (unless you stored the files on s3 using the s3 protocol) - s3 is used for block storage (i.e. similar to hdfs, only on s3) and s3n is for native s3 file system

Here is an example

raw_logs = LOAD 's3n://access:[email protected]/input/access_log_1' USING TextLoader AS (line:chararray);

Solution 2:

  • cp in unaware of s3
cp s3://my-bucket-name/path/to/file 
  • We can also use the below one:
s3cmd get s3://some-s3-bucket/some-s3-folder/local_file.ext ~/local_dir/ 

Another example:

  s3cmd get s3://BUCKET/OBJECT LOCAL_FILE
Get file from bucket

Related Searches to Copying file from s3:// to local file system