How Do I Know if File Is Uploaded Completely to S3
As you can read in this article, I recently had some trouble with my email server and decided to outsource email administration to Amazon's Simple Email Service (SES).
The problem with that solution was that I had SES save new messages to an S3 saucepan, and using the AWS Management Console to read files within S3 buckets gets stale actually fast.
So I decided to write a Bash script to automate the process of downloading, properly storing, and viewing new messages.
While I wrote this script for use on my Ubuntu Linux desktop, it wouldn't crave too much trivial to make information technology work on a macOS or Windows 10 system through Windows SubSystem for Linux.
Here'due south the complete script all in ane piece. After you take a few moments to look it over, I'll walk you through it 1 step at a time.
#!/bin/bash # Retrieve new messages from S3 and save to tmpemails/ directory: aws s3 cp \ --recursive \ s3://bucket-name/ \ /home/david/s3-emails/tmpemails/ \ --profile myaccount # Set location variables: tmp_file_location=/habitation/david/s3-emails/tmpemails/* base_location=/abode/david/s3-emails/emails/ # Create new directory to store today's messages: today=$(date +"%m_%d_%Y") [[ -d ${base_location}/"$today" ]] || mkdir ${base_location}/"$today" # Requite the message files readable names: for FILE in $tmp_file_location do mv $FILE ${base_location}/${today}/email$(rand) done # Open new files in Gedit: for NEWFILE in ${base_location}/${today}/* do gedit $NEWFILE done We'll brainstorm with the unmarried command to download any letters currently residing in my S3 bucket (past the fashion, I've inverse the names of the bucket and other filesystem and authentication details to protect my privacy).
aws s3 cp \ --recursive \ s3://bucket-proper name/ \ /home/david/s3-emails/tmpemails/ \ --profile myaccount Of course, this volition only work if you lot've already installed and configured the AWS CLI for your local arrangement. Now'due south the time to do that if you haven't already.
The cp command stands for "copy," --recursive tells the CLI to use the operation even to multiple objects, s3://bucket-proper name points to my bucket (your bucket proper name will patently be different), the /home/david... line is the accented filesystem address to which I'd like the messages copied, and the --profile argument tells the CLI which of my multiple AWS accounts I'grand referring to.
The next section sets ii variables that volition brand information technology much easier for me to specify filesystem locations through the rest of the script.
tmp_file_location=/habitation/david/s3-emails/tmpemails/* base_location=/habitation/david/s3-emails/emails/ Note how the value of the tmp_file_location variable ends with an asterisk. That'southward because I desire to refer to the files within that directory, rather than the directory itself.
I'll create a new permanent directory within the .../emails/ hierarchy to make it easier for me to find messages later. The proper noun of this new directory will be the current date.
today=$(date +"%m_%d_%Y") [[ -d ${base_location}/"$today" ]] || mkdir ${base_location}/"$today" I starting time create a new shell variable named today that will be populated by the output of the date +"%m_%d_%Y" command. date itself outputs the full date/timestamp, but what follows ("%m_%d_%Y") edits that output to a simpler and more readable format.
I then test for the existence of a directly using that name - which would betoken that I've already received emails on that day and, therefore, there'south no demand to recreate the directory. If such a directory does not exist (||), then mkdir will create information technology for me. If you don't run this exam, your command could return annoying error letters.
Since Amazon SES gives ugly and unreadable names to each of the messages it drops into my S3 bucket, I'll at present dynamically rename them while, at the aforementioned time, moving them over to their new home (in the dated directory I but created).
for FILE in $tmp_file_location practise mv $FILE ${base_location}/${today}/email$(rand) washed The for...practise...done loop will read each of the files in the directory represented by the $tmp_file_location variable and then move it to the directory I just created (represented by the $base_location variable in addition to the current value of $today).
As part of the same operation, I'll requite it its new name, the cord "electronic mail" followed by a random number generated by the rand control. You may need to install a random number generator: that'll be apt install rand on Ubuntu.
An earlier version of the script created names differentiated past shorter, sequential numbers that were incremented using a count=ane...count=$((count+i)) logic inside the for loop. That worked fine equally long as I didn't happen to receive more than than 1 batch of letters on the same day. If I did, and so the new letters would overwrite older files in that day's directory.
I gauge it'due south mathematically possible that my rand control could assign overlapping numbers to two files but, given that the default range rand uses is between 1 and 32,576, that'south a gamble I'm willing to take.
At this bespeak, at that place should be files in the new directory with names like email3039, email25343, etc. for each of the new messages I was sent.
Running the tree control on my own system shows me that five messages were saved to my 02_27_2020 directory, and 1 more than to 02_28_2020 (these files were generated using the older version of my script, and then they're numbered sequentially).
There are currently no files in tmpemails - that'south considering the mv command moves files to their new location, leaving nix behind.
$ tree . ├── emails │ ├── 02_27_2020 │ │ ├── email1 │ │ ├── email2 │ │ ├── email3 │ │ ├── email4 │ │ ├── email5 │ └── 02_28_2020 │ └── email1 └── tmpemails The terminal section of the script opens each new message in my favorite desktop text editor (Gedit). Information technology uses a similar for...practise...washed loop, this fourth dimension reading the names of each file in the new directory (referenced using the "today" command) and then opening the file in Gedit. Note the asterisk I added to the cease of the directory location.
for NEWFILE in ${base_location}/${today}/* do gedit $NEWFILE done There's nevertheless one more matter to exercise. If I don't make clean out my S3 bucket, information technology'll download all the accumulated messages each fourth dimension I run the script. That'll go far progressively harder to manage.
So, later successfully downloading my new letters, I run this curt script to delete all the files in the bucket:
#!/bin/fustigate # Delete all existing emails aws s3 rm --recursive s3://saucepan-name/ --contour myaccount Learn to code for gratuitous. freeCodeCamp's open source curriculum has helped more than 40,000 people get jobs as developers. Get started
Source: https://www.freecodecamp.org/news/bash-script-download-view-from-s3-bucket/
0 Response to "How Do I Know if File Is Uploaded Completely to S3"
Postar um comentário