Tip #1 [ftp]: Use of macros to automate file uploads

Let's suppose you have a directory structure of this type

current working directory
    |
    +--- 2000 --- 2000_file1.txt

    |             2000_file2.txt

    |             2000_file3.txt
    |             .
    |             .
    |             .
    +--- 2001 --- 2001_file1.txt

    |             2001_file2.txt

    |             2001_file3.txt
    |             .
    |             .
    |             .

    |     .
    |     .
    |     .
    +--- 2014 --- 2014_file1.txt

                  2014_file2.txt

                  2014_file3.txt
                  .
                  .
                  .

                                                                    

Let's assume now that you have to upload all these files into an ftp server (ftp.site.com) that has the following directory structure on the home directory when you login (with username "myusername" and password "1234"):

user_remote_home_directory
    |
    +--- 2000 

    | 
    +--- 2001 

    |     .
    |     .
    |     .
    +--- 2014 

If the number of files (or directories) is small, you can do it manually with the put or mput commands, but as soon as the number of files increase, this approach is not an option anymore: you might not have the time to do it this way. In those cases, use macros.

We will be using Linux bash for this task. You will iterate over the local folders and opening and closing ftp sessions whenever you enter in each directory. In order to avoid having to login each time (you want to automate the login as well), create the file .netrc in your home directory (ftp will use it):

> cd ~
> touch .netrc

and add the following line

# Autologin
ftp.site.com myusername 1234

This line will make the autologin on ftp.site.com with your user account without FTP prompting you every time you open a session. Then, add the following lines to the .netrc file:

macdef siteput
prompt
binary
hash
cd $1
mput *
quit

The macdef statement is used to define an FTP macro (in this case we have named it siteput). This macro does the following

1. prompt: first deactivate the prompt. Since we are using mput, we do not want confirmation to upload every single file (that is the whole point of this, right?)
2. binary: we are enforcing binary upload (instead of ASCII)
3. hash:  we want some sort of upload progress, that is what hash is used for
4. cd $1: change remote folder, specified by the argument passed to the macro (see below)
5. mput *: Proceed to upload all the files of the local folder.
6. quit: close ftp session

Now this macro would be called using a command like the following:

echo "$ macro_name arg1 arg2 ... argN" | ftp ftp.site.com

In our particular case, we will be looping over all our local folders and calling the macro for the corresponding remote folder

for year in `year 2000 2014`
do
    # Informative printout
    echo $year

    # We now call the macro for the current folder/year using
    # a sub-shell (note the use of parenthesis) to avoid 
    # the need of executing "cd .." after the command has been
    # launched (i.e. the "cd $year" affects the sub-shell, not
    # the main shell where this loop takes place).
    ( cd $year; echo "$ siteput $year" | ftp ftp.site.com )

done