I didn't see a lot of threads that talk about using lftp to mirror seedbox downloads onto your own computer so thought I would post the method/script that I use in case it is helpful to others.
I use Linux. I don't own a Windows machine at all. I do have a mac but it seldom is used. I'm a command line guy. GUIs are fine for some things but completely unnecessary for lots of things.
This post will probably not be very helpful to you if you don't like the command line and do not have a Linux (might work on a mac) computer.
The has the following features:
- copies files to the seedbox watch directory from a local watch directory. That way, if I am browsing the web and see a torrent... I just place it into my local watch directory for the script to pick up and move it to the seedbox. I run the script periodically via cron to process the watch folders.
- mirrors completed downloads on my seedbox to my local system. It uses rtcontrol to find a list of torrents that are not complete and excludes them from mirroring. It uses lftp for mirroring, downloading 5 files at a time.. each broken into 5 segments. Again, I run the script via cron to keep my local mirror up to date.
- allows web retrieval of files via wget. If I just want to copy down one file, no need to wait for mirroring to complete
- provides handy ways to see what torrents are complete and not-complete on the seedbox
- provides a handy shortcut to ssh into the seedbox
- configuration is held in a test file in your ~/.config directory; not in the script itself. The config file is named <scriptname>.config
- If you have two (or more) seedboxes, you can either copy the script to two different names or symlink it. This allows you to have two (or more) different configuration files... enabling you to work with two seedboxes.
- running the script without any arguments provides a little help message
seedbox ssh
ssh into the
seedbox
seedbox ftp [delete|all|test]
copy all completed files from the
seedbox
optionally, 'delete' the files that have been removed from the
seedbox
optionally, download 'all' files regardless of completion status
optionally, just 'test' to see what would be done
note that delete|all|test are mutually exclusive
seedbox complete
List the files that are complete on the
seedbox
seedbox incomplete
List the files that are incomplete on the
seedbox
seedbox seed <torrent files>
Add torrent files to blackhole directory (torrents/watch)
seedbox wget <filenames>
get one or more file over http
<filenames> names of the file to get (path is relative to
https://seedbox.example.com/example/path)
seedbox info
shows the
seedbox host and credentials
seedbox edit
Edit the configuration
In order for it to all work, you need a couple of things:
- lftp needs to be installed on the local system. It should be available via all package managers.
- rtcontrol needs to be setup on the seedbox. rtcontrol is part of the Pyroscope project and installation instructions can be found at https://code.google.com/p/pyroscope/...tallFromSource
- ssh key authentication needs to be setup. There are tons of tutorials for this so I won't go into it. Make sure to just put your public key on the seedbox, NOT your private.
- copy the script to someplace on your system in your path. My script is /home/me/bin/seedbox. The first time you run it, it will create a template configuration file for you. Edit that configuration file with your values and you're good to go.
I am sure I am leaving out something important, ask me if you have any questions.
Here is the script:
Code:
#!/bin/bash
# temporary file, no need to change
TFILE="/tmp/seedbox-notcomplete"
EXCLUDES=""
NOW=`date`
# check how I was invoked
ME=`basename $0`
MEHOST="${ME} - `hostname`"
LFTPOPTS="--continue --verbose --parallel=5 --use-pget-n=5 "
CONFIGFILE=~/.config/${ME}.config
# display some usage information
usage() {
echo
echo "Needs to be invoked as one of the following:"
echo
echo "${ME} ssh"
echo " ssh into the seedbox"
echo
echo "${ME} ftp [delete|all|test]"
echo " copy all completed files from the seedbox"
echo " optionally, 'delete' the files that have been removed from the seedbox"
echo " optionally, download 'all' files regardless of completion status"
echo " optionally, just 'test' to see what would be done"
echo " note that delete|all|test are mutually exclusive"
echo
echo "${ME} complete"
echo " List the files that are complete on the seedbox"
echo
echo "${ME} incomplete"
echo " List the files that are incomplete on the seedbox"
echo
echo "${ME} seed <torrent files>"
echo " Add torrent files to blackhole directory (${WATCHDIR})"
echo
echo "${ME} wget <filenames>"
echo " get one or more file over http"
echo " <filenames> names of the file to get (path is relative to ${HTTPDIR})"
echo
echo "${ME} info"
echo " shows the seedbox host and credentials"
echo
echo "${ME} edit"
echo " Edit the configuration"
echo
}
message() {
echo "$*"
}
# source the .profile (in case we were executed from a cron job)
if [ -f ~/.profile ]; then
. ~/.profile
fi
if [ ! -z `which lockme` ]; then
. lockme
else
message "Cannot find the lockme script, running without a process lock"
message "This means that more than one instance of this script could be run"
fi
if [ ! -f ${CONFIGFILE} ]; then
# create a template configfile
cat<<EOF > ${CONFIGFILE}
# Put all the seedbox information here
# seedbox hostname
HOST=seedbox.host.com
# username
USER=username
# password (unencrypted)
PASS=password
# the home directory on the seedbox
HOMEDIR=/home/\${USER}
# The directory where completed torrents are stored (full path)
COMPLETEDIR=\$HOMEDIR/torrents/downloads
# the seedbox blackhole/watch directory (full path)
WATCHDIR=\$HOMEDIR/torrents/watch
# the HTTP download directory (https://HOST/HTTPDIR)
HTTPDIR=/dracula/downloads
# The local directory where completed torrents get downloaded to (full path)
TO=/mnt/save
EOF
message "Could not find the configfile, ${CONFIGFILE}"
message "but a template file was created."
message "Please edit ${CONFIGFILE} and rerun your command"
exit 2
else
# source the config file
. ${CONFIGFILE}
fi
# command is the second argument
if [ -z "$1" ]; then
usage
exit 1
fi
OPERATION=$1
shift
case "$OPERATION" in
"edit")
vi ${CONFIGFILE}
;;
"info")
message "From ${CONFIGFILE}:"
message "------------------"
cat ${CONFIGFILE} | grep -v "^#"
;;
"ssh")
ssh $USER@$HOST
;;
"complete")
ssh $USER@$HOST "bin/rtcontrol is_complete=yes -o name" | grep -v "^INFO "
;;
"incomplete")
ssh $USER@$HOST "bin/rtcontrol is_complete=no -o name" | grep -v "^INFO "
;;
"seed")
if [ $# -eq 0 ]; then
message "At least one argument must be supplied to seed"
usage
exit 1
fi
for var in "$@"
do
if [ -f "$var" ]; then
echo "Processing $var"
echo " .... uploading"
scp "$var" $USER@$HOST:$WATCHDIR
if [ $? -eq 0 ]; then
echo " .... success, deleting"
rm "$var"
else
echo " .... failed"
fi
fi
done
;;
"wget")
if [ $# -eq 0 ]; then
message "At least one argument must be supplied to wget"
usage
exit 1
fi
for var in "$@"
do
echo "Processing $var"
wget --user=$USER --password=$PASS --no-check-certificate https://${HOST}/${HTTPDIR}/$var
done
;;
"ftp")
if [ $# -gt 1 ]; then
message "${OPERATION} only accepts one argument"
usage
exit 1
fi
echo "Starting lftp sync"
if [ ! -d "${TO}" ]; then
echo "Target directory, ${TO}, does not exist. Exiting"
exit 1
fi
STATUSFILE="${TO}/${MEHOST} - ${NOW}"
touch "${STATUSFILE}"
# first get the list of non-complete torrents
if [ -f "${TFILE}" ]; then
rm "${TFILE}"
fi
# if we're not getting all files, build the list of files to NOT get
if [ "$1" != "all" ]; then
# get the list of incomplete files
ssh $USER@$HOST "bin/rtcontrol is_complete=no -o name" | grep -v "INFO " > $TFILE
# if we got back a list, add each line to the excludes list for rsync
if [ -f "${TFILE}" ]; then
# note that any 'odd' characters in this file cause problems, try escaping them
sed -i 's/\[/\\\[/g' $TFILE
sed -i 's/\]/\\\]/g' $TFILE
sed -i 's/(/\\(/g' $TFILE
sed -i 's/)/\\)/g' $TFILE
sed -i 's/&/\\&/g' $TFILE
sed -i "s/'/\\'/g" $TFILE
while read -r line
do
EXCLUDES="$EXCLUDES -x \"${line}\" "
done < "${TFILE}"
fi
fi
if [ "$1" = "delete" ]; then
LFTPOPTS="$LFTPOPTS --delete "
fi
if [ "$1" = "test" ]; then
LFTPOPTS="$LFTPOPTS --dry-run "
fi
MIRRORCMD="mirror ${LFTPOPTS} ${EXCLUDES} ${COMPLETEDIR} ${TO} ; bye"
echo $MIRRORCMD
lftp -e "${MIRRORCMD}" -u $USER,$PASS sftp://$HOST
# if we excluded files, print a message
if [ -s "${TFILE}" ]; then
echo
echo "Note: Some files were excluded because they were incomplete"
cat "${TFILE}"
rm "${TFILE}"
fi
rm "${STATUSFILE}"
;;
*)
message "An invalid command was specified: $OPERATION"
usage
exit 1
;;
esac
exit 0