Skip to end of metadata
Go to start of metadata

(https://confluence.xsede.org/display/XT/GridFTP+Logging+Upload+Configuration)
Each XSEDE SP should be logging all transfer activity, and uploading all XSEDE-related transfer activity to the central GridFTP logging infrastructure (currently hosted at TACC). There are three steps to set up the automated upload of logs to the upload location: Setting up an account with access to the GridFTP logs and the upload location, setting up log rotation for your GridFTP transfer logs (not mandatory but desirable for ease of management), and setting up daily log uploads. 

 

Account access to upload logs

You will need an account with access to the GridFTP log files which will be used for uploading these logs to the log collection server. The easiest way to do this is simply to configure your "globus" user (sometimes called "globust" or "gridftp") to perform the uploads, but you may choose to use a Unix Group setup or some other mechanism to allow a specific user account to access the logs for upload. This is at the discretion of the SP. Once you have selected a user account, you will need to create an SSH RSA keypair for configuring authentication to the upload server from this account. This key can be created using the command: 

$ ssh-keygen -t rsa 

If you intend to use this key for automated uploads, leave the passphrase empty. Once you have created the keypair, the public key will be stored in ~/.ssh/id_rsa.pub ; copy the contents of this file into e-mail and send it to siva@tacc.utexas.edu to be granted access to the upload server. When you have received confirmation you may proceed with testing uploads to the central logging server. 

Setting up Log Rotation (optional)

Most systems will use the "logrotate" software tool to manage the rotation of system logs, and this tool can easily be extended to support rotation of the GridFTP transfer logs. You will need to know the location and name of these transfer logs, and you will need to decide on the frequency of log rotation. You may need to consult your system-specific documentation for specifics on how to configure certain options for log rotation. On CentOS/RedHat-based systems, you can create a file named /etc/logrotate.d/gridftp with the following contents to configure daily log rotation (this example assumes you are following the examples from the XSEDE GridFTP installation guide. Replace /var/log/gridftp/ with the actual path to your GridFTP transfer logs if you are using a different location): 

/var/log/gridftp/*.log {
daily
missingok
notifempty
compress
create 644 root root
postrotate
# CentOS 7
#systemctl restart xinetd.service
# CentOS 6
service xinetd restart
endscript
}

Setting up Daily log uploads

Regardless of whether you are using log rotation, SPs should upload logs for each of their GridFTP server instances on a daily basis to ensure that the XSEDE centralized transfer information service has fresh data. Assuming that your server is configured to rotate logs on a daily basis, you can use the following script in combination with cron to accomplish this daily upload. You will need to modify the script to set "SITENAME" which you will be given when you exchange your SSH key with Siva, and to ensure that you are selecting the right log location from which to copy files. 

 

#!/bin/bash
set -x

# Go to gridftp transfer log location
# Replace /var/log/gridftp with the appropriate directory, if a different location is used.
cd /var/log/gridftp/

# Get date ("YYMMDD")
A=`date +"%Y%m%d"`
# Set date to previous day
let A=A-1

# Get and parse hostname and system ("data1.stampede")
B=`hostname  -f | awk -F\. '{print $1 "." $2 }'`

# Get and parse resource name ("tacc")
C=`hostname  -f | awk -F\. '{print $3 }'`

# Get and parse hostname  ("data1")
D=`hostname  -f | awk -F\. '{print $1}'`

# Debug
echo $A $B $C $D

#Create a file transfer.txt that contains command for SFTP transfers
## The file contains the following line

echo cd /sitename/ > transfer.txt
echo put $D.transfer.log $B.transfer.log >> transfer.txt

echo exit

# Copy log file to gridftp transfer log location
#For sites using logrotate:
sftp -b transfer.txt globust@s254-108.wrangler.tacc.utexas.edu

#Sites not using logrotate, You can just start here 

#Create a file transfernonlog.txt that contains command for SFTP transfers for the log files
## The file contains the following line

echo cd /sitename/ > transfernonlog.txt
echo put $D.transfer.log $B.transfer.log >> transfernonlog.txt

echo exit

# Copy log file to gridftp transfer log location

sftp -b transfernonlog.txt globust@s254-108.wrangler.tacc.utexas.edu

An example crontab entry to execute this script shortly after midnight each day would look like the following: 

05 1 * * * /home/globust/gridftp-log.cron >/dev/null 2>&1