Child pages
  • GridFTP logging upload to centralized log collector configs and steps
Skip to end of metadata
Go to start of metadata

 

You will need an account with access to the GridFTP log files which will be used for uploading these logs to the log collection server. The easiest way to do this is simply to configure your "globus" user (sometimes called "globust" or "gridftp") to perform the uploads, but you may choose to use a Unix Group setup or some other mechanism to allow a specific user account to access the logs for upload. Once you have decided on a user account to perform the griftp transfers, make sure that that account is available at Wrangler at TACC as well. If this is not the case, please create an account for the same user through the XSEDE User Portal, requesting access to the Wrangler resource. Also, finalize with Wrangler admins on where the files should be transferred to on Wrangler.


Setting up Log Rotation


Most systems will use the "logrotate" software tool to manage the rotation of system logs, and this tool can easily be extended to support rotation of the GridFTP transfer logs. You will need to know the location and name of these transfer logs, and you will need to decide on the frequency of log rotation. You may need to consult your system-specific documentation for specifics on how to configure certain options for log rotation. On CentOS/RedHat-based systems, you can create a file named /etc/logrotate.d/gridftp with the following contents to configure daily log rotation (this example assumes you are following the examples from the XSEDE GridFTP installation guide. Replace /var/log/gridftp/ with the actual path to your GridFTP transfer logs if you are using a different location):


/var/log/gridftp/*.log {
       daily

       missingok

       compress

       rotate 60

       notifempty

       create 644 root root

       prerotate

           sudo -u {transfers_usersname}  {path_to_transfer_script}

       endscript

       postrotate

         systemctl restart globus-gridftp-server.service

       endscript


}

Here, transfers_username is the XSEDE account that is used to transfer files from your site to TACC’s Wrangler. The {path_to_transfer_script} is the script that will be used to transfer scripts using gridftp from your site to Wrangler.  The script is outlined in the section below.

Setting up Daily log uploads

Regardless of whether you are using log rotation, SPs should upload logs for each of their GridFTP server instances on a regular (monthly is prefered) basis to ensure that the XSEDE centralized transfer information service has fresh data. Assuming that your server is not configured to rotate logs on a monthly basis, you can use the following script in combination with cron to accomplish this monthly upload. This is also the script that is referenced in the prerotate command above to transfer files while using logrotate.


cat /nics/c/home/xsede/nics_dtn/transfergridftplog.sh

#!/bin/bash

 

# specify where the user certificate will be stored

export X509_USER_PROXY=/nics/c/home/xsede/nics_dtn/x509up_u94

 

# get the logfile name

logfile=$(ls /var/log/gridftp | grep 'datamover[0-9].log$')

 

# get current date to append to filename that is transferred

datename=`date +%F`

 

# transfer log file to Wrangler location using gridftp

globus-url-copy /var/log/gridftp/$logfile gsiftp://gridftp.wrangler.tacc.utexas.edu/data/projects/xsede/gridftp-stats/nics/$logfile"_"$datename

  • No labels