NOTUR2011Workshop

ARC Administrator's Tutorial

Given at NOTUR 2011: The 10th Annual Meeting on High Performance Computing and Infrastructure in Norway

Part 1. Installation and basic configuration of ARC

We have 10 servers in the Amazon cloud to use as a testbed for ARC installation. Get the name of your server according to the number the lecturers gave you.

1. ec2-50-19-14-21.compute-1.amazonaws.com
2. ec2-50-17-20-208.compute-1.amazonaws.com
3. ec2-75-101-173-97.compute-1.amazonaws.com
4. ec2-174-129-127-246.compute-1.amazonaws.com
5. ec2-50-17-109-125.compute-1.amazonaws.com
6. ec2-184-73-85-65.compute-1.amazonaws.com
7. ec2-50-19-18-92.compute-1.amazonaws.com
8. ec2-184-72-211-122.compute-1.amazonaws.com
9. ec2-50-19-78-180.compute-1.amazonaws.com
10. ec2-50-16-115-182.compute-1.amazonaws.com

export CLOUD_HOSTNAME=<your_hostname>

Now, get a private key to log in to a virtual server in Amazon cloud.

wget http://folk.uio.no/dmytrok/NOTUR2011/cloudkey.pem
# Make sure ssh will not complain
chmod 600 cloudkey.pem

Now you can login there, using ec2-user as your account, and become root

bash-3.2$ ssh -i cloudkey.pem ec2-user@$CLOUD_HOSTNAME
[ec2-user@ip-10-112-66-229 ~]$ sudo -i

The first thing a grid server needs is a host certificate to identify itself in the grid. To get a proper certificate, you need to apply to your local Certification Authority and follow their rules. For the sake of tutorial, we can use temporary host certificates. Download it according to the number you've got from the lecturers and place it under the default location, which is /etc/grid-security/ folder. Note that the package contains 2 files: the host certificate itself (hostcert.pem) and a secret corresponding key (hostkey.pem). Hostkey has to have very restrictive permissions on it.

# Suppose your number is 1
[root@ip-10-112-66-229 ~]# wget http://folk.uio.no/dmytrok/NOTUR2011/certs/hostcerts/host1.tgz
[root@ip-10-112-66-229 ~]# tar xzf host1.tgz
[root@ip-10-112-66-229 ~]# mkdir /etc/grid-security/
[root@ip-10-112-66-229 ~]# cp hostcert.pem /etc/grid-security/
[root@ip-10-112-66-229 ~]# cp hostkey.pem /etc/grid-security/
[root@ip-10-112-66-229 ~]# chmod 644 /etc/grid-security/hostcert.pem
[root@ip-10-112-66-229 ~]# chmod 400 /etc/grid-security/hostkey.pem

Now, having a host certificate is only a half way. In order to communicate with other resources and users on the grid your resource has to trust the CA that signed the certificates for these users/resources. And they have to trust your CA as well. "Trusting" is usually enabled by placing the CA certificate and couple of corresponding files under the special directory. Usually it's done automatically by installation of proper packages, but since we're using a special temporary CA, we have to do it manually now (we do want to trust our own CA, don't we?):

[root@ip-10-112-66-229 ~]# wget http://folk.uio.no/dmytrok/NOTUR2011/certs/cacerts/cacerts.tgz
[root@ip-10-112-66-229 ~]# tar xzf cacerts.tgz
[root@ip-10-112-66-229 ~]# mkdir /etc/grid-security/certificates
[root@ip-10-112-66-229 ~]# cp cacerts/* /etc/grid-security/certificates/
[root@ip-10-112-66-229 ~]# chmod 644 /etc/grid-security/certificates/*

Now we are ready to start the ARC installation itself. If you use your laptop (or when you'll come back later to installation of ARC on your resources), use Instructions on installation through repositories. If you're in clouds, you can just follow this wiki-page further.

Have no fear, ignore all warnings and answer "yes" to yum's questions!

Setup the nordugrid repo first:

[root@ip-10-112-66-229 ~]# cd /etc/yum.repos.d/
# Import the key with which the packages are signed
[root@ip-10-112-66-229 yum.repos.d]# rpm --import http://download.nordugrid.org/RPM-GPG-KEY-nordugrid
# Enable the nordugrid repository itself, using release and architecture for this particular Amazon Linux.
[root@ip-10-112-66-229 yum.repos.d]# cat <<EOF > nordugrid.repo
[nordugrid]
name=NorduGrid - x86_64 - base
baseurl=http://download.nordugrid.org/repos/11.05/redhat/el5/x86_64/base
enabled=1
gpgcheck=1
gpgkey=http://download.nordugrid.org/RPM-GPG-KEY-nordugrid

[nordugrid-updates]
name=NorduGrid - x86_64 - updates
baseurl=http://download.nordugrid.org/repos/11.05/redhat/el5/x86_64/updates
enabled=1
gpgcheck=1
gpgkey=http://download.nordugrid.org/RPM-GPG-KEY-nordugrid

[nordugrid-testing]
name=NorduGrid - x86_64 - testing
baseurl=http://download.nordugrid.org/repos/11.05/redhat/el5/x86_64/testing
enabled=0
gpgcheck=1
gpgkey=http://download.nordugrid.org/RPM-GPG-KEY-nordugrid
EOF
# For RedHat-based systems (Amazon Linux is one) EPEL repo needs to be set up to resolve all dependencies
[root@ip-10-112-66-229 yum.repos.d]# cd ~
[root@ip-10-112-66-229 ~]# wget http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
[root@ip-10-112-66-229 ~]# rpm -i epel-release-5-4.noarch.rpm
# Now, Amazon Linux by default is missing some dependencies ARC will complain about. Let's prevent this.
[root@ip-10-112-66-229 ~]# yum install libtool-ltdl
[root@ip-10-112-66-229 ~]# yum install libxslt
[root@ip-10-112-66-229 ~]# wget ftp://ftp.muug.mb.ca/mirror/centos/5.6/os/x86_64/CentOS/xmlsec1-1.2.9-8.1.1.x86_64.rpm
[root@ip-10-112-66-229 ~]# rpm -i xmlsec1-1.2.9-8.1.1.x86_64.rpm
[root@ip-10-112-66-229 ~]# yum install openssl098e
[root@ip-10-112-66-229 ~]# wget ftp://ftp.muug.mb.ca/mirror/centos/5.6/os/x86_64/CentOS/xmlsec1-openssl-1.2.9-8.1.1.x86_64.rpm
[root@ip-10-112-66-229 ~]# rpm -i xmlsec1-openssl-1.2.9-8.1.1.x86_64.rpm
# Amazon Linux peculiars -- we have to install libltdl.so.3 in parallel with libltdl.so.7
[root@ip-10-112-66-229 ~]# wget ftp://ftp.muug.mb.ca/mirror/centos/5.6/os/x86_64/CentOS/libtool-ltdl-1.5.22-7.el5_4.x86_64.rpm
[root@ip-10-112-66-229 ~]# rpm -i --force libtool-ltdl-1.5.22-7.el5_4.x86_64.rpm
# Finally, we got to nordugrid repo. Install first the packages with respectful CA certificates (remember, we mentioned it?)
[root@ip-10-112-66-229 ~]# yum install ca_*
# Each CA should maintain the list of revoked certificates. It has to be regularly updated. It's done by fetch-crl tool.
[root@ip-10-112-66-229 ~]# yum install fetch-crl
# Installation of fetch-crl has created a default cron job. But for the first time we have to run it manually to be ready for the first launch of ARC
# You may want to go for a cup of coffee during this. It takes time.
[root@ip-10-112-66-229 ~]# /usr/sbin/fetch-crl -r 10 -a 24
# At last! We can install the ARC server itself!
[root@ip-10-112-66-229 ~]# yum install nordugrid-arc-compute-element

Now we're coming to the configuration

# Create a spool directory, that will contain the information about all jobs
[root@ip-10-112-66-229 ~]# mkdir -p /var/spool/arc
# Download the simplest configuration file
[root@ip-10-112-66-229 ~]# wget http://folk.uio.no/dmytrok/NOTUR2011/conf/arc.conf
[root@ip-10-112-66-229 ~]# cp arc.conf /etc/

Looking for more options? We will cover some of them later. Description for every possible option you can find:

• Locally: /usr/share/arc/examples/arc.conf.template
• On-line: here

Continue with the configuration. We will need a special account that will run grid jobs locally.

# Create a grid user
[root@ip-10-112-66-229 ~]# adduser grid

Now, we will want to test our installation, won't we? Let's be our own users!

# Create a user for testing the grid
[root@ip-10-112-66-229 ~]# adduser gridtestuser
# Install an ARC client for test
[root@ip-10-112-66-229 ~]# yum install nordugrid-arc-client
# A grid user needs a certificate
[root@ip-10-112-66-229 ~]# su - gridtestuser
# Again, use the number you got from the lecturers
[gridtestuser@ip-10-112-66-229 ~]$ wget http://folk.uio.no/dmytrok/NOTUR2011/certs/usercerts/user1.tgz
[gridtestuser@ip-10-112-66-229 ~]$ tar xzf user1.tgz
[gridtestuser@ip-10-112-66-229 ~]$ mkdir .globus
[gridtestuser@ip-10-112-66-229 ~]$ mv usercert.pem userkey.pem .globus/
[gridtestuser@ip-10-112-66-229 ~]$ chmod 644 .globus/usercert.pem
[gridtestuser@ip-10-112-66-229 ~]$ chmod 400 .globus/userkey.pem
# Back to root
[gridtestuser@ip-10-112-66-229 ~]$ exit

Go!

# Start the ARC services
[root@ip-10-112-66-229 ~]# /etc/init.d/grid-infosys start
[root@ip-10-112-66-229 ~]# /etc/init.d/gridftpd start
[root@ip-10-112-66-229 ~]# /etc/init.d/a-rex start
# Create grid-mapfile, a special file which contains names of the certificates that are allowed to access this resource
[root@ip-10-112-66-229 ~]# touch /etc/grid-security/grid-mapfile
# Add a test user to the grid-mapfile. Use your number again.
[root@ip-10-112-66-229 ~]# echo "\"/DC=eu/DC=KnowARC/O=NT2011/CN=demo1\" grid" >> /etc/grid-security/grid-mapfile

Let's go to tests. To make things simpler, pick up the internal hostname of your server in the Amazon cloud.

1. domU-12-31-39-10-26-31.compute-1.internal
2. ip-10-72-74-187.ec2.internal
3. ip-10-111-59-168.ec2.internal
4. domU-12-31-39-16-BD-99.compute-1.internal
5. ip-10-64-73-47.ec2.internal
6. ip-10-64-11-96.ec2.internal
7. domU-12-31-39-00-0C-52.compute-1.internal
8. ip-10-72-41-175.ec2.internal
9. ip-10-72-45-152.ec2.internal
10. domU-12-31-39-00-44-F2.compute-1.internal

# Become a test user
[root@ip-10-112-66-229 ~]# su - gridtestuser
[gridtestuser@ip-10-112-66-229 ~]$ export HOSTNAME=<internal_hostname>
# Get a user proxy. Password is on the board.
[gridtestuser@ip-10-112-66-229 ~]$ arcproxy
# Check the proxy
[gridtestuser@ip-10-112-66-229 ~]$ arcproxy -I
# Check health state
[gridtestuser@ip-10-112-66-229 ~]$ arcinfo $HOSTNAME
# Check if you are authorized by trying to list the files
[gridtestuser@ip-10-112-66-229 ~]$ arcls gsiftp://$HOSTNAME
# Finally, submit a simple job
[gridtestuser@ip-10-112-66-229 ~]$ arcsub -e '&(executable="/bin/echo")(arguments="NOTUR says Hello to Grid")(stdout="hello.txt")' -c $HOSTNAME
# Check the state of the job
[gridtestuser@ip-10-112-66-229 ~]$ arcstat -a
# If it's finished -- get the results
[gridtestuser@ip-10-112-66-229 ~]$ arcget -a
# Check the hello.txt file that must have appeared in the new directory in your home catalogue

Problems? Really?? Then check the logs under /var/log/arc/ (default, can be configured through arc.conf).

If all the problems are solved, then you may ask your neighbour to put the contents of your grid-mapfile into his and submit a job to his resource. And vice versa.

Part 2. Advanced activities

Caching of input files

Add configuration to /etc/arc.conf:

[grid-manager]
cachedir="/var/spool/arc/cache"

Restart the a-rex:

[root@ip-10-112-66-229 ~]# /etc/init.d/a-rex restart

Become a user again and continue.

[root@ip-10-112-66-229 ~]# su - gridtestuser
# Download a description of a test job
[gridtestuser@ip-10-112-66-229 ~]$ wget http://folk.uio.no/dmytrok/NOTUR2011/tasks/cachejob.xrsl

Now submit this job more than once:

[gridtestuser@ip-10-112-66-229 ~]$ arcsub -c $HOSTNAME cachejob.xrsl

Check the cache directory manually (as root):

[gridtestuser@ip-10-112-66-229 ~]$ exit
[root@ip-10-112-66-229 ~]# ls -lR /var/spool/arc/cache

Enabling the web interface

Edit /etc/arc.conf

[grid-manager]
arex_mount_point="https://$HOSTNAME/arex"

Restart services, become a regular user and try to submit a previous job to the web-interface.

[gridtestuser@ip-10-112-66-229 ~]$ arcinfo ARC1:$HOSTNAME/arex
[gridtestuser@ip-10-112-66-229 ~]$ arcsub -c ARC1:$HOSTNAME/arex cachejob.xrsl

Automatic grid-mapfile creation

The nordugridmap tool can populate the grid map file from several sources. As an example, let's add the support for ARC demo VO. Edit arc.conf.

[vo]
id="demovo"
vo="demovo"
file="/etc/grid-security/grid-mapfile"
source="http://www.nordugrid.org/demousers.dn"
mapped_unixid="grid"

Run nordugridmap and check the output

[root@ip-10-112-66-229 ~]# nordugridmap 
[root@ip-10-112-66-229 ~]# cat /etc/grid-security/grid-mapfile

Did you notice, that your test user went missing? Yes, grid-mapfile is fully rewritten by nordugridmap. As an exercise: try now to make sure your test user will be there every time you run nordugridmap. Hint: you can user local files with file:/// prefix as the source to the VO

Check documentation to learn about the several ways and options of configuring authorization

Good Luck!

Please, explore ARC further: [1]