ARC server-side logging

This page is meant to evolve into a set of guidelines for writing logs that are useful for sysadmins as well as developers. For now it is just a place to collect the current state of logs in ARC and come with suggestions on how to improve the logs.

Overview of current server side logs in ARC

Tables

Log usefulness

log name	sensibly formatted	has useful information	Is logging sufficient at default level? (%)	Is loging lacking at default level? (%)	Comments
cache-clean.log				new default name|
grid-manager.log	yes	yes
ws-interface.log
gm-jobs.log
jura.log
job.[id].errors
gridftpd.log
inforegistration.log
infoprovider.log
bdii-update.log
nordugridmap.log
acix-cache.log
gangliarc.log
job.logger.errors
job.helper.errors	no	yes			Bugzilla #3486

Full overview

Is OK

arc.conf.reference name

Conf block

Conf name

Log level conf name

Log levels

Default location

Default log level

ARC Logger long format

Has system log rotation

Log format

timestamp format

Source

cache-clean.log

grid-manager

cachelogfile

cacheloglevel

0-5

/var/log/arc/cache-clean.log

3

Y

[time] who: level: msg

YYYY-MM-DD hh:mm:ss

cashe-clean

Y

grid-manager.log

grid-manager

logfile

debug

0-5

/var/log/arc/grid-manager.log

2

Y

Y

[time][who][level][pid/thread]msg

YYYY-MM-DD hh:mm:ss

arched

Y

ws-interface.log

grid-manager

wslogfile

debug

0-5

/var/log/arc/ws-interface.log

2

Y

Y

[time][who][level][pid/thread]msg

YYYY-MM-DD hh:mm:ss

arched

gm-jobs.log

grid-manager

joblog

?

?

no

?

time msg

YYYY-MM-DD hh:mm:ss

arched

jura.log

grid-manager

jobreport_logfile

?

?

controldir/job.logger.errors

?

level: msg

YYYY-MM-DD hh:mm:ss

jura

data-staging

debug

0-5

controldir/job.[id].errors

2

[time][level][pid/thread] who msg

YYYY-MM-DD hh:mm:ss

backends, data, arched

dtrstate.log

data-staging

dtrlog

?

?

controldir/dtrstate.log

-

msg

-

gridftpd.log

gridftpd

logfile

debug

0-5

/var/log/arc/gridftpd.log

3

Y

Y

[time][who][level][pid/thread]msg

YYYY-MM-DD hh:mm:ss

infosys

?

debug

0-1

?

0?

infosys

?

slapd_loglevel

0-?

?

?

inforegistration.log

infosys

registrationlog

?

?

/var/log/arc/inforegistration.log

?

Y

?

Thu Mar 19 11:49:24 CET 2015

infoprovider.log

infosys

providerlog

provider_loglevel

0-5

/var/log/arc/infoprovider.log

1?

Y

[time] who: level: msg

YYYY-MM-DD hh:mm:ss

bdii-update.log

infosys

bdii_log_dir

bdii_debug_level

ERROR | WARNING | INFO | DEBUG

/var/log/arc/bdii/bdii-update.log

INFO, but InfosysHelper hardcoded to VERBOSE

Y

[time] who: level: msg

YYYY-MM-DD hh:mm:ss

bdi and infosyshelper

nordugridmap.log

nordugridmap

logfile

debug

0-5

/var/log/arc/nordugridmap.log

2?

msg

-

acix-cache.log

acix/cacheserver

logfile

?

?

/var/log/arc/acix-cache.log

?

time [what] msg

YYYY-MM-DD hh:mm:ss+tz00

gangliarc

logfile?

-

0-5

/var/log/arc/gangliarc.log

3

time [level] msg

YYYY-MM-DD hh:mm:ss,mmm

controldir/job.helper.errors

?

[time] who: msg

YYYY-MM-DD hh:mm:ss

Changes to be made

• one logfile per configuration block, config parameter name "logfile": this will require restructuring arc.conf (e.g. new blocks, subblocks for jura, cache)
• WHO from the grid-manager logs should be documented on a wiki page
• unify jobid-s in the grid-manager.log, ws-interface.log
• change job.id.errors to job.id.log
• rename dtrstate.log dtr.state, conf parameter statefile instead of dtrlog
• infosys/debug: remove
• inforegistration.log: will dissappear together with the registration functionality, no need to fix this log
• nordugridmap: sync with the agreed format, to be enabled by default
• job.logger.errors is the jobreport_logfile default location in controldir: move to var/log/arc
• job.helper.errors: move to var/log/arc, own confiuration parameter helperlog in gridmanager block

TODO: ARC_server-side_logging-TODO

Log levels

Ideally all log messages in ARC should fall into one of the following categories (in decreasing rank):

• DEBUG: Detailed information, typically of interest only when diagnosing problems.
• VERBOSE: Like DEBUG, but less verbose? (Do we need this level?)
• INFO: Confirmation that things are working as expected.
• WARNING: An indication that something unexpected happened, or indicative of some problem in the near future (e.g. ‘disk space low’). The software is still working as expected.
• ERROR: Due to a more serious problem, the software has not been able to perform some function.
• FATAL: A serious error, indicating that the program itself will be unable to continue running.

To configure the level of logging for a logfile either a number between 0 (FATAL) and 5 (DEBUG) or the name of the lowest category should be used.

Log configuration

All logs should be configurable and have a default value. All logs should have a configurable log level.

Log format

All log messages should at least have a timestamp. The preferred log format is the ARC::LongFormat

[time][who][level][pid/thread]msg

Example:

[2015-06-01 03:26:03] [Arc.JobDescriptionParserPlugin] [INFO] [25456/3] String successfully parsed as nordugrid:xrsl

Log rotation

All logs will grow in time. Sysadmins with disks running full because of big logs are not happy sysadmins. All ARC logs should have corresponding system logrotate rules by default.