Automated Statistics Page for Human Rights Library

helmetpastoralSoftware and s/w Development

Dec 13, 2013 (4 years and 7 months ago)

77 views


Human Rights Library

helmetpastoral_f5ef9532
-
980c
-
48a0
-
8fc8
-
b357420f4229.doc

Automated Statistics Page for

Human Rights Library



Author:

Duncan McQueen

Date Last Updated:

Friday, December 13, 2013






Revision History

The following revisions have been made to this document:

Date

Revision

Develop
er

Summary of Changes

12/13/201
3

1.0.0

Duncan McQueen

Created document


Automated Statistics Page for

Human Rights Library




Human Rights Library


-

2

-
helmetpastoral_f5ef9532
-
980c
-
48a0
-
8fc8
-
b357420f4229.doc


Contents

1

Executive Summary

................................
................................
..........................
3

2

Benefits

................................
................................
................................
...............
3

3

Potential Hazards

................................
................................
..............................
3

4

Design Overview

................................
................................
................................
3

5

Implementation

................................
................................
................................
.
3

6

Troubleshooting

................................
................................
................................
4

7

Appendix A: Shell Script

................................
................................
..................
5

8

Appendix B: Perl Script to fix statis
tics.htm

................................
..................
6

9

Appendix C: Report Magic Configuration File (rmagic.ini)

........................
7

10

Appendix D: Analog Configuration File

................................
.........................
8

11

Appendix E: crontab entry

................................
................................
.............
16

12

Appendix F: Installing Perl Modules

................................
............................
16

13

Appendix G: Related Documentation

................................
...........................
17


Automated Statistics Page for

Human Rights Library




Human Rights Library


-

3

-
helmetpastoral_f5ef9532
-
980c
-
48a0
-
8fc8
-
b357420f4229.doc


1

Executive Summary


The statistics page will be automatically generated at the end of the month. This doc
ument
describes the process involved. The created HTML document will then need to be edited to add the
Total User Sessions from FastStats Analyzer.


The following process occurs:

1.

The statistics page is generated for the previous month and named in the for
mat
YYYYMM.html (for example, August 2002 is 200208.html).

2.

A folder is created under the stats folder in the format YYYYMM (August 2002 is 200208).

3.

The html file is copied to the newly created directory

4.

The /info/statistics.htm page is update to link to th
e new log analysis page.

2

Benefits


By making the process automatic, manual labor will be minimal.

3

Potential Hazards


At any point the system administrator may change the server configuration to prevent the
automatic generation of the log files or preven
t the monthly ‘cron’ job from being run. The system
administrator may also disallow the use of Perl scripts.

4

Design Overview


A shell script will be created to call a dns resolver (dnstran) , analog log analyzer (analog) and
Report Magic (rmagic.pl) web s
tats formatter. This shell script will also call a perl script (fixstats.pl)
to insert a link to the HTML file into the statistics.htm page on the web site. A ‘cron’ job will be set
-
up to run the shell script monthly on the first of the month.

5

Implementa
tion


A shell script was created that runs, in sequence, the DNS lookup program (dnstran), Analog log
analyzer, Report Magic formatter, and the statistics update Perl script. The script is viewable in the
Appendix and is located on the server in /hotel/hu
manrts/opt/logs/logrun.sh.


A Perl script was created to insert a link to the new statistics page in the
http://www1.umn.edu/humanrts/info/statistics.htm

page. This script requires the text

‘<!
--

INSERT
--
>’ to be in the statistics.htm page at the point where the new link will be added. The statistics.htm
page was edited to include this tag.


All scripts (analog, dnstran and rmagic.pl) are installed in the hotel/humanrts/opt/logs/ directory
.


A bot.html page was created to store the lower portion of the generated statistics page. Report
Magic uses this file when creating the HTML file.


The configuration files for Analog, DNSTran, and Report Magic were edited. DNSTran was
edited to just
create a cache file that contains all the DNS lookups (called ‘dnscache’). Analog’s
Automated Statistics Page for

Human Rights Library




Human Rights Library


-

4

-
helmetpastoral_f5ef9532
-
980c
-
48a0
-
8fc8
-
b357420f4229.doc

configuration was edited to use this file and only produce the reports needed (General Report, Domain
Report, and the File Size Report). Report Magic’s configuration was
edited to generate an HTML file
that fits within the format of past statistics pages.


An entry was added into the web
-
server’s cron daemon to automatically run the logrun.sh script at
2:00 am on the first of the month.

6

Troubleshooting

Common Problems an
d Potential Solutions:



The script is no longer running at the beginning of the month

o

Verify that cron is still configured to run the script. Type ‘crontab
–
l’ on the shell
prompt and verify that the display matches the one in the Appendix. If it does not
:



run the logrun.sh script manually by typing
‘
/hotel/humanrts/opt/logs/logrun.sh
’ on a shell prompt. This will generate
the log file page for the previous month.




type ‘crontab
–
e’ to re
-
add the cron entry. This will use the ‘vim’ editor
1
.
In order to
edit the file, type ‘i’ to insert a line. Then type the line as shown
in the Appendix. When finished, type the ‘Esc’ key and “Shift
-
Z” twice to
save the file. This will enable cron to run the job for the next month.



The link is no longer being added to
the statistics.htm page

o

Verify that the ‘<!
--

INSERT
--
>’ comment tag is in the statistics.htm document. If
it is not, add it at the location prior to the last table entry in the HTML page.

o

Verify that the fixstats.pl script is still functional.



Potenti
al causes that may break the script are:



Date::Calc perl modules was removed from system. Reinstall
following the link and instructions in the Appendix.



LWP::Simple Perl module was removed from system
.
Reinstall
following the link and instructions in the

Appendix.



A new version of Report Magic was installed and it no longer functions.

o

This is because the UMN server does not have all the required modules to run Report
Magic. The rmagic.pl needs to point to the user Perl modules in ~/perl. Add the
followi
ng line after the line that states ‘use strict’ in rmagic.pl:


use lib "$ENV{HOME}/perl";






1

See
http://www.newriders.com/books/opl/ebooks/0735710015.html

for detailed

documentation on Vim.

Automated Statistics Page for

Human Rights Library




Human Rights Library


-

5

-
helmetpastoral_f5ef9532
-
980c
-
48a0
-
8fc8
-
b357420f4229.doc

7

Appendix A: Shell Script

#!/bin/bash

echo "Running Logs on `date`"


## Change into the logs application directory


cd /hotel/humanrts/opt/logs


## Figure out what

the name of the file should be

## It will be in the YYYYMM format (for example, '200208').

## It is used to figure out what log file to use, as well as

## what to name the resulting HTML



VAR=$(date +%m)

VAR=${VAR#0}

let VAR=VAR
-
1

if [ $VAR
-
gt 9 ]; the
n VAR=$VAR; else VAR=0${VAR} ; fi

VAR=`date +%Y`${VAR}



# Run DNStran against the log file

./dnstran /hotel/humanrts/data/Accesslog
-
${VAR}


# Run Analog

./analog


# Run Report Magic

./rmagic.pl


# Make the directory under the stats folder to hold the HTML

file

mkdir /hotel/humanrts/web
-
docs/stats/${VAR}


# ensure that it is world readable

chmod ugo+rx /hotel/humanrts/web
-
docs/stats/${VAR}


# Copy the index.html file created by Report Magic to the new folder

cp index.html /hotel/humanrts/web
-
docs/stats/${V
AR}/${VAR}.html


# ensure that it is world readable

chmod ugo+r /hotel/humanrts/web
-
docs/stats/${VAR}/${VAR}.html


# Run the Perl script to update the statistics page with the new HTML page

./fixstats.pl


# cleanup (remove extra files created)

rm index.htm
l

Automated Statistics Page for

Human Rights Library




Human Rights Library


-

6

-
helmetpastoral_f5ef9532
-
980c
-
48a0
-
8fc8
-
b357420f4229.doc

8

Appendix B: Perl Script to fix statistics.htm

#!/usr/bin/perl

##############################################################################

# Define use environment


use strict;

use LWP::Simple;

use Date::Calc qw( Date_to_Text_Long English_Ordinal Mont
h_to_Text Today);


# Define Variables

my $year = 1;

my $month = 1;

my $day = 1;

my $statspager = "http://www1.umn.edu/humanrts/info/statistics.htm"; #Remote Stats Page

my $statspagel = "/hotel/humanrts/web
-
docs/info/statistics.htm";


($year,$month,$day)

= Today();


# Get today's date

$month = $month
-

1;

# Find the previous month

if ($month = 0)

{


$month == 12; # Set to December

}

$month = sprintf("%02d", $month);

my $monthtxt = Month_to_Text($month);

# Name of previous Month

my $datecomb = $year .

$month;


# YYYYMM variable for filename

my $datecombh = $datecomb . ".html";

# with .html extension


# Retrieve the Statistics Page

my $doc = get $statspager;


# Insert the text for the new page into the HTML Page

$doc =~ s/<!
--

INSERT
--
>/<TR>
\
n
\
t<TD
WIDTH=
\
"467
\
" height=
\
"20
\
">
\

\
t
\
t <div align=
\
"center
\
">
\

\
t
\
t <font size=
\
"
-
1
\
"> <a href=
\
"
\
/humanrts
\
/stats
\
/$datecomb
\
/$datecombh
\
">
\

\
t
\
t $monthtxt $year<
\
/a><
\
/font><
\
/div>
\
n
\
t<
\
/td>
\
n<
\
/tr>
\
n<!
--

INSERT
--
>/;


#Write the result to the file

open(D
AT,">$statspagel") || print "File does not exist" ;

print DAT $doc;





close(DAT);



Automated Statistics Page for

Human Rights Library




Human Rights Library


-

7

-
helmetpastoral_f5ef9532
-
980c
-
48a0
-
8fc8
-
b357420f4229.doc

9

Appendix C: Report Magic Configuration File
(rmagic.ini)

[statistics]

File_In = report.dat

Active_Column = R

Format = HTML 3.2


[reports]

Font = Verdana, Arial, Hel
vetica, sans
-
serif

Title_BG_Color = blue

Title_Font_Color = white

Title_Font = Tahoma, Arial, Helvetica, sans
-
serif

Title_Font = Tahoma, Arial, Helvetica, sans
-
serif

Data_BG_Color_1 = white

Data_BG_Color_2 = lightgrey

Data_Total_Color = #FFCC66

Data_Header
_Color = blue

Data_Header_Font_Color = white

Data_Font = Tahoma, Arial, Helvetica, sans
-
serif

Graph_Level = 0

GraphType=

#
---

# We give a single filename (not a directory) to tell Report

# Magic to put all reports in one file.

# (MAC: To use this change to

':noframes:report.html')

File_Out = index.html


[GENERAL]

Rows=HN,HU,SR,NF,NH,CL,BT

Description =


[DOMAIN]

Description =


[SIZE]

Description =


[navigation]

File_Out = NONE


[website]

Title = %mmmm yyyy% Statistics

BaseURL = http://www.umn.edu/humanrt
s

Company_Logo = bot.html

Automated Statistics Page for

Human Rights Library




Human Rights Library


-

8

-
helmetpastoral_f5ef9532
-
980c
-
48a0
-
8fc8
-
b357420f4229.doc

10

Appendix D: Analog Configuration File

LOGFILE /hotel/humanrts/data/Accesslog
-
200207


HOSTNAME "Human Rights Library"

HOSTURL http://www1.umn.edu/humanrts/index.html


OUTFILE report.dat


STYLESHEET none


DIRSUFFIX index.html


CACH
EFILE none

CACHEOUTFILE none



TYPEALIAS .html ".html [Hypertext Markup Language]"

TYPEALIAS .htm ".htm [Hypertext Markup Language]"

TYPEALIAS .shtml ".shtml [Server
-
parsed HTML]"

TYPEALIAS .ps ".ps [PostScript]"

TYPEALIAS .gz ".gz
[Gzip compressed files]"

TYPEALIAS .tar.gz ".tar.gz [Compressed archives]"

TYPEALIAS .jpg ".jpg [JPEG graphics]"

TYPEALIAS .jpeg ".jpeg [JPEG graphics]"

TYPEALIAS .gif ".gif [GIF graphics]"

TYPEALIAS .png ".png [PNG graphics]"

TYPEALIAS
.txt ".txt [Plain text]"

TYPEALIAS .cgi ".cgi [CGI scripts]"

TYPEALIAS .pl ".pl [Perl scripts]"

TYPEALIAS .css ".css [Cascading Style Sheets]"

TYPEALIAS .class ".class [Java class files]"

TYPEALIAS .pdf ".pdf [Adobe Portable Do
cument Format]"

TYPEALIAS .zip ".zip [Zip archives]"

TYPEALIAS .hqx ".hqx [Macintosh BinHex files]"

TYPEALIAS .exe ".exe [Executables]"

TYPEALIAS .wav ".wav [WAV sound files]"

TYPEALIAS .avi ".avi [AVI movies]"

TYPEALIAS .arc "
.arc [Compressed archives]"

TYPEALIAS .mid ".mid [MIDI sound files]"

TYPEALIAS .mp3 ".mp3 [MP3 sound files]"

TYPEALIAS .doc ".doc [Microsoft Word document]"

TYPEALIAS .rtf ".rtf [Rich Text Format]"

TYPEALIAS .mov ".mov [Quick Time

movie]"

TYPEALIAS .mpg ".mpg [MPEG movie]"

TYPEALIAS .mpeg ".mpeg [MPEG movie]"

TYPEALIAS .asp ".asp [Active Server Pages]"

TYPEALIAS .jsp ".jsp [Java Server Pages]"

TYPEALIAS .cfm ".cfm [Cold Fusion]"

TYPEALIAS .php ".php [PHP
]"

TYPEALIAS .js ".js [JavaScript code]"



Automated Statistics Page for

Human Rights Library




Human Rights Library


-

9

-
helmetpastoral_f5ef9532
-
980c
-
48a0
-
8fc8
-
b357420f4229.doc

304ISSUCCESS ON


ROBOTINCLUDE REGEXPI:robot

ROBOTINCLUDE REGEXPI:spider

ROBOTINCLUDE REGEXPI:crawler

ROBOTINCLUDE Googlebot*

ROBOTINCLUDE Infoseek*

ROBOTINCLUDE Scooter*

ROBOTINCLUDE Slurp*

ROBOTINCLUDE Ult
raseek*


# Whether each report is wanted: see docs/output.html

GENERAL ON

YEARLY OFF

QUARTERLY OFF

MONTHLY OFF

DAILYREP OFF

DAILYSUM OFF

WEEKLY OFF

HOURLYREP OFF

HOURLYSUM OFF

QUARTERREP OFF

QUARTERSUM OFF

FIVEREP OFF

FIVESUM OFF

DOMAIN ON

ORGANISATION OFF

DIRECTORY OFF

FILETYPE OFF

REQUEST OFF

REDIR OFF

FAILURE OFF

SIZE ON

PROCTIME OFF

HOST OFF

REDIRHOST OFF

FAILHOST OFF

REFERRER OFF

REFSITE OFF

REDIRREF OFF

FAILREF OFF

VHOST OFF

REDIRVHOST OFF

FAILVHOST OFF

USER OFF

REDIRUSER OFF

FAILUSER OFF

SEARCHQUERY
OFF

SEARCHWORD OFF

INTSEARCHQUERY OFF

INTSEARCHWORD OFF

BROWSERREP OFF

BROWSERSUM OFF

OSREP OFF

STATUS OFF

Automated Statistics Page for

Human Rights Library




Human Rights Library


-

10

-
helmetpastoral_f5ef9532
-
980c
-
48a0
-
8fc8
-
b357420f4229.doc


# The columns in each report: see docs/timereps.html and docs/othreps.html

TIMECOLS RP

YEARCOLS RP

QUARTERLYCOLS RP

MONTHCOLS RP

WEEKCOLS RP

DAYREPC
OLS RP

DAYSUMCOLS RP

HOURREPCOLS RP

HOURSUMCOLS RP

QUARTERREPCOLS RP

QUARTERSUMCOLS RP

FIVEREPCOLS RP

FIVESUMCOLS RP

DOMCOLS Rb

ORGCOLS Rb

HOSTCOLS Rb

REDIRHOSTCOLS R

FAILHOSTCOLS R

DIRCOLS Rb

TYPECOLS Rb

REQCOLS RbD

REDIRCOLS R

FAILCOLS R

REFCOLS R

REFSIT
ECOLS R

REDIRREFCOLS R

FAILREFCOLS R

BROWREPCOLS R

BROWSUMCOLS R

OSCOLS NRP

SIZECOLS Rb

PROCTIMECOLS R

VHOSTCOLS Rb

REDIRVHOSTCOLS R

FAILVHOSTCOLS R

USERCOLS Rb

REDIRUSERCOLS R

FAILUSERCOLS R

SEARCHQUERYCOLS R

SEARCHWORDCOLS R

INTSEARCHQUERYCOLS R

INTSEARC
HWORDCOLS R

STATUSCOLS R


# How the graphs are plotted: see docs/timereps.html

ALLGRAPH P

YEARGRAPH P

MONTHGRAPH P

DAYREPGRAPH P

DAYSUMGRAPH P

HOURREPGRAPH P

HOURSUMGRAPH P

WEEKGRAPH P

Automated Statistics Page for

Human Rights Library




Human Rights Library


-

11

-
helmetpastoral_f5ef9532
-
980c
-
48a0
-
8fc8
-
b357420f4229.doc

QUARTERREPGRAPH P

QUARTERSUMGRAPH P

FIVEREPGRAPH P

FIVESUMGRAPH P


# Wh
ether the time reports run forwards or backwards: see docs/timereps.html

ALLBACK OFF

YEARBACK OFF

QUARTERLYBACK OFF

MONTHBACK OFF

DAYREPBACK OFF

HOURREPBACK OFF

WEEKBACK OFF

QUARTERREPBACK OFF

FIVEREPBACK OFF


# The number of rows to display in each time r
eport: see docs/timereps.html

YEARROWS 0

QUARTERLYROWS 0

MONTHROWS 0

WEEKROWS 0

DAYREPROWS 42

HOURREPROWS 72

QUARTERREPROWS 96

FIVEREPROWS 288


# The floors for each report: see docs/othreps.html and docs/hierreps.html

REQFLOOR 20r

REQARGSFLOOR 10r

REDIRFL
OOR
-
30r

REDIRARGSFLOOR 10r

fAILFLOOR
-
30r

FAILARGSFLOOR 10r

DOMFLOOR 0b

SUBDOMFLOOR 0.5%b

ORGFLOOR
-
20r

SUBORGFLOOR 0.5%r

DIRFLOOR 0.01%b

SUBDIRFLOOR 0.01%b

TYPEFLOOR 0.1%b

SUBTYPEFLOOR 0.1%b

HOSTFLOOR
-
50r

REDIRHOSTFLOOR
-
20r

FAILHOSTFLOOR
-
20r

REFFLOOR
20r

REFARGSFLOOR 10r

REFSITEFLOOR
-
30r

REFDIRFLOOR 50r

REDIRREFFLOOR
-
30r

REDIRREFARGSFLOOR 10r

FAILREFFLOOR
-
30r

FAILREFARGSFLOOR 10r

BROWSUMFLOOR
-
20r

SUBBROWFLOOR
-
20r

BROWREPFLOOR
-
40r

Automated Statistics Page for

Human Rights Library




Human Rights Library


-

12

-
helmetpastoral_f5ef9532
-
980c
-
48a0
-
8fc8
-
b357420f4229.doc

OSFLOOR 0p

SUBOSFLOOR 0p

VHOSTFLOOR
-
20b

REDIRVHOSTFLOOR
-
20r

FAILV
HOSTFLOOR
-
20r

USERFLOOR
-
50r

REDIRUSERFLOOR
-
20r

FAILUSERFLOOR
-
20r

SEARCHQUERYFLOOR
-
30r

SEARCHWORDFLOOR
-
30r

INTSEARCHQUERYFLOOR
-
30r

INTSEARCHWORDFLOOR
-
30r

STATUSFLOOR 0r


# How each report is sorted: see docs/othreps.html and docs/hierreps.html

REQSO
RTBY REQUESTS

REDIRSORTBY REQUESTS

FAILSORTBY REQUESTS

DOMSORTBY BYTES

ORGSORTBY REQUESTS

DIRSORTBY BYTES

TYPESORTBY BYTES

HOSTSORTBY ALPHABETICAL

REDIRHOSTSORTBY REQUESTS

FAILHOSTSORTBY REQUESTS

REFSORTBY REQUESTS

REFSITESORTBY REQUESTS

REDIRREFSORTBY REQ
UESTS

FAILREFSORTBY REQUESTS

BROWREPSORTBY REQUESTS

BROWSUMSORTBY REQUESTS

OSSORTBY PAGES

SUBOSSORTBY PAGES

VHOSTSORTBY BYTES

REDIRVHOSTSORTBY REQUESTS

FAILVHOSTSORTBY REQUESTS

USERSORTBY REQUESTS

REDIRUSERSORTBY REQUESTS

FAILUSERSORTBY REQUESTS

SEARCHQUER
YSORTBY REQUESTS

SEARCHWORDSORTBY REQUESTS

INTSEARCHQUERYSORTBY REQUESTS

INTSEARCHWORDSORTBY REQUESTS

STATUSSORTBY ALPHABETICAL


# Whether and how pie charts are plotted: see docs/othreps.html

REQCHART ON

REDIRCHART ON

FAILCHART ON

TYPECHART ON

SIZECHART O
N

PROCTIMECHART ON

DIRCHART ON

HOSTCHART ON

REDIRHOSTCHART ON

Automated Statistics Page for

Human Rights Libr
ary




Human Rights Library


-

13

-
helmetpastoral_f5ef9532
-
980c
-
48a0
-
8fc8
-
b357420f4229.doc

FAILHOSTCHART ON

DOMCHART ON

ORGCHART ON

REFCHART ON

REFSITECHART ON

REDIRREFCHART ON

FAILREFCHART ON

BROWREPCHART ON

BROWSUMCHART ON

OSCHART ON

VHOSTCHART ON

REDIRVHOSTCHART ON

FAILVHOSTCHART O
N

USERCHART ON

REDIRUSERCHART ON

FAILUSERCHART ON

SEARCHQUERYCHART ON

SEARCHWORDCHART ON

INTSEARCHQUERYCHART ON

INTSEARCHWORDCHART ON

STATUSCHART ON

# Zoom in on some wedges using CHARTEXPAND

# OSCHARTEXPAND Windows


# What to link to in the reports

REQLIN
KINCLUDE pages

REDIRLINKEXCLUDE *

FAILLINKEXCLUDE *

REFLINKINCLUDE *

REDIRREFLINKINCLUDE *

FAILREFLINKINCLUDE *

# BASEURL http://my.server.com


# Cosmetic commands: mostly in docs/output.html

OUTPUT COMPUTER

LANGUAGE ENGLISH

HEADERFILE top.html

FOOTERFILE
none

LOGO NONE

REPORTORDER x1QmWDdHwh4657oZSlLujJkKfsNnBbpvRMcPztiEIYyr

GENSUMLINES ALL

GOTOS ON

RUNTIME ON

DESCRIPTIONS OFF

REPORTSPAN OFF

REPORTSPANTHRESHOLD 60

WEEKBEGINSON SUNDAY

LASTSEVEN ON

MARKCHAR +

# SEPCHAR and DECPOINT are usually taken from the

language file

# SEPCHAR ,

# DECPOINT .

REPSEPCHAR none

COMPSEP "

"

RAWBYTES OFF

Automated Statistics Page for

Human Rights Library




Human Rights Library


-

14

-
helmetpastoral_f5ef9532
-
980c
-
48a0
-
8fc8
-
b357420f4229.doc

NOROBOTS ON

SEARCHCHARCONVERT ON

BARSTYLE b

HTMLPAGEWIDTH 65

ASCIIPAGEWIDTH 76

MINGRAPHWIDTH 15


# The following commands are only available on some systems: see docs/dns.html

DNSFILE dnscache

DNS READ



# Sub
-
item commands: see docs/hierreps.html. Examples are

# SUBDOM *.com

# SUBDIR /*/*/

SUBTYPE *.gz,*.Z


# A list of search engines. More SEARCHENGINE commands can be found at

# http://www.analog.cx/helpers/#conffiles

SEARCH
ENGINE http://*altavista.*/* q

SEARCHENGINE http://*yahoo.*/* p

SEARCHENGINE http://*google.*/* q

SEARCHENGINE http://*lycos.*/* query

SEARCHENGINE http://*aol.*/* query

SEARCHENGINE http://*excite.*/* search

SEARCHENGINE http://*go2net.*/* general

SEARCHE
NGINE http://*metacrawler.*/* general

SEARCHENGINE http://*msn.*/* MT

SEARCHENGINE http://*hotbot.com/* MT

SEARCHENGINE http://*netscape.*/* search

SEARCHENGINE http://*looksmart.*/* key

sEARCHENGINE http://*infoseek.*/* qt

SEARCHENGINE http://*webcrawler.
*/* search,searchText

SEARCHENGINE http://*goto.*/* Keywords

SEARCHENGINE http://*snap.*/* keyword

SEARCHENGINE http://*dogpile.*/* q

SEARCHENGINE http://*askjeeves.*/* ask

SEARCHENGINE http://*ask.*/* ask

SEARCHENGINE http://*aj.*/* ask

SEARCHENGINE http:
//*directhit.*/* qry

SEARCHENGINE http://*alltheweb.*/* query

SEARCHENGINE http://*northernlight.*/* qr

SEARCHENGINE http://*nlsearch.*/* qr

SEARCHENGINE http://*dmoz.*/* search

SEARCHENGINE http://*newhoo.*/* search

SEARCHENGINE http://*netfind.*/* query,
search,s

SEARCHENGINE http://*/netfind* query

SEARCHENGINE http://*/pursuit query


# Search engines on your own site

# INTSEARCHENGINE /cgi
-
bin/my.cgi q


# Use one of these if you are short of memory: see docs/lowmem.html

FILELOWMEM 0

HOSTLOWMEM 0

Automated Statistics Page for

Human Rights Library




Human Rights Library


-

15

-
helmetpastoral_f5ef9532
-
980c
-
48a0
-
8fc8
-
b357420f4229.doc

BROWLOWM
EM 0

REFLOWMEM 0

USERLOWMEM 0

VHOSTLOWMEM 0


# Commands for debugging

SETTINGS OFF

DEBUG OFF

WARNINGS ON

PROGRESSFREQ 0

# ERRFILE somefile

ERRLINELENGTH 78

#

# By Stephen Turner 1999
-

2002. No warranty for this file.

# This file is given into the public d
omain: it may be used, modified and

# redistributed without restriction.

Automated Statistics Page for

Human Rights Library




Human Rights Library


-

16

-
helmetpastoral_f5ef9532
-
980c
-
48a0
-
8fc8
-
b357420f4229.doc

11

Appendix E: crontab entry

0 2 1 * * /hotel/humanrts/opt/logs/logrun.sh

12

Appendix F: Installing Perl Modules


This short tutorial will utilize the Perl CPAN module to install additiona
l modules. This module is
included, by default, with a Perl installation.



The first setup is ensuring that the CPAN module is configured correctly. There should be a
folder structure (under the home directory) as such. Make sure when listing the direc
tory contents to
use the
–
a switch, since many of the files are hidden



.cpan

o

CPAN



MyConfig.pm

o

build

o

sources


The MyConfig.pm file should look like this:

$CPAN::Config = {


'build_cache' => q[10],


'build_dir' => q[/home/h/humanrts/.cpan/build],


'cache_
metadata' => q[1],


'cpan_home' => q[/home/h/humanrts/.cpan],


'dontload_hash' => { },


'ftp' => q[/usr/bin/ftp],


'ftp_proxy' => q[],


'getcwd' => q[cwd],


'gzip' => q[/usr/local/bin/gzip],


'http_proxy' => q[],


'inactivity_timeout' => q[0],


'
index_expire' => q[1],


'inhibit_startup_message' => q[0],


'keep_source_where' => q[/home/h/humanrts/.cpan/sources],


'lynx' => q[/usr/local/bin/lynx],


'make' => q[/usr/ccs/bin/make],


'make_arg' => q[],


'make_install_arg' => q[UNINST=1],


'makep
l_arg' => q[LIB=~/perl INSTALLMAN1DIR=~/perl/man/man1 INSTALLMAN3DIR=~/p

erl/man/man3],


'ncftp' => q[/usr/local/bin/ncftp],


'ncftpget' => q[],


'no_proxy' => q[],


'pager' => q[more],


'prerequisites_policy' => q[ask],


'scan_cache' => q[atstart],


'shell' => q[tcsh],


'tar' => q[/usr/bin/tar],


'term_is_latin' => q[1],


'unzip' => q[/usr/bin/unzip],


'urllist' => [q[ftp://archive.progeny.com/CPAN/]],


'wait_list' => [q[wait://ls6.informatik.uni
-
dortmund.de:1404]],


'wget' => q[],

};

1;

__END
__


Automated Statistics Page for

Human Rights Library




Human Rights Library


-

17

-
helmetpastoral_f5ef9532
-
980c
-
48a0
-
8fc8
-
b357420f4229.doc


The important line in the configuration file is :

'makepl_arg' => q[LIB=~/perl INSTALLMAN1DIR=~/perl/man/man1 INSTALLMAN3DIR=~/p
erl/man/man3],


This line tells the CPAN module to install all new modules in the /hotel/humanrts/perl directory
(=~/perl)
. The perl script will look to this directory to find any modules that are not installed system
-
wide.


Once everything is configured, the CPAN module will be used to build and install new modules.
Enter into the CPAN module’s interactive mode by typing “
perl
-
MCPAN
-
e shell”. Help can be
found by typing ‘h’ on the command line. In order install the modules Date::Calc or LWP::Simple (as
used in the fixstats.pl script), type ‘install Date::Calc’ or ‘install LWP::Simple’. Watch for any errors
and attempt
to resolve as the information states. If there are no errors, the modules will be installed
and usable in a Perl script.

13

Appendix G: Related Documentation

Document

Location/Filename

Perl

http://www.perl.com/

Info on

setting up a ‘cron’ job

http://www.faqts.com/knowledge_base/view.phtml/aid/100
5/fid/436

Analog Log Analyzer

http://www.analog.cx

Repor
t Magic Log File Formatter

http://www.reportmagic.org


DNSTran

http://www.summary.net/soft/dnstran.html


Perl CPAN Module Readme on
installing Perl Modu
les

http://www.perl.com/CPAN/modules/by
-
module/CPAN/CPAN
-
1.62.readme


Date::Calc Perl Module

http://www.perl
.com/CPAN/modules/by
-
module/Date/


LWP::Simple (libwww) Perl Module

http://www.perl.com/CPAN/modules/by
-
module/LWP/


Documentation

\
\
LAW
-
ADMIN
\
VOL2
\
Depts
\
Human Rights
Center
\
hrcenter
\
Documentation
\