advanced_bio-linux_logfiles

caddiepastΔιαχείριση Δεδομένων

31 Ιαν 2013 (πριν από 4 χρόνια και 9 μήνες)

276 εμφανίσεις

Environmental Genomics Thematic Programme

Data Centre

http://envgen.nox.ac.uk

Advanced Bio
-
Linux

Dan Swan: Log files and log monitoring

Environmental Genomics Thematic Programme

Data Centre

http://envgen.nox.ac.uk

What, where and why?


Log files are files of 'useful' information that are written in an
ongoing fashion by various programs on the system.


Most (but not all) logs are stored in
/var/log/


Apache (webserver logs) are stored in
/etc/httpd/logs/


Logs are your first port of call for detecting things like upcoming
hardware issues (such as disk drive failure), accounting visits
(hits) on the webserver, detection of intrusion attempts,
detection of intrusions.


Going over the log files is good practice, but for an
inexperienced admin making sense of them is not always
straightforward.


We will try to demistify some of this process and show you how
to automate it.

Environmental Genomics Thematic Programme

Data Centre

http://envgen.nox.ac.uk

Apache logs


Environmental Genomics Thematic Programme

Data Centre

http://envgen.nox.ac.uk

Why check these logs?


If you have opened up your web server to the outside world it
will be probed by hackers. Apache has had a number of
vulnerabilities in the past but is generally considered to be good
in terms of security.


Things to look out for are
-

huge entries trying to get filenames
with thousands of characters
-

these are attempted “buffer
overflow” exploits. Anything which contains more strings like
../../../../../../../../../ is attempting to locate files outside of an
improperly secured Apache set up. Any requests for
executable files, or cgi files that you did not put there are cause
for concern.


You can safely ignore anything looking for .com .exe files,
these are scans for Microsoft IIS vulnerabilities.

Environmental Genomics Thematic Programme

Data Centre

http://envgen.nox.ac.uk

/var/log


This containst the majority of the log files.


Each one stores different information.


It is important to keep an eye on these.


Some can be viewed with a 'logchecker' to give the
administrator essential information.


It is important that as an administrator you learn to understand
the information given in the logs.


The only way is to look and learn. You will often not
understand the entries, use Google to look them up if you are
concerned.


We will go through them one by one briefly.

Environmental Genomics Thematic Programme

Data Centre

http://envgen.nox.ac.uk


Environmental Genomics Thematic Programme

Data Centre

http://envgen.nox.ac.uk


Environmental Genomics Thematic Programme

Data Centre

http://envgen.nox.ac.uk

dmesg


less /var/log/dmesg


keeps a log of printed kernel messages.


Most often this will have information about the boot process,
and it should be checked to keep an eye on the actual
hardware as it can harbour warnings of upcoming problems.


Whilst the system is running
dmesg

will print out any other
kernel messages, and will include items like firewall incidents
etc.


There is a great article demystifying the dmesg output (which
can be quite cryptic) here:


http://www.linuxgazette.com/issue59/nazario.ht
ml

Environmental Genomics Thematic Programme

Data Centre

http://envgen.nox.ac.uk


Log files not to stress about


/var/log/gdm : gdm is the graphical login manager.


/var/log/httpd : the same as /etc/httpd/logs/


/var/log/iptraf : it's empty!


/var/log/kdm.log : another graphical login manager.


/var/log/ksyms.* : kernel symbols logs (incomprehensible).


/var/log/lastlog : Use the last command instead.


/var/log/mysqld.log : Of course if you start using MySQL you might look!


/var/log/pgsql : PostgreSQL logs
-

as above.


/var/log/rcd/* : RedCarpet logs
-

only look if you have issues.


/var/log/rpmpkgs : Same as running rpm
-
qa


/var/log/sa : output from sar command on cron job


/var/log/scrollkeeper.log : GNOME messages, might be good to look at if you have a
GNOME problem!


/var/log/spooler : UUCP spools (no
-
one uses this anymore!)


/var/log/up2date : RedHat update mechanism (unused).


/var/log/wtmp : not human readable


/var/log/Xfree86.0.log : If you have X errors this is the place to look.

Environmental Genomics Thematic Programme

Data Centre

http://envgen.nox.ac.uk


Environmental Genomics Thematic Programme

Data Centre

http://envgen.nox.ac.uk


Environmental Genomics Thematic Programme

Data Centre

http://envgen.nox.ac.uk


Environmental Genomics Thematic Programme

Data Centre

http://envgen.nox.ac.uk

What is with the .1 files?


The first thing you notice about /var/log/ files is that you have


file


file.1


file.2 etc.


This is because logs are rotated weekly to make sure they do
not clog up the disk.


Rotation is handled by logrotate.sh in /etc/cron/cron.daily


Each week file is moved to file.1, file.1 to file.2 etc. and file.4 is
deleted as we work a 4 week cycle of logs.


You can alter the behaviour of logrotate in /etc/logrotate.conf

Environmental Genomics Thematic Programme

Data Centre

http://envgen.nox.ac.uk

Automated log checking


Currently Bio
-
Linux machines are set to email the EGTDC your
logs. Now you are all advanced users :) you can take the
responsibility for your own log checking. We wil advise you
how to do this later.


Log checking is done by a program called logcheck.sh
-

appropriately! This is run as a daily cron job and the results
emailed to the defined manager. Currently this is channelled to
the EGTDC account and forwarded to us. In future this should
be sent to the manager account.


This tries to pick out the most obvious security problems and
report them, although it still reports a lot of false positives it is
imperative you watch this file for curious actions on the system.