pptx - Information Technology Services

puffyyaphankyonkersΔίκτυα και Επικοινωνίες

26 Οκτ 2013 (πριν από 3 χρόνια και 9 μήνες)

91 εμφανίσεις

Centralized Logging:

Why Would I Want to Do That?

Garrett Lanzy

Access more information security training for
campus technical staff and earn CEUs:


its.mnscu.edu/security/training/

Outline


Problems


Centralized Logging at Metropolitan State


Architecture and Configuration


Other Considerations


Real
-
Life Scenario


How Does Logging Help?


Future Directions


Conclusions


Q&A


Access more information security training for
campus technical staff and earn CEUs:


its.mnscu.edu/security/training/

Problems: Logging Issues


Logs spread across servers and network
devices


Different devices use different log formats


Default settings often omit important
information


Log data retention varies


An “event” can involve numerous different
devices


hard to search/correlate

Key Questions


What should be logged?


How can this data be gathered?


How long should this data be retained?


How can searching be simplified?


An Answer:

Centralized Logging

Access more information security training for
campus technical staff and earn CEUs:


its.mnscu.edu/security/training/

Centralized Logging at
Metropolitan State University


Single central logging server (axe)


Linux (SUSE Linux Enterprise Server)


syslog
-
ng

logging daemon


30GB partition for logs


Stores 65 days worth of logs for 60+ devices (including
firewall logs for 2 campuses)


Physical server


Same server used for network monitoring (
Nagios
)

What is
syslog
?


Logging architecture for Unix/Linux


Developed since 1980s


Provides services for local & centralized
logging (single or multi
-
tier)


Communication protocol standardized by IETF



RFC 3164
-

“classic” BSD Unix
syslog

(UDP only)



RFC 5424


TLS transport support


Neither defines format of log messages


What is
syslog
-
ng
?


“Next generation”
syslog


Open
-
source & commercial versions


Supports UDP, TCP, and TLS transports


Flexible configuration options


Many options for storing log data


Simple files


Structured files (CSV, etc)


Database interfaces



syslog

Message Facilities


syslog

defines 24 types of sources for logs (called
facilities
)


common ones:


kernel


messages from OS kernel


user


user
-
level messages


mail


e
-
mail related messages


system


messages from system daemons/services


auth


security/authorization messages


syslog



messages generated by
syslog

itself


local0


local7


implementation
-
defined message


Metro design creates separate files per facility for
each device


syslog

Message Severities


Each message assigned a severity by creator


Can be used to help filter by importance



Severity

Description

0

Emergency: system is unusable

1

Alert: action must be taken immediately

2

Critical: critical conditions

3

Error: error conditions

4

Warning: warning conditions

5

Notice: normal but significant condition

6

Informational: informational messages

7

Debug: debug
-
level messages

Metro Centralized Logging Design


Based on Novell Cool Solutions article
Centralized
Syslogging

with
Syslog
-
NG on
SUSE Linux

by Scott Flowers

http://www.novell.com/coolsolutions/feature/18044.html


Supports classic
syslog

on UDP port 514 &
syslog

over TCP using TCP port 5140


Simpler directory structure for easier searches


Added automated compression via
logrotate

Logging server
syslog
-
ng

config


Default
SuSE

configuraton

contains logging source
src

for messages originating on local server


Metro design adds logging source
nethosts

for logs
coming form other servers


Adds logging destination
byhost

which writes logs for
each host to a specific directory

/
var
/log/HOSTS/
hostname


Log files named
facilityname
.log


Local logs for central logging server not changed

syslog
-
ng.conf

for
nethosts

source

# define source for messages from network hosts

Source
nethosts

{


#


# process log messages from network:


#


udp
(
ip
(“0.0.0.0") port(514));


tcp
(
ip
(“0.0.0.0") port(5140) keep
-
alive(yes)
max_connections
(100));

};


syslog
-
ng.conf

for
byhost

destination

# separate out hosts into individual log directories

destination
byhost

{


file("/
var
/log/HOSTS/$HOST/$FACILITY.log"


owner(root) group(root) perm(0640)
dir_perm
(0750)
create_dirs
(yes)


);

};

log {


source(
src
);


source(
nethosts
);


destination(
byhost
);

};


Configuring other devices/servers


Use TCP over UDP when possible


Exception: Cisco ASA devices stop forwarding if
syslog

over TCP configured & not working


Check server/device logging settings to make
sure appropriate information is logged


Example: some servers only log failed login
attempts, best to also log successful ones


Make sure to adjust firewall configuration on
centralized logging server

Configuring Linux Server
-

syslog


syslog

always sends via UDP (default port 514)


Sample
config

to send messages for all
facilities of severity
informational

or higher to
central log server


add lines to
syslog.conf
:


# send all logs of severity info or higher to logging server

*.info @
x.x.x.x


Configuring Linux Server
-

syslog
-
ng


Define destination
loghost

for TCP port 5140
on logging server, send all messages there


Add following lines to
syslog
-
ng

configuration:


destination
loghost

{


tcp
("
x.x.x.x
" port(5140));

};

log {


source(
src
);


destination(
loghost
);

};


Configuring Network Devices


Most managed network devices support
logging to central
syslog

server


Many configurable via web/GUI interfaces


Sample Cisco IOS configuration:


Core
-
6500#conf term

Enter configuration commands, one per line. End with CNTL/Z.

Core
-
6500(
config
)#logging source
-
interface Vlan1

Core
-
6500(
config
)#logging
x.x.x.x

Core
-
6500(
config
)#end




Configuring ASA Firewall Logging


MnSCU
-
owned ASA firewalls can be
configured to log to campus logging servers
-

contact
MnSCU

network support team:

hostmaster@mnscu.edu


For campus
-
owned devices, easy to set up
with Cisco ASDM:


Configuration
-
> Device Management
-
>
Syslog

Servers


Configuration
-
> Device Management
-
>
Syslog

Setup

Configuring ASA Firewall Logging


MnSCU
-
owned ASA firewalls can be
configured to log to campus logging servers
-

contact
MnSCU

network support team:

hostmaster@mnscu.edu


For campus
-
owned devices, easy to set up
with Cisco ASDM:


Configuration
-
> Device Management
-
>
Syslog

Servers


Configuration
-
> Device Management
-
>
Syslog

Setup

Configuring Cisco ACS Logging


Can configure logging for each “report name”


To start: from web admin interface, select
System
Configuration

-
>
Logging


Select
Configure

link for
syslog

column of desired
“report”


Click
Enable Logging

checkbox


Scroll down, enter IP address of server, set port to
514, and max bytes to 1024


Submit

Sample ACS
Config

(pt. 1)

Sample ACS
Config

(pt. 2)

Sample ACS
Config

(pt. 3)

Windows Servers


Windows event logging traditionally has had
no support for centralized logging


Several add
-
on products available to send
Windows events to
syslog

server


Snare for Windows


open
-
source product


Microsoft has added capability for centralized event
collection in Windows 2008


for info, see:

http://www.windowsecurity.com/articles/Centralized
-
Auditing
-
here
-
FREE.html

Snare Agent for Windows


Snare Agent for Windows is open
-
source, free
software (GNU Public License)


Available at:

http://www.intersectalliance.com/projects/SnareWindows/


2 versions:


Snare for Windows: supports NT, 2000, XP, 2003


Snare for Windows Vista: supports Vista, 2008,
Windows 7


Either will run on 32 or 64
-
bit installations


Configuring Snare Agent

Windows Event Log Settings


Security audit settings


clean Win2008 install

A better idea?

Other Considerations


Clock synchronization


Log rotation/space utilization


Backups


Network Monitoring

Clock Synchronization


System/device clocks need to be synchronized
in order to correlate log data


Any state/
MnSCU

router can be used as an
NTP time source


Most network devices can sync to NTP


Windows standalone servers or domain
controllers should be synchronized to NTP

http://support.microsoft.com/kb/816042

Linux NTP Configuration


Linux VMs: try kernel option clock=pit


Remove any references to local
unsynchronized clock (127.0.0.1)


Simple sample
ntp.conf


driftfile

/
var
/lib/
ntp
/drift/
ntp.drift

# path for drift file


logfile

/
var
/log/
ntp

server 172.16.255.238

server 156.98.1.1

Log Rotation


logrotate



Linux utility to rotate and
compress log files


Creates new log files when run and deletes
after specified interval


Metro configuration:


Rotate files daily


Delay compression for 1 day


Keep 65 days worth of logs

Log server
logrotate.conf

additions

# rotate logs in /
var
/logs/HOSTS tree

/
var
/log/HOSTS/*/*.log {


daily


compress


delaycompress


dateext


missingok


notifempty


rotate 65


maxage

90


sharedscripts


create 640 root
root


postrotate


/etc/
init.d
/
syslog

reload


endscript

}


Real
-
Life Scenario


Have your received one of these?

> Subject:

Notice ID: 22
-
79162500 Notice of Unauthorized Use of Paramount Pictures Corporation Property

> Date:

Fri, 23 Apr 2010 10:07:58
-
0700

> From:

paramount
-
no
-
reply@copyright
-
compliance.com

> Reply
-
To:

paramount
-
no
-
reply@copyright
-
compliance.com

> To:

abuse@mnscu.edu

>
-----
BEGIN PGP SIGNED MESSAGE
-----

> Hash: SHA1

> Notice ID: 22
-
79162500

> Notice Date: 23 Apr 2010 16:58:09 GMT

> Minnesota State University System

> Dear Sir or Madam:

>

>
BayTSP
, Inc. ("
BayTSP
") swears under penalty of perjury that Paramount

> Pictures Corporation ("Paramount") has authorized
BayTSP

to act as

> its non
-
exclusive agent for copyright infringement notification.

>
BayTSP's

search of the protocol listed below has detected

> infringements of Paramount's copyright interests on your IP

> addresses as detailed in the below report
.

… And to the Relevant Information:

> Evidentiary Information:

> Notice ID: 22
-
79162500

> Initial Infringement Timestamp:
23 Apr 2010 16:23:27 GMT

> Recent Infringement Timestamp: 23 Apr 2010 16:23:27 GMT

> Infringers IP Address:
199.17.228.252

> Protocol:
BitTorrent

> Infringed Work: She's Out of My League

> Infringing File Name: She's Out of My League 2010 CAM
-
lu

> Infringing File Size:
608227413

> Bay ID: d116ad77ba844d11a350055c9227e2b0d12a29d9|608227413

> Port ID:
45739

> Infringer's DNS Name:

> Infringer's User Name:

How to proceed?


199.17.228.252 is a NAT address for the
student wireless network on the Midway
campus


Search the Midway ASA firewall log to
determine the local IP address


Then search the ACS appliance log to find the
user logged in

…Um, but wait….


Logs older than 1 day old are compressed, so
need to decompress before they can be
searched


axe:~ #
cd

/
var
/log/HOSTS/
mwpix
/

axe:/
var
/log/HOSTS/
mwpix

#
bzcat

local4.log
-
20100423.bz2 >/
tmp
/mwpix.log

axe:/
var
/log/HOSTS/
mwpix

#
cd

/
var
/log/HOSTS/172.16.210.49/

axe:/
var
/log/HOSTS/172.16.210.49 #
bzcat

auth.log
-
20100423.bz2 >/
tmp
/acs.log

axe:/
var
/log/HOSTS/172.16.210.49 #



What does the ASA log look like?

Apr 23 09:00:00
mwpix

%ASA
-
6
-
305011: Built dynamic TCP translation from
inside:10.15.38.69/49523 to outside:
199.17.228.252/20192

Apr 23 09:00:00
mwpix

%ASA
-
5
-
304001: 10.15.38.69 Accessed URL
65.55.17.27:http://
www.msn.com
/

Apr 23 09:00:00
mwpix

%ASA
-
6
-
302020: Built inbound ICMP connection for
faddr

207.171.116.193/0
gaddr

199.17.231.41/0
laddr

199.17.231.41/0

Apr 23 09:00:00
mwpix

%ASA
-
6
-
302021: Teardown ICMP connection for
faddr

207.171.116.193/0
gaddr

199.17.231.41/0
laddr

199.17.231.41/0

Apr 23 09:00:00
mwpix

%ASA
-
6
-
305011: Built dynamic UDP translation from
inside:10.15.38.67/62560 to outside:199.17.228.252/49397

Apr 23 09:00:00
mwpix

%ASA
-
6
-
305011: Built dynamic UDP translation from
inside:10.15.38.67/60379 to outside:199.17.228.252/41373

Apr 23 09:00:00
mwpix

%ASA
-
4
-
313005: No matching connection for ICMP error
message:
icmp

src

inside:10.15.38.67
dst

outside:199.17.241.241 (type 3, code 3) o

n inside interface. Original IP payload:
udp

src

199.17.241.241/53
dst

10.15.38.67/60379.

Apr 23 09:00:00
mwpix

%ASA
-
6
-
305011: Built dynamic UDP translation from
inside:10.15.35.98/50165 to outside:199.17.228.252/38489

Apr 23 09:00:00
mwpix

%ASA
-
7
-
609001: Built local
-
host outside:74.63.145.159


We’re looking for….


… a line that contains:

199.17.228.252/45739


… at about 16:23:27 GMT (11:23:27 CDT)

grep

is your friend!


grep



utility to search by regular expression


Some basic regular expression syntax:


Most characters “match” themselves


Exceptions: [
\
^$.|?*+()


\

is an escape character; meaning varies by context, always
suppresses special meaning of [
\
^$.|?*+()


.

matches any single character (except the line break
character)


* repeats the previous item 0 or more times


…. So .* matches “anything”

http://www.regular
-
expressions.info/reference.html

grep

in action

axe:/
tmp

#
grep

199.17.228.252/45739 mwpix.log

Apr 23 08:49:17
mwpix

%ASA
-
6
-
305011: Built dynamic TCP translation from
inside:10.15.34.45/1701 to outside:199.17.228.252/45739

Apr 23 08:49:47
mwpix

%ASA
-
6
-
305012: Teardown dynamic TCP translation
from inside:10.15.34.45/1701 to outside:199.17.228.252/45739 duration
0:00:30

Apr 23 11:26:23
mwpix

%ASA
-
6
-
305012: Teardown dynamic TCP
translation from inside:10.15.38.87/50372 to
outside:199.17.228.252/45739 duration 0:03:00


Looks like we have a match
-

a connection of
duration 3 min. which ended at 11:26:23


i.e., was initiated at 11:23:23 (vs. 11:23:27)

Now check the ACS log

(note: ACS timestamps are GMT, not local time)

axe:/
tmp

#
grep

10.15.38.87 acs.log

Apr 23
16:19:06

172.16.210.49 CisACS_01_
PassedAuth

y2gjm36g 1 0
Message
-
Type=
Authen

OK,User
-
Name=
vvvvvv
,NAS
-
IP
-
Address=172.16.212.7,Caller
-
ID=
10.15.38.87
,NAS
-
Port=
vvvvvv,Group
-
Name=Default
Group,Filter

Information=No Filters activated.,

Apr 23
16:19:06

172.16.210.49 CisACS_03_RADIUSAcc y2gjm36h 1 0 User
-
Name=
vvvvvv
,NAS
-
IP
-
Address=172.16.212.7,NAS
-
Port=29,Group
-
Name=Default
Group,Framed
-
IP
-
Address=10.15.38.87,Calling
-
Station
-
Id=10.15.38.87,Acct
-
Status
-
Type=
Start,Acct
-
Session
-
Id=4bd1c873
/00:24:2b:ae:79:c1/38738,

Apr 23
18:05:44

172.16.210.49 CisACS_03_RADIUSAcc xjgh1n40 1 0 User
-
Name=
vvvvvv
,NAS
-
IP
-
Address=172.16.212.7,NAS
-
Port=29,Group
-
Name=Default
Group,Framed
-
IP
-
Address=
10.15.38.87
,Calling
-
Station
-
Id=10.15.38.87,Acct
-
Status
-
Type=
Stop,Acct
-
Input
-
Octets=495945068,Acct
-
Output
-
Octets=1553689408
,Acct
-
Session
-
Id=4bd1c873/00:24:2b:ae:79:c1/38738,Acct
-
Session
-
Time=6398,Acct
-
Input
-
Packets=1155133,Acct
-
Output
-
Packets=1692907,

axe:/
tmp

#


Determination


User
vvvvvv

made the connection outlined in
the e
-
mail and downloaded 1.5GB


this case
will be handled as inappropriate use


Actual time to perform the log search process
(including reading the e
-
mail, logging in to the
logging server, decompressing the logs, and
searching): less than 5 minutes

Future Directions


Centralized logging for Windows workstations


Use TLS connections instead of TCP


Makes snooping much more difficult


Prevents insertion of “bogus” logs


Automated log monitoring/proactive
notification


Log to database instead of stream files?


Disk space is cheap, right?

Conclusions


Centralized logging is not that difficult to set
up


Significant benefits for both normal
operations and incident handling


Can save large amounts of time for
network/server/security administrators

Questions?

Garrett Lanzy

Through 5/11/10:

Garrett.Lanzy@metrostate.edu

5/12/10 & After:

Garrett.Lanzy@csu.mnscu.edu





Access more information security training for
campus technical staff and earn CEUs:


its.mnscu.edu/security/training/