Active Directory Performance Measurement And Troubleshooting

guideflannelΔιακομιστές

4 Δεκ 2013 (πριν από 3 χρόνια και 8 μήνες)

186 εμφανίσεις

©2004 Microsoft Corporation. All rights reserved.

WIN303


Active Directory
Performance Measurement
And Troubleshooting


Joe Wong

Technical Account Manager

Microsoft Services, Microsoft Hong
Kong

©2004 Microsoft Corporation. All rights reserved.

New Product

Server Performance Advisor 1.0

Released 5/24/2004:

http://download.microsoft.com/download/6/2/c/6
2c587bf
-
0d42
-
4ca2
-
9b04
-
5e6771dd209a/spa.msi

Used to measure and report system
performance statistics

Reports are especially detailed for Active
Directory, IIS, and Print.

©2004 Microsoft Corporation. All rights reserved.

Agenda

Overview of performance analysis for
Active Directory

Using the tools to troubleshoot
common performance issues

Common Active Directory Performance
issues

Putting it all together: How to approach
performance issues in Active Directory


©2004 Microsoft Corporation. All rights reserved.

Why we measure performance

Troubleshoot operational problems

Capacity planning

New deployment

Server Consolidation

Application development

How optimized are my LDAP queries?


©2004 Microsoft Corporation. All rights reserved.

What is being measured

Performance Counters

Event logs for searches

LDAP Extended controls

Event Tracing for Windows (ETW)

Hooks are at function and component level

“hooked” components:

NTDSA (DSREPL, LDAP, NTLM, Kerberos, etc)

DNS Client

DNS Server (Sp1)

Various Kernel (Disk, Network, TCP/IP, ACPI)

Many more in the future!

©2004 Microsoft Corporation. All rights reserved.

More about Event tracing

MSDN has information on how to hook
components

©2004 Microsoft Corporation. All rights reserved.

How to measure

Tool

Data Source

Perfmon

Counters (PDH data)

Netmon

Network packets

Server Performance Advisor

Counters (PDH) and

Events (ETW)

Event logs

Event information, logged

per component

Kernrate

Module level tracing using

symbols

©2004 Microsoft Corporation. All rights reserved.

Using LDP and Eventlog to
troubleshoot long searches


Send query to DC with Search Stats extended control
(1.2.840.113556.1.4.970)



Set Field Engineering reg key to 0x5 in the event logs



©2004 Microsoft Corporation. All rights reserved.

Tracking Expensive &
Inefficient Searches

Field Engineering = 0x4

Log a summary event of searches every
time garbage collector runs

Field Engineering = 0x5

Log an event every time an Expensive or
Inefficient search occurs


Using the default values, a search is expensive if it
visits more than x (10000) entries, and it is
inefficient if the returned entries are less that 10% of
the entries that it visited, and in addition the visited
entries are more than x (1000).



©2004 Microsoft Corporation. All rights reserved.

Tracking Expensive &
Inefficient Searches

Internal Processing = 0x4

Show index used to optimize a LDAP
Query

Categorizing expensive or inefficient
LDAP search
(HKLM
\
SYSTEM
\
CCS
\
Services
\

NTDS
\
Parameters)

Expensive Search Results Threshold

Inefficient Search Results Threshold

©2004 Microsoft Corporation. All rights reserved.

Agenda

Overview of performance analysis for
Active Directory

Using the tools to troubleshoot
common performance issues

Common Active Directory Performance
issues

Putting it all together: How to approach
performance issues in Active Directory


©2004 Microsoft Corporation. All rights reserved.

High CPU usage (100% LSASS)


Summary

CPU usage on a Domain Controller is
constantly at 100%, or hitting sustained
spikes of 100% usage

Typical Causes

Non
-
indexed searches!

Rapid retry authentication (how does an
application handle password expired
accounts?)


©2004 Microsoft Corporation. All rights reserved.

Troubleshooting 100% CPU

©2004 Microsoft Corporation. All rights reserved.

Long
\
Latent Searches


Summary

Applications start failing because DS
searches are timing out. DS Searches
take a long time to return (if at all)

Typical Causes

Server is out of resources (CPU, Disk,
Memory)

Bad searches (expensive and timeout)

Tools to use

Server Performance Advisor

Eventlog (Field Engineering), Netmon


©2004 Microsoft Corporation. All rights reserved.

Troubleshooting long
searches

©2004 Microsoft Corporation. All rights reserved.

Long
\
Latent Searches (con’t)


Using Perfmon “ATQ threads” to gauge server load:



DS Search Routines
,
DB Layer
,
JET
Single Proc Domain Controller
LDAP Queries
ATQ Pool Threads
(
4
x proc
)
ATQ
ATQ
ATQ
ATQ
There are a limited number of worker threads for
answering LDAP requests. Long searches can tie
up all of these and deny other searches. If 'ATQ
Threads LDAP' is constantly pegged to the value of
'ATQ Threads Total‘ then LDAP searches can be
delayed. Check in SPA for long running searches

©2004 Microsoft Corporation. All rights reserved.

Agenda

Overview of performance analysis for
Active Directory

Using the tools to troubleshoot
common performance issues

Common Active Directory Performance
issues

Putting it all together: How to approach
performance issues in Active Directory


©2004 Microsoft Corporation. All rights reserved.

Common Active Directory
Performance issues

Directory Search
-

DNT Index


Cause

DNT Index is used to walk most of the data tree.
This search may suggest a non
-
indexed attribute
based on the input search filter.


Resolution

Consider the rate per second, CPU used, and
response times. If these numbers are high, learn
more about the search. For custom attributes,
add an index that more closely fits the filter being
used.

©2004 Microsoft Corporation. All rights reserved.

Common Active Directory
Performance issues

Directory Search
-

Referrals

Cause

The LDAP_OPT_REFERRALS option is on by default
when you search the root of the domain. In small
domains with default naming contexts, a subtree search
at the domain root results in three searches. The client
program usually, unless designed for referrals, shows a
high client
-
side response time as it waits for the LDAP
client to assemble the referred objects.

Resolution

Determine which program is searching at the domain
root. Look up and disable the LDAP_OPT_REFERRALS
option from your custom program or search the subtree
(deep) from a more specific container.

©2004 Microsoft Corporation. All rights reserved.

Common Active Directory
Performance issues

Directory Search


High rate of
Authentication failures (Kerberos)

Cause

The Ticket Granting Server (TGS) ticket request is performed
when a resource is required in the domain. It is fairly easy to
try to use domain resources with an expired ticket or to try to
use some unauthorized resource without the proper
credentials. Such activity suggests a poorly functioning
application or even a bad application trying to deny service to
other uses.

Resolution

Determine the major client or clients generating the frequent
errors. Decide if there is a business need to continue to let
them fail.

©2004 Microsoft Corporation. All rights reserved.

System Health Index

*
Disk I/O Status (rough guide):

Disk I/O Rate <= 100

Disk Rate Idle

100 < Disk I/O Rate < 500

Disk Rate Normal

Disk I/O Rate >= 500

Disk Rate Busy


Memory Status (rough guide):

<% Mem Utilization < 15


Low Memory Usage


15 < % Mem Utilization < 75


Normal Memory Usage


75 < % Mem Utilization < 100


High Memory Usage


*
Disk rates can be very different depending on HW. Fiber
attached storage could do more I/O’s then 500. If this situation
applies, switch to looking at “Current disk queue length” and
ensure the number is consistently < 10

©2004 Microsoft Corporation. All rights reserved.

System Health Index

CPU Status (rough guide):

% CPU Utilization <= 20


System Idle

20 < % CPU Utilization < 80


System Normal

% CPU Utilization >= 80


System Busy


Network Status (rough guide):

% Network Utilization <= 15

Low Traffic


15 < % Network Utilization < 60

Normal Traffic


% Network Utilization >= 60

High Traffic

©2004 Microsoft Corporation. All rights reserved.

Memory Usage by LSASS

Responsible for

Local Security Authority, Net Logon
service, Security Accounts Manager
service, LSA Server service, Secure
Sockets Layer (SSL), Kerberos v5
authentication protocol, NTLM
authentication protocol

Use /3GB switch if you have more than
1G memory (Not available in Windows
2000 Standard)

©2004 Microsoft Corporation. All rights reserved.

Memory Usage by LSASS

LSASS component

Fixed

code, the stacks, the heaps, and various fixed
size data structures (e.g. the schema cache)

Usually 100M


300M memory

Variable (Database Buffer Cache)

Size: 1M to entire database

Windows 2000: ((totalVA
-

1GB) / 2)

Window 2003: Memory 2.6G


fixed memory
usage


©2004 Microsoft Corporation. All rights reserved.

Agenda

Overview of performance analysis for
Active Directory

Using the tools to troubleshoot
common performance issues

Common Active Directory Performance
issues

Putting it all together: How to approach
performance issues in Active Directory


©2004 Microsoft Corporation. All rights reserved.

Measuring Perf

Approach 1:
Client centered

(
2
)
Internal query
processing
Client
Domain Controller
(
1
)
Issue query
(
3
)
Results
(
4
)
Analyze
Pros
: Most accurate representation of
user
\
application experience (latency)

Cons
: Complex to setup due to multiple
systems. Doesn’t take into account multiple
clients causing issues

Tools
: (1) LDP, Scripts, (4) Netmon

©2004 Microsoft Corporation. All rights reserved.

Measuring Perf

Approach 2:
Server centered

Pros
: Easier to measure, more holistic
approach on the server

Cons
: Difficult to track true client
performance

Tools
: (1) LDP, (1,3) Netmon, (2) Server
Performance Advisor, (2) Perfmon

(
2
)
Internal query processing
-
Receive request
-
Allocate resources
-
Marshall and send response
Domain Controller
(
1
)
Queries
(
3
)
Results
Computer
Computer
Computer
Computer
Servers
©2004 Microsoft Corporation. All rights reserved.

Approaching a performance
problem

Step 1
: Get a clear problem description

Step 2
: Isolate and trap failure to the
fewest possible suspects that need to
be analyzed

Step 3
: Log on to server and install
SPA

Step 4
: Run a collection and look at the
report, starting with the summary page.
Analyze historical (saved) data

©2004 Microsoft Corporation. All rights reserved.

Load simulation

ADTEST.EXE

Simulates logons and searches

http://www.microsoft.com/downloads/details.a
spx?FamilyID=4814fe3f
-
92ce
-
4871
-
b8a4
-
99f98b3f4338&DisplayLang=en


©2004 Microsoft Corporation. All rights reserved.

Summary

Use Server Performance Advisor as a
starting point in understanding
performance issues

©2004 Microsoft Corporation. All rights reserved.

More Information

Server Performance Advisor Download:

http://download.microsoft.com/download/6/2/c/62c587bf
-
0d42
-
4ca2
-
9b04
-
5e6771dd209a/spa.msi

Further Reading:

Help text in Server Performance Advisor

AD Deployment Guide

Designing and Deploying Directory and Security Services
(
http://go.microsoft.com/fwlink/?LinkId=4719
)

AD Capacity Planning:

AD Sizer
(
http://www.microsoft.com/windows2000/techinfo/planning/activedirectory/a
dsizer.asp
)

“Planning Domain Controller Capacity”
(
http://www.microsoft.com/resources/documentation/WindowsServ/2003/all/deployguid
e/en
-
us/Default.asp?url=/resources/documentation/WindowsServ/2003/all/deployguide/en
-
us/DSSBJ_DCC_OVERVIEW.asp
)


Event Tracing for Windows:

http://msdn.microsoft.com/library/default.asp?url=/library/en
-
us/perfmon/base/about_event_tracing.asp





©2004 Microsoft Corporation. All rights reserved.

More information

Performance Monitor Wizard

http://www.microsoft.com/downloads/details.asp
x?familyid=31fccd98
-
c3a1
-
4644
-
9622
-
faa046d69214&displaylang=en


Configure the correct counters to collect, sample
intervals and log file sizes

Create logs for troubleshooting OS or Exchange
server performance issues.

Creating More Efficient Microsoft Active
Directory
-
Enabled Applications

http://msdn.microsoft.com/library/default.asp?url
=/library/en
-
us/dnactdir/html/efficientadapps.asp

Memory Usage by LSASS.EXE

http://support.microsoft.com/default.aspx?scid=k
b;en
-
us;Q308356



© 2004 Microsoft Corporation. All rights reserved.

This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.