trouble_re - 자바서비스넷

judgedrunkshipServers

Nov 17, 2013 (3 years and 4 months ago)

126 views

웹서비스

장애

관련

강호성






1.
서버

시스템

구성도

2.
서버

다운

증상

3.
서버

정보

(vmstat )


서버

정보

(netstat )


서버

정보

(netstat )


서버

다운

정보

(startup.sh
옵션

: GC
정보
)


서버

정보

(
ps
-
aux

)


서버

정보

(
메모리

heap
변화

모니터링

)

4.
질문사항

5.
예상

장애

요인


Web Server

Env :
Linux Redhat 9.0


Tomcat 4.1.18


Apache2.0.43

IP :
생략

위치

:
생략

상세내용

:

FileServer
와연동
(URLConnection
이용
)

사양

: LGIBM


intel pentium 3 Xeon 550 Dual


1G Memory

Search Server

Env :
Linux Redhat 9.0


Tomcat 4.1.18


Apache2.0.43

IP :
생략

위치

:
생략

상세내용

:


JSP


Servlet


C
검색모듈

사양

: CPU
-

Xeon 2.6 GHz * EA


Board
-

Intel SE7501BR2


Mem
-

1G SDRAM * 4 EA

File Server

(
웹용

썸네일

이미지

파일들
)

Env : Windows


Tomcat 4.1.18


Apache2.0.43

IP :
생략

위치

:
생략

사양

: CUP 2.4 G / 550MHZ


2G Memory



DataBase

Env : windows + Mssql2000

IP :
생략

위치

:
생략

사양

: HP DL380
모델


CUP 3.06 G / 550MHZ


2G Memory



사용자

실제

원본

Data Server

사용용도

: Web Server


암호모듈


통해서

다운로딩

서비스




기존부터잘사용됨

1.
서버

시스템

구성도

연동방법

DNS : search.XXX.co.kr

Apache : Redirect


이용

연동방법

Thumnail List
출력

: URLConnection
이용

Apache : Redirect


이용

무룡이미지다운서비스

: Socket
통신
,
다운로드

장애발생서버

10 M
단독라인

2.

서버

다운

증상


* Tomcat
로그

(catalina)


May 19, 2004 7:02:28 PM org.apache.tomcat.util.log.CommonLogHandler log INFO: All threads are
busy, waiting. Please increase maxThreads or check the servlet status150 150


TOP
명령어를

이용한



프로세스는

평상시

200


이하
.

서버

장애

시점에는

갑자기

8
개이상씩

증가하면서

5~10
분여

만에



프로세스

400
개이상
.


서버

장애

시점에서의

초기에는

JSP
페이지는

안뜨나
, html
페이지는

열립니다
.

서버

장애

후반에서는

JSP

html
페이지가

뜨지

않는

현상
.



3.

서버

정보

(vmstat )


* vmstat (
평상시
)


procs memory swap io system cpu


r b w swpd free buff cache si so bi bo in cs us sy id


0 0 0 1360 127668 45360 263056 0 0 3 7 9 10 4
2 18


0 0 0 1360 127668 45360 263056 0 0 0 0 728 390 3
1 96


0 0 0 1360 127668 45360 263060 0 0 0 0 877 523 5
2 93


0 0 0 1360 127668 45360 263060 0 0 0 0 656 303 1
1 98


0 0 0 1360 127668 45360 263060 0 0 0 0 727 402 6
1 93


1 0 0 1360 127668 45360 263060 0 0 0 100 835 472
6 2 92


0 0 0 1360 127668 45360 263064 0 0 0 0 860 440 3
4 93


0 0 0 1360 127660 45360 263064 0 0 0 0 883 456 5
2 93


0 0 0 1360 127660 45360 263064 0 0 0 0 658 283 2
0 98


0 0 0 1360 127660 45360 263064 0 0 0 0 779 441 1
2 96


0 0 0 1360 127660 45360 263064 0 0 0 104 697 352
1 1 98


0 0 0 1360 127668 45360 263064 0 0 0 0 573 190 0
4 95


0 0 0 1360 127660 45360 263064 0 0 0 0 873 477 8
2 90


0 0 0 1360 127660 45360 263068 0 0 0 0 943 479 1
2 97


1 0 0 1360 127660 45360 263068 0 0 0 0 942 716 8
2 89


0 0 0 1360 127660 45364 263068 0 0 0 132 732 393
2 2 96


0 0 0 1360 127660 45364 263068 0 0 0 0 562 179 1
0 99


1 0 0 1360 127660 45364 263080 0 0 12 0 567 192 2
1 96


0 0 0 1360 127660 45364 263080 0 0 0 0 779 389 8
1 90


0 0 0 1360 127660 45364 263080 0 0 0 0 740 413 1
6 93


0 0 0 1360 127660 45364 263080 0 0 0 76 690 327 5
2 92


0 0 0 1360 127660 45364 263084 0 0 0 0 790 492 4
1 95


0 0 0 1360 127660 45364 263084 0 0 0 0 689 372 4
3 93


0 0 0 1360 127652 45364 263084 0 0 0 0 962 646 6
6 88


0 0 0 1360 127652 45364 263088 0 0 0 0 877 529 2
1 97


0 0 0 1360 127652 45364 263088 0 0 0 108 768 450
4 2 94


0 0 0 1360 127644 45364 263088 0 0 0 0 669 287 1
1 98


0 0 0 1360 127644 45364 263088 0 0 0 0 680 644 8
1 90


0 0 0 1360 127636 45364 263088 0 0 0 0 803 510 4
2 94


0 0 0 1360 127636 45364 263088 0 0 0 0 647 301 3
1 95


0 0 0 1360 127644 45368 263152 0 0 68 96 745 460
6 2 92


1 0 0 1360 127652 45368 263164 0 0 12 0 782 428 3
2 95


* vmstat (
서버장애시
)


procs memory swap io system cpu


r b w swpd free buff cache si so bi bo in cs us sy id

0 0 0 1360 92372 44664 278980 0 0 0 0 644 327 2 1 97


0 0 0 1360 92096 44664 278980 0 0 0 0 584 177 1 1 98


0 0 0 1360 92064 44668 278980 0 0 0 16 563 178 4 2 94


0 0 0 1360 91752 44668 278980 0 0 0 0 601 228 0 2 98

0 0 0 1360 91384 44668 278980 0 0 0 0 578 191 1 1 97


0 0 0 1360 90500 44668 278980 0 0 0 0 593 241 2 3 95


0 0 0 1360 89320 44668 278980 0 0 0 0 572 212 1 4 95


0 0 0 1360 89188 44672 278976 0 0 0 44 589 208 2 3 94


0 0 0 1360 89064 44672 278976 0 0 0 0 580 181 0 2 97


0 0 0 1360 88860 44672 278976 0 0 0 0 582 207 2 3 95


0 0 0 1360 88792 44672 278976 0 0 0 0 565 173 1 4 94


0 0 0 1360 88728 44672 278976 0 0 0 32 594 186 0 1 98


0 0 0 1360 88552 44672 278976 0 0 0 0 607 216 1 2 96


0 0 0 1360 88164 44672 278976 0 0 0 0 566 170 1 1 98


0 0 0 1360 88156 44672 278976 0 0 0 0 580 172 0 1 98


0 0 0 1360 87848 44672 278976 0 0 0 0 618 213 1 2 97


0 0 0 1360 87356 44672 278976 0 0 0 28 615 266 4 3 93


0 0 0 1360 86848 44672 278976 0 0 0 0 575 180 1 2 97


0 0 0 1360 86848 44672 278976 0 0 0 0 570 166 1 5 94


0 0 0 1360 86756 44672 278976 0 0 0 0 587 193 1 2 97


0 0 0 1360 86700 44672 278976 0 0 0 0 587 173 1 2 97


0 0 0 1360 86492 44672 278976 0 0 0 24 577 205 2 2 95


0 0 0 1360 86300 44672 278976 0 0 0 0 569 179 2 6 92


0 0 0 1360 85988 44672 278976 0 0 0 0 582 188 1 1 98


0 0 0 1360 85540 44672 278976 0 0 0 0 606 260 3 2 95


0 0 0 1360 85416 44672 278976 0 0 0 0 571 170 1 2 97


0 0 0 1360 84944 44672 278976 0 0 0 40 609 270 2 2 96


0 0 0 1360 83992 44672 278976 0 0 0 0 653 289 2 3 95


1 0 0 1360 82340 44672 278980 0 0 0 0 640 294 3 2 94


0 0 0 1360 80124 44672 278980 0 0 0 0 590 250 1 4 95


0 0 0 1360 79724 44672 278980 0 0 0 48 605 202 2 6 92

3.

서버

정보

(netstat : Established)


* netstat
-
n | grep EST (
평상시
)

tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52493 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52492 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52486 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:22 xxx.xxx.29.4:4586 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52575 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52571 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:80 xxx.xxx.152.9:4608 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52567 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52553 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:52493 xxx.xxx.xx.xx:8009 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:52492 xxx.xxx.xx.xx:8009 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:52486 xxx.xxx.xx.xx:8009 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:52571 xxx.xxx.xx.xx:8009 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:52575 xxx.xxx.xx.xx:8009 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:52567 xxx.xxx.xx.xx:8009 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:52553 xxx.xxx.xx.xx:8009 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:52475 xxx.xxx.xx.xx:8009 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:52478 xxx.xxx.xx.xx:8009 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:52464 xxx.xxx.xx.xx:8009 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52478 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52475 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:8009 xxx.xxx.xx.xx:52464 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:80 xxx.xxx.9.59:1625 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:51512 database
-
ip:1433 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:51513 database
-
ip:1433 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:51778 database
-
ip:1433 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:51778 database
-
ip:1433 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:50330 database
-
ip:1433 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:50742 database
-
ip:1433 ESTABLISHED

tcp 0 68 xxx.xxx.xx.xx:50914 database
-
ip:1433 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:50913 database
-
ip:1433 ESTABLISHED

tcp 0 0 xxx.xxx.xx.xx:50952 database
-
ip:1433 ESTABLISHED

tcp 0 904 xxx.xxx.xx.xx:22 xxx.xxx.xx.x:1112 ESTABLISHED

-------------------------------------------------------------------------------------------------



ESTABLISHED
개수

: 33


Database
연동관련

: 9


8009
포트



기타

포트

: 10


기타

포트



8009
포트

: 10


80
포트



client : 2







* netstat
-
n | grep EST (
서버장애시
)

Active Internet connections (w/o servers)

Proto Recv
-
Q Send
-
Q Local Address Foreign Address State

tcp 0 0 webserver
-
ip:80 xxx..xxx.49.174:19848 ESTABLISHED

tcp 0 0 webserver
-
ip:8009 webserver
-
ip:36670 ESTABLISHED

tcp 0 0 webserver
-
ip:80 xxx.xxx.242.71:2976 ESTABLISHED

tcp 0 0 webserver
-
ip:22 xxx.xxx.29.4:4586 ESTABLISHED

tcp 0 0 webserver
-
ip:8009 webserver
-
ip:36703 ESTABLISHED

tcp 0 0 webserver
-
ip:80 xxx.xxx.197.146:1682 ESTABLISHED

tcp 0 0 webserver
-
ip:8009 webserver
-
ip:36696 ESTABLISHED


: : : :
기타

15


동일

내용

(
상동
)


tcp 0 0 webserver
-
ip:80 xxx.xxx.xxx.xx:2177 ESTABLISHED

tcp 0 0 webserver
-
ip:8009 webserver
-
ip:36717 ESTABLISHED


: : : :
기타

28


동일

내용

(
상동
)

tcp 0 0 webserver
-
ip:80 xxx.xxx.158.13:3051 ESTABLISHED

tcp 0 0 webserver
-
ip:8009 webserver
-
ip:36744 ESTABLISHED


: : : :
기타

36


동일

내용

(
상동
)

tcp 0 0 webserver
-
ip:80 xxx.xxx.xxx.107:2604 ESTABLISHED

tcp 0 0 webserver
-
ip:8009 webserver
-
ip:36774 ESTABLISHED


: : : :
기타

6


동일

내용

(
상동
)

tcp 0 0 webserver
-
ip:36817 webserver
-
ip:8009 ESTABLISHED


: : : :
기타

106


동일

내용

(
상동
)

tcp 0 0 webserver
-
ip:8009 webserver
-
ip:36822 ESTABLISHED


: : : :
기타

20


동일

내용

(
상동
)


tcp 0 0 webserver
-
ip:80 xx8.xxx.31.12:3735 ESTABLISHED

tcp 0 0 webserver
-
ip:80 xxx.xxx.2.102:2354 ESTABLISHED

tcp 0 0 webserver
-
ip:35038 database
-
ip:1433 ESTABLISHED

tcp 0 0 webserver
-
ip:35213 database
-
ip:1433 ESTABLISHED

tcp 0 0 webserver
-
ip:35214 database
-
ip:1433 ESTABLISHED

tcp 0 0 webserver
-
ip:33816 database
-
ip:1433 ESTABLISHED

tcp 0 0 webserver
-
ip:80 xxx.215.xx5.55:3877 ESTABLISHED


: : : :
기타

23


동일

내용

(
상동
)

-------------------------------------------------------------------------------------------------



ESTABLISHED
개수

: 256


Database
연동관련

: 4


8009
포트



기타

포트

: 102


기타

포트



8009
포트

: 107


80
포트



client : 32



3.

서버

다운

정보

(startup.sh
옵션

: GC
정보
)


* GC (
평상시
)

[GC 39885K
-
>37285K(40124K), 0.0071930 secs]

[GC 39887K
-
>37605K(40324K), 0.0100080 secs]

[Full GC 37605K
-
>16401K(40324K), 0.7960270 secs]

[GC 19328K
-
>17062K(40124K), 0.0126660 secs]

[GC 19892K
-
>17599K(40124K), 0.0111760 secs]

[GC 20432K
-
>18012K(40124K), 0.0408080 secs]

[GC 20850K
-
>18626K(40124K), 0.0109500 secs]

[GC 21463K
-
>18830K(40124K), 0.0118570 secs]

[GC 21451K
-
>19196K(40124K), 0.0095420 secs]

[GC 22037K
-
>19555K(40124K), 0.0093910 secs]

[GC 22399K
-
>20149K(40124K), 0.0098520 secs]

[GC 22854K
-
>20379K(40124K), 0.0083420 secs]

[GC 22996K
-
>20612K(40124K), 0.0076090 secs]

[GC 23455K
-
>20845K(40124K), 0.0063710 secs]

[GC 23655K
-
>21308K(40124K), 0.0075850 secs]

[GC 24370K
-
>21892K(40124K), 0.0072240 secs]

[GC 24732K
-
>22138K(40124K), 0.0060080 secs]

[GC 24949K
-
>22380K(40124K), 0.0061300 secs]

[GC 39762K
-
>37170K(40124K), 0.0058780 secs]

[GC 39727K
-
>37429K(40300K), 0.0068420 secs]

[Full GC 37429K
-
>16191K(40300K), 0.7953360 secs]

[GC 39679K
-
>37276K(40124K), 0.0069350 secs]

[GC 39925K
-
>37610K(40300K), 0.0076660 secs]

[Full GC 37610K
-
>14276K(40300K), 0.8748960 secs]

[GC 17581K
-
>15373K(40124K), 0.0095520 secs]

[GC 18449K
-
>15861K(40124K), 0.0104900 secs]

[GC 18483K
-
>16526K(40124K), 0.0137420 secs]

[GC 19146K
-
>17209K(40124K), 0.0346310 secs]

[GC 19832K
-
>17766K(40124K), 0.0206310 secs]

[GC 20389K
-
>17794K(40124K), 0.0103710 secs]

[GC 20391K
-
>17813K(40124K), 0.0061950 secs]

[GC 20584K
-
>17906K(40124K), 0.0054300 secs]

[GC 20498K
-
>17998K(40124K), 0.0061370 secs]

[GC 20617K
-
>18008K(40124K), 0.0054560 secs]

[GC 20730K
-
>18018K(40124K), 0.0055230 secs]

[GC 20619K
-
>18137K(40124K), 0.0055160 secs]

[GC 20735K
-
>18145K(40124K), 0.0058770 secs]





GC (
서버장애시
)

[GC 35807K
-
>32869K(44368K), 0.0074160 secs]

[GC 44010K
-
>41290K(44368K), 0.0071010 secs]

[Full GC 41290K
-
>16677K(44368K), 0.9458450 secs]

[GC 22642K
-
>20059K(44368K), 0.0078300 secs]

[GC 33637K
-
>30787K(44368K), 0.0072790 secs]

[GC 43817K
-
>41049K(44368K), 0.0063940 secs]

[GC 43865K
-
>41657K(44496K), 0.0124660 secs]

[Full GC 41657K
-
>17253K(44496K), 0.9132000 secs]

[GC 23044K
-
>20879K(44368K), 0.0138540 secs]

[GC 23691K
-
>21423K(44368K), 0.0141100 secs]

[GC 24234K
-
>21868K(44368K), 0.0131710 secs]

[GC 24684K
-
>23311K(44368K), 0.0262750 secs]

[GC 26126K
-
>23893K(44368K), 0.0163500 secs]

[GC 26708K
-
>24277K(44368K), 0.0124560 secs]

[GC 27081K
-
>24749K(44368K), 0.0143000 secs]

[GC 27563K
-
>25908K(44368K), 0.0191340 secs]

[GC 28708K
-
>26359K(44368K), 0.0139570 secs]

[GC 29175K
-
>27535K(44368K), 0.0185610 secs]

GC 30351K
-
>27820K(44368K), 0.0125210 secs]

[GC 30635K
-
>28441K(44368K), 0.0281460 secs]

[GC 31247K
-
>28785K(44368K), 0.0128440 secs]

[GC 31600K
-
>29805K(44368K), 0.0275490 secs]

[GC 32621K
-
>31303K(44368K), 0.0394960 secs]

[GC 34118K
-
>33026K(44368K), 0.0486900 secs]

[GC 35836K
-
>34692K(44368K), 0.0554940 secs]

[GC 37508K
-
>36243K(44368K), 0.0576330 secs]

[GC 39056K
-
>37789K(44368K), 0.0641460 secs]

[GC 40605K
-
>39362K(44368K), 0.0687840 secs]

[GC 42178K
-
>41334K(44368K), 0.0804310 secs]

[GC 44149K
-
>42679K(45520K), 0.0803990 secs]

---------------------------------------------

여기부터

서버

느려서

페이지가

뜨지않음

[Full GC 42679K
-
>31065K(45520K), 1.3358060 secs]

--------------------------------------------


페이지

안뜬상태에서

조금있다가

로그

남음


[GC 34649K
-
>33041K(55748K), 0.0914980 secs]

--------------------------------------------


페이지

안뜬상태에서

조금있다가

로그

남음

[GC 36624K
-
>35406K(55748K), 0.1062460 secs]

--------------------------------------------


웹서비스를

위해

톰캣을

재시작처리함



3.

서버

정보

(
ps
-
aux

)


* ps
내용

(
평상시
)

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

root 1 0.0 0.0 1400 468 ? S May15 0:19 init

root 2 0.0 0.0 0 0 ? SW May15 0:00 [migration_CPU0]

root 3 0.0 0.0 0 0 ? SW May15 0:00 [migration_CPU1]

root 4 0.0 0.0 0 0 ? SW May15 0:00 [keventd]

root 5 0.0 0.0 0 0 ? SWN May15 0:00 [ksoftirqd_CPU0]

root 6 0.0 0.0 0 0 ? SWN May15 0:00 [ksoftirqd_CPU1]

root 7 0.0 0.0 0 0 ? SW May15 2:43 [kswapd]

root 8 0.0 0.0 0 0 ? SW May15 0:05 [bdflush]

root 9 0.0 0.0 0 0 ? SW May15 0:10 [kupdated]

root 10 0.0 0.0 0 0 ? SW May15 0:00 [mdrecoveryd]

root 16 0.0 0.0 0 0 ? SW May15 0:00 [scsi_eh_0]

root 19 0.0 0.0 0 0 ? SW May15 1:08 [kjournald]

root 75 0.0 0.0 0 0 ? SW May15 0:00 [khubd]

root 167 0.0 0.0 0 0 ? SW May15 0:00 [kjournald]

root 441 0.0 0.0 1472 524 ? S May15 0:02 syslogd
-
m 0

root 445 0.0 0.0 1400 444 ? S May15 0:00 klogd
-
x

root 483 0.0 0.1 3356 1260 ? S May15 0:18 /usr/sbin/sshd

root 494 0.0 0.0 2036 608 ? S May15 0:00 xinetd
-
stayalive
-
reuse
-
pidfile
/var/run

root 513 0.0 0.1 5604 1512 ? S May15 0:43 [sendmail]

smmsp 522 0.0 0.1 4952 1372 ? S May15 0:00 [sendmail]

root 532 0.0 0.0 1460 580 ? S May15 0:01 crond

daemon 550 0.0 0.0 1444 544 ? S May15 0:00 [atd]

root 561 0.0 0.0 3432 544 ? S May15 0:00 rhnsd
--
interval 120

root 567 0.0 0.0 1376 368 tty1 S May15 0:00 /sbin/mingetty tty1

root 3974 0.0 0.2 6936 2156 ? S May24 0:29 /usr/sbin/sshd

root 4059 0.0 0.1 3228 1176 ? S May24 0:01 /usr/libexec/openssh/sftp
-
server

root 4087 0.0 0.1 6868 2048 ? S May24 0:01 /usr/sbin/sshd

root 4089 0.0 0.1 3208 1108 ? S May24 0:00 /usr/libexec/openssh/sftp
-
server

root 26908 8.2 57.8 728308 596340 ? S 08:58 10:17 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 26925 0.0 0.2 4972 2344 ? S 08:58 0:00 /usr/local/httpd/bin/httpd
-
k start

root 27415 0.0 0.1 4400 1464 pts/0 S 11:02 0:00
-
bash

nobody 27502 0.0 0.2 5080 2556 ? S 11:08 0:00 [httpd]


nobody 27511 0.0 0.2 5076 2524 ? S 11:09 0:00 [httpd]

nobody 27527 0.0 0.2 5080 2552 ? S 11:11 0:00 [httpd]

nobody 27528 0.0 0.2 5080 2552 ? S 11:11 0:00 [httpd]

nobody 27530 0.0 0.2 5076 2520 ? S 11:12 0:00 [httpd]

nobody 27531 0.0 0.2 5072 2544 ? S 11:12 0:00 [httpd]

nobody 27533 0.0 0.2 5076 2516 ? S 11:12 0:00 [httpd]

nobody 27535 0.0 0.2 5068 2512 ? S 11:14 0:00 [httpd]

nobody 27540 0.0 0.2 5068 2508 ? S 11:17 0:00 [httpd]

nobody 27543 0.0 0.2 5068 2504 ? S 11:17 0:00 [httpd]

nobody 27544 0.0 0.2 5068 2504 ? S 11:17 0:00 [httpd]

root 27545 0.0 0.0 2776 836 pts/0 R 11:17 0:02 ps
-
aux


* ps
내용

(
평상시
)

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

root 1 0.0 0.0 1400 468 ? S May15 0:19 init

root 2 0.0 0.0 0 0 ? SW May15 0:00 [migration_CPU0]

root 3 0.0 0.0 0 0 ? SW May15 0:00 [migration_CPU1]

root 4 0.0 0.0 0 0 ? SW May15 0:00 [keventd]

root 5 0.0 0.0 0 0 ? SWN May15 0:00 [ksoftirqd_CPU0]

root 6 0.0 0.0 0 0 ? SWN May15 0:00 [ksoftirqd_CPU1]

root 7 0.0 0.0 0 0 ? SW May15 2:43 [kswapd]

root 8 0.0 0.0 0 0 ? SW May15 0:05 [bdflush]

root 9 0.0 0.0 0 0 ? SW May15 0:10 [kupdated]

root 10 0.0 0.0 0 0 ? SW May15 0:00 [mdrecoveryd]

root 16 0.0 0.0 0 0 ? SW May15 0:00 [scsi_eh_0]

root 19 0.0 0.0 0 0 ? SW May15 1:08 [kjournald]

root 75 0.0 0.0 0 0 ? SW May15 0:00 [khubd]

root 167 0.0 0.0 0 0 ? SW May15 0:00 [kjournald]

root 441 0.0 0.0 1472 524 ? S May15 0:02 syslogd
-
m 0

root 445 0.0 0.0 1400 444 ? S May15 0:00 klogd
-
x

root 483 0.0 0.1 3356 1260 ? S May15 0:18 /usr/sbin/sshd

root 494 0.0 0.0 2036 608 ? S May15 0:00 xinetd
-
stayalive
-
reuse
-
pidfile /var/run

root 513 0.0 0.1 5604 1512 ? S May15 0:44 [sendmail]

smmsp 522 0.0 0.1 4952 1372 ? S May15 0:00 [sendmail]

root 532 0.0 0.0 1460 580 ? S May15 0:01 crond

daemon 550 0.0 0.0 1444 544 ? S May15 0:00 [atd]

root 561 0.0 0.0 3432 544 ? S May15 0:00 rhnsd
--
interval 120

root 567 0.0 0.0 1376 368 tty1 S May15 0:00 /sbin/mingetty tty1

root 3974 0.0 0.2 6936 2156 ? S May24 0:29 /usr/sbin/sshd

root 4059 0.0 0.1 3228 1176 ? S May24 0:01 /usr/libexec/openssh/sftp
-
server

root 4087 0.0 0.1 6868 2048 ? S May24 0:01 /usr/sbin/sshd

root 4089 0.0 0.1 3208 1108 ? S May24 0:00 /usr/libexec/openssh/sftp
-
server

root 26908 5.8 58.5 738628 603804 ? S 08:58 9:33 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 26925 0.0 0.2 4972 2344 ? S 08:58 0:01 /usr/local/httpd/bin/httpd
-
k start

root 27066 0.3 58.5 738628 603804 ? S 10:07 0:16 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27067 0.3 58.5 738632 603816 ? S 10:07 0:19 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27072 0.4 58.5 738628 603816 ? S 10:07 0:22 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27073 0.4 58.5 738628 603816 ? S 10:07 0:23 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27074 0.2 58.5 738628 603816 ? S 10:07 0:13 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27075 0.2 58.5 738628 603816 ? S 10:07 0:13 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27079 0.0 58.5 738628 603816 ? S 10:07 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27080 0.0 58.5 738628 603816 ? S 10:07 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27081 0.0 58.5 738628 603816 ? S 10:07 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27082 0.0 58.5 738628 603816 ? S 10:07 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27087 0.0 58.5 738628 603816 ? S 10:07 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27088 0.0 58.5 738628 603816 ? S 10:07 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27089 0.0 58.5 738628 603816 ? S 10:07 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27090 0.3 58.5 738628 603816 ? S 10:07 0:15 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27415 0.0 0.1 4400 1464 pts/0 S 11:02 0:00
-
bash

nobody 27620 0.0 0.2 5068 2512 ? S 11:27 0:00 [httpd]

nobody 27622 0.0 0.2 5076 2524 ? S 11:27 0:00 [httpd]

nobody 27624 0.0 0.2 5076 2516 ? S 11:28 0:00 [httpd]

nobody 27633 0.0 0.2 5080 2552 ? S 11:30 0:00 [httpd]

nobody 27637 0.0 0.2 5080 2552 ? S 11:30 0:00 [httpd]

nobody 27638 0.0 0.2 5080 2552 ? S 11:30 0:00 [httpd]

nobody 27642 0.0 0.2 5080 2548 ? S 11:30 0:00 [httpd]

nobody 27643 0.0 0.2 5068 2508 ? S 11:30 0:00 [httpd]

nobody 27645 0.0 0.2 5068 2496 ? S 11:31 0:00 [httpd]

nobody 27646 0.0 0.2 5068 2488 ? S 11:31 0:00 [httpd]

nobody 27647 0.0 0.2 5052 2480 ? S 11:31 0:00 [httpd]

nobody 27648 0.0 0.2 5052 2480 ? S 11:31 0:00 [httpd]

nobody 27649 0.0 0.2 5052 2480 ? S 11:31 0:00 [httpd]

nobody 27650 0.0 0.2 5052 2480 ? S 11:31 0:00 [httpd]

nobody 27651 0.0 0.2 5052 2488 ? S 11:31 0:00 [httpd]

nobody 27652 0.0 0.2 5052 2492 ? S 11:31 0:00 [httpd]

nobody 27653 0.0 0.2 5044 2472 ? S 11:32 0:00 [httpd]

nobody 27654 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd]

nobody 27655 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd]

nobody 27656 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd]

nobody 27657 0.0 0.2 5052 2488 ? S 11:32 0:00 [httpd]

nobody 27658 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd]

nobody 27659 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd]

nobody 27660 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd]

nobody 27661 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd]

nobody 27662 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd]

nobody 27663 0.0 0.2 5060 2496 ? S 11:32 0:00 [httpd]

nobody 27664 0.0 0.2 5060 2484 ? S 11:32 0:00 [httpd]

nobody 27665 0.0 0.2 5052 2492 ? S 11:32 0:00 [httpd]

nobody 27666 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd]

nobody 27667 0.0 0.2 5060 2484 ? S 11:32 0:00 [httpd]

nobody 27668 0.0 0.2 5068 2500 ? S 11:32 0:00 [httpd]

nobody 27669 0.0 0.2 5068 2496 ? S 11:32 0:00 [httpd]

nobody 27670 0.0 0.2 5068 2488 ? S 11:32 0:00 [httpd]

nobody 27671 0.0 0.2 5068 2496 ? S 11:32 0:00 [httpd]

nobody 27672 0.0 0.2 5060 2492 ? S 11:32 0:00 [httpd]

nobody 27673 0.0 0.2 5068 2484 ? S 11:32 0:00 [httpd]

nobody 27674 0.0 0.2 5076 2500 ? S 11:32 0:00 [httpd]

nobody 27675 0.0 0.2 5060 2484 ? S 11:32 0:00 [httpd]

nobody 27676 0.0 0.2 5076 2496 ? S 11:32 0:00 [httpd]

nobody 27677 0.0 0.2 5052 2488 ? S 11:32 0:00 [httpd]

nobody 27678 0.0 0.2 5052 2480 ? S 11:32 0:00 [httpd]

nobody 27679 0.0 0.2 5068 2504 ? S 11:32 0:00 [httpd]

nobody 27680 0.0 0.2 5060 2488 ? S 11:32 0:00 [httpd]

nobody 27681 0.0 0.2 5060 2488 ? S 11:33 0:00 [httpd]

nobody 27682 0.0 0.2 5060 2484 ? S 11:33 0:00 [httpd]

nobody 27683 0.0 0.2 5052 2488 ? S 11:33 0:00 [httpd]

nobody 27684 0.0 0.2 5068 2496 ? S 11:33 0:00 [httpd]

nobody 27685 0.0 0.2 5060 2484 ? S 11:33 0:00 [httpd]

nobody 27686 0.0 0.2 5052 2488 ? S 11:33 0:00 [httpd]

root 27687 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27688 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27689 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27690 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

nobody 27691 0.0 0.2 5052 2484 ? S 11:33 0:00 [httpd]

root 27692 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27693 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27694 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27695 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

nobody 27696 0.0 0.2 5060 2492 ? S 11:33 0:00 [httpd]

nobody 27697 0.0 0.2 5076 2500 ? S 11:33 0:00 [httpd]

nobody 27698 0.0 0.2 5060 2492 ? S 11:33 0:00 [httpd]

nobody 27699 0.0 0.2 5060 2492 ? S 11:33 0:00 [httpd]

nobody 27700 0.0 0.2 5076 2508 ? S 11:33 0:00 [httpd]

root 27701 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27702 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27703 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27704 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

nobody 27705 0.0 0.2 5068 2508 ? S 11:33 0:00 [httpd]

nobody 27706 0.0 0.2 5068 2496 ? S 11:33 0:00 [httpd]

nobody 27707 0.0 0.2 5044 2472 ? S 11:33 0:00 [httpd]

nobody 27708 0.0 0.2 5076 2508 ? S 11:33 0:00 [httpd]

nobody 27709 0.0 0.2 5068 2496 ? S 11:33 0:00 [httpd]

nobody 27710 0.0 0.2 5104 2540 ? S 11:33 0:00 [httpd]

nobody 27711 0.0 0.2 5068 2496 ? S 11:33 0:00 [httpd]

nobody 27712 0.0 0.2 5044 2476 ? S 11:33 0:00 [httpd]

root 27713 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27714 0.1 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27715 0.1 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27716 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27717 0.2 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27718 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27719 0.0 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27720 0.2 58.5 738628 603816 ? S 11:33 0:00 /usr/java/j2se/bin/java
-
Xmx512m
-
Xms512m

root 27721 0.0 0.0 2844 904 pts/0 R 11:33 0:02 ps
-
aux

nobody 27722 0.0 0.2 4996 2384 ? S 11:33 0:00 [httpd]


3.

서버

정보

(
메모리

heap
변화

모니터링

)


*
메모리

heap (
평상시
)


Tue May 25 11:47:28 KST 2004

F:443MB T:508 MB

F:469MB T:508 MB

Tue May 25 11:49:23 KST 2004

F:466MB T:508 MB

F:491MB T:508 MB

Tue May 25 11:51:11 KST 2004

F:466MB T:508 MB

F:495MB T:508 MB

Tue May 25 11:53:27 KST 2004

F:434MB T:508 MB

F:476MB T:508 MB

Tue May 25 12:08:09 KST 2004

F:347MB T:508 MB

F:471MB T:508 MB

Tue May 25 12:08:25 KST 2004

F:452MB T:508 MB

F:468MB T:508 MB

Tue May 25 12:10:32 KST 2004

F:481MB T:508 MB

F:487MB T:508 MB

Tue May 25 12:15:33 KST 2004

F:431MB T:508 MB

F:470MB T:508 MB

Tue May 25 13:47:02 KST 2004

F:148MB T:508 MB

F:462MB T:508 MB

Tue May 25 14:38:21 KST 2004

F:77MB T:508 MB

F:461MB T:508 MB


메모리

heap (
서버장애시
)



JSP
페이지

자체가

뜨지

않아

파악하지

못함



4.

N/W
정보

(10M
단독

라인
)

[
주간

]

[
월간

]

[
년간

]

4.
질문

사항


* Runtime.exec()


통해

JVM
외부의

shell script


c program


호출하는

부분이

있는지요
?


현재

다운되는

서버

내의

JVM
에서는

shell
이나

c


호출하는

부분이

없습니다
.


*
프로그램

어디선가

Socket
혹은

HttpUrlConnection
으로

"localhost:80"
으로

접속을

시도하는

부분이

있는지요
?


어떤

용도로

쓰였는지요
?


Socket


파일

서버



연동중

무료이미지

관련

부분이

사용되고

있습니다
. (
첨부파일참조
)


URLConnection


파일서버와

연동중

폴더내

파일리스트를

가져오는

부분이

있습니다
. (
첨부파일참조
)


* java process
개수가

80+1
에서

80


갖는

의미가

뭘까요
?
어떤

파라메터가

80
인가요
?


저도

정확히는

모르겠습니다만

아파치

프로세스중에

좀비가

가끔씩

뜨지만

잠시

떴다가

금방

사라지며
,
이부분에


대해

자바서비스넷

자료를

찾는중에

금방

사라지는

좀비프로세스는

괜찮다는

글을

읽은걸로

알고있습니다
.


* dump
파일

생성

관련



ibm JDK


아닌

sun JDK
사용으로

인하여

dump
파일이

생기지

않습니다
.


* server.xml


내용



<Host name="test.test.co.kr" debug="0" appBase=

appbase" liveDeploy="true" unpackWARs="true" autoDeploy="true">


<Context cookies="true" docBase=

test" path="/test" reloadable="true" debug="0" crossContext="false">


<Logger className="org.apache.catalina.logger.FileLogger" prefix="localhost_test_log." suffix=".txt" timest
amp="true"/>


<Realm className="org.apache.catalina.realm.JDBCRealm"


connectionName="sa" connectionPassword=

xxxxx"


connectionURL="jdbc:microsoft:sqlserver://database
-
ip:1433;DatabaseName=test;SelectMethod=cursor;SendStringParametersAsUnicode=true"


debug="0"


driverName="com.microsoft.jdbc.sqlserver.SQLServerDriver"


userTable="T_testmember" userNameCol=

userid" userCredCol=

pwd"


userRoleTable="T_

" roleNameCol="U_

"/>


<ResourceLink global="jdbc/TESTDB" name="jdbc/TESTDB"/>


</Context>


~~~~~~~~~~~~
이하생략

5.
예상

장애

요인

FileServer
페이지

<%@page
선언부

%>

<%@page import=

~~

%>

<% String fpath="";


fpath=(String)(request.getParameter("fpath")); //
---
request
받음


int filesize=0;


try {


File file = new File(fpath);


filesize=(int)file.length();


if(filesize<1){


String fileList[]=new String[filesize];


if(file.list()!=null){


fileList=file.list();


for(int k=0;k<fileList.length;k++){



File file_List = new File(fpath+fileList[k]);



out.println(file_List.getName());


}//
-------
end for for(int k=0;k<


}else{//
---
if(file.list()!=null


out.println("");


}


file_List=null;


}else{


out.println("");


}


} catch(Exception e) {


e.printStackTrace();


}finally{


System.out.println("fpath==>"+fpath);


}

%>

1


이미지

다운로드

관련

로직

(
첨부파일

참조

) :
웹서버



파일서버

소켓연동


(
파일서버에서

socket


체크후

해제

처리를

합니다
.
현재

첨부파일소스는

웹서버의

client
소스

입니다
.)


2


폴더내

이미지

리스트

호출

관련

로직

:
웹서버



파일서버

URLConnection
이용



아래

소스

참고

WebServer
페이지

public static Vector mbJspFileList(String path){


Vector list = new Vector();


BufferedInputStream is = null;


try {


URL url = new URL("http://FileServer
페이지주소
?fpath="+path);


URLConnection conn = url.openConnection();


if(conn!=null){


is = new BufferedInputStream(conn.getInputStream());


int _value;


char value;


StringBuffer sb = new StringBuffer();


while((_value = is.read())!=
-
1){


value = (char)_value;


if(value != '
\
n') sb.append(value);


else {


list.addElement(sb.toString());


sb = new StringBuffer();


}


}


}else{


list=null;


}


}catch(Exception e){


e.printStackTrace();


}finally {


if(is!=null){try {is.close();}catch(Exception e){}}


}

return list;

}