InspectOS - Extending Features for Operating System ... - ixlabs

brickcompetitiveΛογισμικό & κατασκευή λογ/κού

14 Δεκ 2013 (πριν από 3 χρόνια και 9 μήνες)

133 εμφανίσεις











InspectOS
-

Extending Features for
Operating System Inspection


















Supervisors:

Drd. Ing. Octavian Purdil
ă

As. Drd. Răzvan Deaconescu

Author:

Alexandru Șutîi





Bachelor Thesis

Computer Science and Engineering Department

Automa
tic Control and Computers

Faculty

University
Politehnica
of Bucharest

July
2011




A
bstract


Operating Systems are one of the most interesting Computer Science fields. In order to become an
excellent software engineer,
one

should have at least basi
c knowled
ge about the operating s
ystems
concepts. However, for most of the students it is hard to und
erstand many

of

them.

The most spread teaching technique is
with repeated explanations of the low level mechanisms. As
there are some tools that inspect the processes for industrial purposes, there is a lack of educational
tools that would present the OS c
oncepts dynamically to the students based on

a

given program
.

InspectOS is coming to solve this problem. It is a useful educational tool that gathers information
about a traced process directly from the running operating system. It comes to exten
d the OS
-
T
ool,
an application

developed a year before by our
colleagues

Lucian Cojocar,

Vlad Dogaru and Andrei
Buhaiu.

In this paper we shall present both the initial and the added features
, however

focusing more

on the
design, the implementation and the testing of
the newly added features.




Contents

InspectOS
-

Extending Features for Operating System Inspection

................................
.........................

1

Abstract

................................
................................
................................
................................
...................

2

Acronyms

................................
................................
................................
................................
................

5

1

Introduction

................................
................................
................................
................................
....

6

1.1

InspectOS

................................
................................
................................
................................

6

2

Operating System Theory

................................
................................
................................
...............

7

3

Architecture and Features

................................
................................
................................
..............

9

3.1

Architecture

................................
................................
................................
............................

9

3.2

OS
-
Tool Features

................................
................................
................................
..................

10

3.3

InspectOS New Features

................................
................................
................................
.......

11

4

Implementation

................................
................................
................................
............................

12

4.1

Underlying Technology

................................
................................
................................
.........

12

4.1.1

GTK

................................
................................
................................
................................

12

4.1.2

Graphviz and Cairo

................................
................................
................................
........

12

4.1.3

GDB

................................
................................
................................
...............................

12

4.2

Graphical User

Interface Redesign

................................
................................
.......................

13

4.3

GDB Asynchronous Commands

................................
................................
............................

16

4.3.1

Another Possible Implementation

................................
................................
................

17

4.3.2

A Known Issue

................................
................................
................................
...............

17

4.4

Thre
ad Support

................................
................................
................................
.....................

17

4.4.1

Problems encountered

................................
................................
................................
.

19

4.5

Process Standard Input

................................
................................
................................
.........

19

4.5.1

Known Issues

................................
................................
................................
................

20

4.6

Process Standard Output

................................
................................
................................
......

20

4.6.1

The First Failed Approach


Redirect the Process Output

................................
............

20

4.6.2

The Second Failed Approach


Parse the GDB Output

................................
.................

20

4.6.3

The Final Solution

................................
................................
................................
.........

20

4.7

Variable Value Display

................................
................................
................................
..........

22

5

Test Scenarios

................................
................................
................................
...............................

23

5.1

Compiling and Tracing a Process

................................
................................
..........................

23

5.2

Unit Tests

................................
................................
................................
..............................

23

5.2.1

Memory Management

................................
................................
................................
..

23

5.2.2

File Descriptor Table

................................
................................
................................
.....

24

5.2.3

Asynchronous Commands

................................
................................
............................

25

5.2.4

Thread Support

................................
................................
................................
.............

25

5.2.5

Process Output

................................
................................
................................
.............

28

5.2.6

Process Input

................................
................................
................................
................

28

6

User F
eedback

................................
................................
................................
..............................

29

6.1

InspectOS Developers’ Feedback
................................
................................
..........................

29

6.2

Feedback from the OS
-
Tool Developers and our Supervisors

................................
..............

29

6.3

Feedback from
Students

................................
................................
................................
.......

30

7

Conclusions and Future Plans

................................
................................
................................
.......

31

A. Compiling, Installing and Running InspectOS

................................
................................
...................

32

System Requirements

................................
................................
................................
.......................

32

Installation

................................
................................
................................
................................
........

32

Cloning the repository

................................
................................
................................
..................

32

Installing auxiliary packages

................................
................................
................................
.........

32

Running InspectOS

................................
................................
................................
............................

32

Tracing a process

................................
................................
................................
..............................

33

B. InspectOS Repository Description

................................
................................
................................
....

34

References

................................
................................
................................
................................
............

35




A
cronyms


OS


O
perating System

COW


Copy
-
On
-
Write

VMA


Virtual Memory Area

FDT


File Descriptor Table

GDB



GNU Debugger

GUI


Graphical User Interface

PID


Process Identifier

TID


Thread Identifier
Alexandru Șutîi


1

I
ntroduction


An Operating System is a set of software programs and data that runs on a computer, manages the
computer ha
rdware and provides

common

services for the application software

[
1
]
. The main goal
of an operating system is the hardw
are abstraction for the running processes. In order to do this an
operating system
has a
series of management mechanisms for every hardware resource

type
.


The main features of an operating system are:



Process management



Memory management



File system management



Networking



I/O



Security

In this paper I am presenting the InspectOS, a tool that p
rovides dynamic information about some of
the most im
portant above mentioned mechanisms
: process management, memory management,
file system management.


1.1

Inspect
OS

InspectOS is a continuation of the project called OS
-
Tool developed by our colleagues Lucian
Cojocar, Vlad Dogaru and Andrei Buhaiu in 2010. They have
designed

the basic structure and worked
hard on the information gathering, processing and on presentation. They have done a great work,
but there were needed a series of improvements. Therefore we
,
Alexandru Șutîi and Radu Velea,

continued the project development, calling it InspectOS this year.

As it is described later in the

Architecture and Features

and
I
mplementation

chapters we have added a series of features and
performed some refactoring and redesign in the user space modules.

InspectOS


Extending Features for Operating System Inspection


2

Operating System Theory


In order to better understand the way InspectOS

works and the concepts it is describing you have to
be presented some basic op
erating systems theory. I will present the concepts, their definition and a
short explanation.



Process

-

an
instance

of a
computer program

that is being executed. It contains the program
code and its current activity. Depending on the
operating system

(OS), a process may be
made up of multiple
threads of exe
cution

that execute instructions
concurrently

[
3
]
.

A
process h
as an independent address space, a file descriptor table, a process id, etc.



Thread
-

In
computer science
, a thread of execution is the smallest unit of processing that
can

be
scheduled

by an
operating system
. It generally results from a
fork

of a
computer
program

into two or more
concurrently

running
tasks
. The implementation of threads and
processes

differs from one
operating system

to another, but in most cases, a thread is
contained

inside a process. Multiple threads can exist within the same process and share
resources such as
memory
, while different
processes

do not share these resources.

[
4
]



Differences between a process and a thread

-

Threads differ from traditional
multitasking

operating system processes

in that

[
4
]
:

o

P
rocesses are typically independent, while threads exist as subsets of a process

o

P
rocesses carry considerable
stat
e

information, whereas
multiple threads within a
p
rocess share state as well as
memory

and other
resources

o

P
rocesses have separate
address spaces
, whereas

threads shar
e their address space

o

P
rocesses
interact only through system
-
provided
inter
-
process communication

mechanisms.

o

Context switching

between threads
in the same process is typically faster than
context switching between processes.



Virtual memor
y

-

virtual memory is a
memory management

technique developed for
multitasking

kernels
. This technique
virtualizes

a
computer architect
ure
's various
hardware memory devices

(such as
RAM

modules and
disk storage

drives), allowing
a
program

to be designed as though

[
5
]
:

o

T
here is only one hardware memory device and this "virtual" device acts like a RAM
module
.

o

T
he program has, by default, sole access to this virtual RAM module as the basis for
a contiguous working
memory (an
address space
).



Demand paging

-

demand paging (as opposed to
anticipatory

paging) is an application of
virtual memory
. In a system that uses demand p
aging, the operating system copies a disk
page

into physical memory only if an attempt is made to access it (i.e., if a
page fault

occurs). It follows that a
process

begins execution with none of its pages in physical
memory, and many page faults will occur until most of a process'
s
working set

of pages is
located in physical memory

[
8
]
.



Cop
y
-
On
-
Write

-

Copy
-
on
-
write (somet
imes referred to as "COW") is an
optimization

strategy used in
computer programming
. The fundamental idea is that if multiple callers ask
for resources which are initially indistinguishable, they can all be given pointers to the same
resource. This function can be maintained until a caller tries to mo
dify its "copy" of the
Alexandru Șutîi


resource, at which point a true private copy is created to prevent the changes becoming
visible to everyone else. All of this happens
transparently

to the callers. The primary
advantage is that if a caller never makes any modifications, no private copy need ever be
created.

[
9
]



File Descriptor

-

In
computer programming, a file descriptor is an abstract indicator for
accessing a file.

Generally, a file descriptor is an
index

for an entry in a
kernel
-
resid
ent data
structure containing the details of all open files. In POSIX this data structure is called a
fil
e
descriptor table
, and each process has its own file descriptor table. The user application
passes the abstract key to the kernel through a
system call
, and the kernel will access t
he
file on behalf of the application, based on the key. The application itself cannot read or write
the file descriptor table directly

[
6
]
.



Virtual Memory Area

-

Th
e virtual memory area (VMA) is the kernel data structure used to
manage distinct regions of a process's address space. A VMA represents a homogeneous
region in the virtual memory of a process: a contiguous range of virtual addresses that have
the same perm
ission flags and are backed up by the same object (a file, say, or swap space).
It corresponds loosely to the concept of a "segment," although it is better described as "a
memory object with its own properties."

[
7
].



InspectOS


Extending Features for Operating System Inspection


3

Architecture and Features


3.1


A
rchitecture

InspectOS was designed with a high degree of modularity as can be seen in
Figure
1
. This approach is
better suited for parallel development and for efficient sustainment.

The four main modules are as follows:



The
Gathering M
odule


This is the module that is responsible wi
th gathering the
information about the traced process directly from the Operating System. It collects data
about the VMA structures, the File Descriptor Table and the Virtual and Physical Memory
pages. The data is presented to the user space via a
procfs

i
nterface.



The Information Processing Module


As the GUI and Process Tracing modules, it is written
entirely in Python. The main purpose of this Processing Module is to take the information
from the Gathering Module via the
procfs

interface and transform i
n an easy presentable
way for the GUI module.



The GUI module



Displays the
processed process information in a friendly way. Each
process is presented in its container with several panes displaying specific type of
information. The GUI also intera
cts with
the Process Tracer for both sending commands to
the process and retrieving certain information about the process state.



Process Tracer



This is an interface for manipulating the process run state. It is a wrapper
over GDB and uses
expect

for interaction w
ith GDB. It gives the opportunity to step the
process, provide process inpu
t and

retrieve line information, process output and variable
values.

Alexandru Șutîi



Figure
1

-

InspectOS Architecture D
iagram


3.2

OS
-
Tool F
eatures

As mentioned above
OS
-
Tool i
s the application InspectOS bega
n from. The architecture was
designed in OS
-
Tool and InspectOS came to improve the application with new features. The features
that OS
-
Tool already had are the following:



Virtual and Physical Pages



The application

display
s

the table of virtual and physical pages
and show the correspondence between a virtual page and the physical frame it is mapped
on. Also it notifies whenever a
Copy
-
On
-
Write
or a
Demand Paging

event occurs.



File Descriptor Table



For the traced p
rocess you have the opp
ortunity to visualize the open

file descriptors

and the type of the open

files.



VMA Tree



The red
-
black
Virtual Memory Area Tree

can be found in the VMA pane.



Source code visualization



Whenever a process is traced
,

its source cod
e is displayed
. B
y
pressing the
Next

button you can

advance the process an instruction at a time.




InspectOS


Extending Features for Operating System Inspection


3.3

I
nspectOS New Features

InspectOS main purpose was to
improve the graphical user interface in order to make it more
intuitive for

beg
i
n
ners

and extend the O
S
-
Tool by developing new features.

Below you can see a list of new features we have developed:



GUI R
edesign



In OS
-
Tool the GUI was divided in components displaying a specific resource.
For example, whenever a new process was created a new widget was add
ed for each the
Source Code
, Memory, File T
able
and VMA containers, rather than adding a container
encapsulating all above mentioned widgets. Therefore we redesign GUI so a better per
process encapsulation could be achieved.



Support for blocking instructio
ns



If the traced process happened to block at a specific
instruction (e.g. an I/O instruction) the application used to freeze until the instruction was
finished. To avoid this situation we had to redesign the Process Tracer
to make it use

asynchronous GDB commands for m
anipulating the traced process.



“Back”

F
eature



This feature is meant to memorize the states the process passed through
and give the opportunity to restore a previous state of the traced process.



Process Input



The GUI dis
plays a container for each traced process that allows introducing
text that would be sent to process
on

its standard input.



Process Output


Also the GUI displays a container that shows the output of the traced
process.



Process Tree


This feature helps th
e user to visualize in a tree form the processes created
by the traced process. The user can select any of the created processes and see details
about its state.



Command line arguments



InspectOS is able to run the program with the given command
line argu
ments. The user is displayed a text field that allows him/her to introduce them.



Action Logging



At

the button of the main window there appears a text area that displays
all user actions and the events.



Alexandru Șutîi


4

I
mplementation


4.1

Underlying

T
echnology

The
InspectOS modules are build using some libraries and tools. I will emphasize on the description
of the technology used by the user space modules, as our work in InspectOS was mainly oriented to
improvement of this parts.

4.1.1

GTK

All the Graphical User Interfa
ce was developed using a Python bi
n
ding of GTK library called
pyGTK
.
This is

a

ubiquitous library on Linux. Although it contains the basic needed widgets, it lacks more
so
phisticated widgets for drawing
pages, graphs, trees
, etc
. Therefore some other graph
ical libraries
had to be added for displaying this kind of structures.

4.1.2

Graphviz and Cairo

For drawing the physical and virtual memory tables, Cairo was chosen. It has the necessary methods
for drawing this kind of structures. Graphviz is the tool used for
drawing graphs. The input graph is
given in its specific description language and
it provides an output image displaying

the graph. That
image is taken by our GUI and displayed in a frame.

4.1.3

GDB

In order to manipulate the traced process we implemented our tr
acer over the known GNU
debugger GDB.


For every traced process we create a GDB instance

[
10
]
. If the process forks new processes they
remain in the same GDB instance. The Tracer module sends the GDB commands via the Expect
interface. Expect is implemented in TCL programming language for inter
acting with running
processes (in this case GDB). The tracer gets the GDB response via Expect, too, and takes the
appropriate action.


InspectOS


Extending Features for Operating System Inspection


4.2

Graphical User Interface R
edesign

The OS
-
Tool devel
opers chose a bottom up development mode, i.e. they first of all thoug
ht how to
gather and organize the information about the traced process from the OS, and only after that they
came with a way to display this information in the user interface. In consequence, the GUI was not
as intuitive for the beginners as it should be f
or an educational application. When we began working
on InspectOS our first goal was to redesign the user interface so that it was easier to understand and
encapsulate all the process data in the same graphical container.

In
Figure
2

you can see the

diagram of the OS
-
Tool GUI and

Figure
3

displays a screenshot of the
graphical user

interface. As you can

see,

the main window contained five main containers: the
source container, the VMA container, the memory container, the file descriptor table container and
the “Next” buttons container.


Figure
2

OS
-
Tool GUI

D
esign

Alexandru Șutîi


Whenever a new process was spawned by the fork() system call, new frames were added to Source
and Memory containers and a new “Next” button was created. Also the VMA and File Table tabs
showed only the last process’s table.


Figure
3

OS
-
Tool Graphical User Interface

We did not find this interface well organized and we decided to have a better per process
organization. Therefore we reorganized the GUI as seen in the UML diagram from

Figure
5

and in
the screenshot from
Figure
4
.


Figure
4

-

InspectOS S
creenshot

InspectOS


Extending Features for Operating System Inspection


The window contains two main frames. The frame in the right displays the physical memory. It
displays the physical memory frames for all the traced processes. In the right frame there is a pane
that allows the visualization

of maximum two processes at the same time (you can select the
processes to view). All of the process resources are encapsulated in the same container.

The whole window is created in the Main class. When first opening the application, it displays a form
th
a
t prompts for the program to trace
. For

the

each new
ly

created process the Main object creates a
ProcessView object. This object encapsulates all the process information and returns a frame to be
displayed by Main. The Main object creates a VMA, a COW and

a FileTableTab object and passes
these objects to all the new created ProcessView Objects. Also al
l the child processes created from a
traced process share the same Tracer object

as they all of them reside in the same GDB session
.
Whenever the ProcessView

object has to display its frame, it gets its virtual memory table frame
from the COW object, its file descriptor table from the FileTableTab object and the VMA frame from
the VMA objects, by passing them its process id. Also it adds to the process frame t
he source, the
standard input and output frames.


Figure
5

-

InspectOS New GUI D
esign

Alexandru Șutîi


4.3

GDB Asynchronous C
ommands

At

the beginning the process tracer used only synchronous commands for interactions with GDB. It
worked well as long as there was no blocking instruction in the traced process, such as I/O
operations, synchronization object operations (semaphores, mutexes, w
aiting queues). When
the
user was
pressing the “
Next
” button, the tracer waited for the GDB response. If the process

had

blocked in an instruction, the GUI thread also blocked, therefore the whole application was getting
frozen.

We had a lot of time search
ing the web

for

a solution to this problem. However, we did not find a
solution. So we wrote an email to the GDB users list to ask for help. This way we found out about
the GDB asynchronous commands.

So we have redesigned the tracer to send asynchronous co
mmands

and interro
gate the

status in
order to see if the
current instruction ended. The detailed flow can be seen in the
Figure
6
. When
stepping the process the “
step&
” command is sent to

GDB. The GDB prompt immediately returns as
this is an asynchronous command. In order to check the actual state of the process
,

the tracer sends
the “
where
” command. If the response contains the string “
Target is executing
” it means the

process is still blocked in the stepped instruction. To be sure that
a non blocking instruction is not

perceived as blocking, the tracer waits 0.5 secon
ds before requesting the status. I
n this time the non
blocking instruction will end for sure.


Figure
6

-

GDB asynchronous command flow

When the tracer
announces

that process is blocked, the GUI changes the “
Next
” button’s caption to

Blocked
” for a small amount of time. The next time the user presses “
Next
”, the tracer will check
again whether process is still blocked or not.

InspectOS


Extending Features for Operating System Inspection


In order to
prepare GDB for asynchronous commands we have to input the following c
ommand

when starting the GDB instance:

> gdb ./program

[…]

(gdb) set target
-
async on


4.3.1

Another Possible I
mplementation

For the

first time

we

thought to create a separate thread whenever a process got blocked. This
thread would have had to send
“where”

or “
info program”

commands in order to check whether the
process unblocked
,

and

after that

update the “
Next”
button’s state
. We ha
ve renounced this idea as
adding multithreading to the application would have added a lot of new problems regarding thread
synchronization.

Furthermore, this would have forced us to implement

a safe access t
o the GDB
session. This would have been

a bad ide
a, as we found a lot of bugs in GDB and this feature would
have revealed a lot more. Also it would have incre
ased the used processing power.

4.3.2

A Known I
ssue

We have found a bug related to GDB asynchronous commands that we could not solve. The

scenario
is as follows. Say we have a process A that blocks in reading from a pipe. The process B has to write
something in that pipe, causing the process A to unblock. However if we step the process B a few
times, but not enough that it would get to the
instruction that will write in the pipe, and we step
again the process A (that is being blocked in the read instruction), the GDB sends a SIGKILL signal to
both the processes.

In order to solve this issue, we tried to get the process’s blocking state befor
e send
ing

a “
step&
” to it
again. But this approach was unsuccessful, as GDB takes all the debugged processes in
to the

Stopped
” state (stopped at a breakpoint), rather than leaving the blocked processes in “
Sleeping”

state. Therefore, we could not find out

the status of a process without resending the “
step&”
command to it.

The same problem occurs with multiple threads in the same situation.

4.4

Thread S
upport

An interesting and a challenging feature we came with, is the thread support. We first of all thought
that it would be useful for the users to visualize on real examples, problems like producer
-
consumer.

Threads in a process share the same address memory, the same VMA and the same file descriptor
table. The only private properties and resources of a threa
d are: t
he stack, the registers,

current line
number (this is actually also stored in a register), the thread private data.

Therefore we didn’t need

to change the memory, VMA and file table views as they are the same for all the threads

in a
process
.

Alexandru Șutîi



Figure
7

-

Thread Diagram

We chose to display the line number for each thread and the variable values. Also InspectOS is able
to step each thread.

For every time a new thread is created, GDB announces this and the ProcessView objec
t adds a new
entry for the newly created thread. The entry contains a thread id, a line number and a next button.

You can find a diagram in the
Error! Reference source not found.

and a screenshot of how the GUI
looks like. The thread TID is displayed in the left of its current line. Whenever the “
Next”
button is
pressed the
tracer gets the TID of the thread, switches the thread number in GDB and sends the

step&”

command.

InspectOS


Extending Features for Operating System Inspection



Figure

8

-

Thread Screenshot

As I have mentioned in the previous section, the thread support meets the same issue as for the
situation when there are more processes open. You have to step a thread until it unblocks another
before steppin
g the blocked thread again. Otherwise, GDB will kill all the threads.

We focused first of all on making

the thread support feature work for the producer
-
consumer
problem. Therefore we have not tested it much for processes that both create processes and
thr
eads.

4.4.1

Problems encountered

Whenever the process being debugged creates a new thread or process the GDB notifies by saying

New process/thread created
”. So to step a specific thread/process you have to switch to that
thread by entering the command “
thread t
hread_no”
. When you step a process, all the other
processes in the GDB session stay stopped.

However, GDB was not performing the same with the threads. Whenever you step a thread, the
other threads of that process begin running as well. So we had to figure

out how to make them stay
stopped. In order to solve this problem we have to enter the following commands:

> gdb ./program

[…]

(gdb) run


(gdb) set scheduler
-
locking on

[…]


Alexandru Șutîi


4.5

Process

S
tandard

I
nput

In its initial stage InspectOS

was not designed to interact in any way with the traced process.
Therefore, we considered it necessary to provide a means to interact with the process’s input and
output and to pass the process with initial command line arguments.

In order to send input t
o the process, we have tuned the GUI and the Tracer modules.

You can see a
diagram in the
Error! Reference source not found.
.

In the GUI we added a frame that
provides a text
box where the use
r can input the text and submit it.


Figure
9

-

Input Flow

In order to make the process receive our input we had to redirect its standard input from a named
pipe. When the process is to be traced, InspectOS creates a named pipe. A
fter this
,

it creates a GDB
session for the given process with the following commands:

> touch named_pipe

> gdb ./process

[…]

(gdb) run < named_pipe


When the user inserts some text in the input frame, the GUI sends the text to the Tracer by calling
its
send_text()

method with the text as an argument. The Tracer opens the write edge of the named
pipe and writes the text to it.

4.5.1

Known I
ssues

This approach is the only one we have thought of. Although, it works well for most of the cases, it
has some drawback
s
.

If a process creates more processes with
fork()

system calls, all the spawned processes have the
same named pipe redirected to the standard input. Therefore, the user has to be careful that the
given text goes to the intended process. Moreover, if a pro
cess performs a read from the input, it
flushes all data from the pipe, even if it does not need the whole of it.

InspectOS


Extending Features for Operating System Inspection


4.6

Process Standard O
utput

4.6.1

The First Failed A
pproach



Redirect the Process O
utput

We have had a lot of headaches developing this feature. First
time we thought to implement it the
easy
,

and we though the normal
,

way: by redirecting the output to a named pipe, and reading it
from the pipe, as we did with the standard input.

We realized that GDB has several ways of flushing the output for de debugging process. If you debug
the program without redirecting the output (its output is displayed in
the
terminal)
, GDB flushes
every line to the terminal every time the process outputs
a new line. However, when redirecting the
output to a file, GDB realizes that this is not a terminal and flushes blocks of data, i.e. it buffers many
lines, and flushes them when the buffer is full. As a result we didn’t get the output from the pipe
immedi
ately after it

was written. So we renounced

this solution.

4.6.2

The Second Failed Approach


Parse the GDB O
utput

The second approach was not very reliably, however it worked for most of the situations. This time
the Tracer was parsing the GDB output. We implem
ented a way to differentiate between the output
of the GDB commands and the output of the traced process. In most of the cases the GDB
commands’ output was following an easy patter so we could easily differentiate it. However things
got worse when we imple
mented the support for blocking instructions, i.e. using the GDB
asynchronous com
mands. The GDB output began to be less organized and we were failing to
differentiate the process output very well. Therefore, we thought to the last and the good solution
pre
sented in the next section.

4.6.3

The Final S
olution

We have found that there is a way to redirect the GDB output to a file, i.e. the output of the GDB
commands, while the output of the process being debugged is displayed in the terminal. This way
we do not have

to parse the GDB output, we now know that we can get the process status (current
line, running status, etc.) from the redirected file, and in the terminal we get the pure output of the
process.

In order to enable this behavior we give the following GDB c
ommands at the beginning of the debug
session:

> gdb ./program

(gdb) set logging redirect on

(gdb) set logging file redirect_file

(gdb) set logging on


Whenever the “
Next”

button is pressed
,

the ProcessView object calls the
step()

method of the
Tracer by passing the process

id. The Tracer sends the “
step&”

commands to the GDB. If the process
has not

got

blocked, the Tracer reads the response of the “
step&”

command from the redirection
file and returns the status of the process to t
he ProcessView (i.e. current line, blocked/not blocked,
Alexandru Șutîi


process created, thread created, process terminated, etc.). Also the tracer reads the output from the
terminal and saves it internally in a list of lines.

After the
step()

method ends
,

the ProcessView

asks
for the output from the Tracer and appends the
new output lines to the Output Frame text field.


Figure
10

-

The Output flow

4.7

Variable Value D
isplay

When we implemented the thread support
,

in a problem like Producer
-
Consumer the user could
only learn about the thread synchronization. However, this was not enough. So we decided to
implement the feature that would display the values of the variables, both global and local.

We just used the GD
B “
print”

command that takes a variable name and returns its value. When the
user wants to find the value of a variable he is displayed a frame that contains a text input field
where he can give the variable name, a button and a read only text field where
the value is
displayed. When the user presses the button, the ProcessView object calls the
get_variable_value()

method of the Trac
er with the variable name. The T
racer sends the
print

command to GDB

and gets
the value that it

returns to the ProcessView ob
ject.

The event flow can be better seen in

Figure
11
.

InspectOS


Extending Features for Operating System Inspection



Figure
11

-

Print variable value command flow


Alexandru Șutîi


5

Test Scenarios


5.1

Compiling and Tracing
a Process

In order to trace a process you have to have the source code and compile it with debugging symbols
.
You can see an example of source code below:

#include

<stdio.h>

#include

<string.h>

#include

<assert.h>


#define

PAGE_SIZE (1024 * 4)


int

main(
void
)

{


char

buf[5 * PAGE_SIZE];



memset(buf,
'x'
, PAGE_SIZE);


memset(buf + PAGE_SIZE,
'x'
, PAGE_SIZE);


memset(buf + 2 * PAGE_SIZE,
'x'
, PAGE_SIZE);


memset(buf + 3 * PAGE_SIZE,
'x'
, PAGE_SIZE);


memset(buf + 4 * PAGE_SIZE,
'x'
, PAGE_SIZE);



return

0;

}

You can compile this example by entering the following commands:

> gcc



P慬a

o⁦慵 琠晡f汴⹣

周攠牥獵rW敤eb楮慲礠Ș慮 b攠獥汥捴
敤e楮 䥮獰V捴Ș匠anT 瑲慣敤⸠坨P汥

瑲慣楮g WU攠p牯Ș敳e
H

you are
T楳i污祥T 瑨攠獯u牣攠捯T攠慮T you 捡Ș 獴Vp 瑨攠p牯Ș敳献eNo瑥 瑨慴W瑨攠獯u牣攠f楬攠U慳a瑯 b攠楮 瑨攠
獡Ve⁦ 汤敲e睩瑨⁴桥Wb楮慲y⁳o⁇M䈠Bou汤⁦楮T⁩琮

5.2

Unit Tests

In order to test InspectOS we have created a folder called
examples

in our repository [
11
]
. For every
implemented feature we have a unit test that resides in this directory. This directory is a good start
for our users
,

in or
der to make them understand where to start from when using InspectOS.

I will
present below the tests used for the basic features.

5.2.1

Memory Management

The following source code is a very good example to show the way the physical memory frames are
allocated
while running the process.


1
#include

<stdio.h>


2
#include

<string.h>


3
#include

<assert.h>


4


5
#define

PAGE_SIZE (1024 * 4)


6


7
int

InspectOS


Extending Features for Operating System Inspection



8
main(
void
)


9
{

10

char

buf[5 * PAGE_SIZE];

11

char

*pbuf;

12

memset(buf,
'x'
, PAGE_SIZE);

13

memset(buf

+ PAGE_SIZE,
'x'
, PAGE_SIZE);

14

memset(buf + 2 * PAGE_SIZE,
'x'
, PAGE_SIZE);

15

memset(buf + 3 * PAGE_SIZE,
'x'
, PAGE_SIZE);

16

memset(buf + 4 * PAGE_SIZE,
'x'
, PAGE_SIZE);

17

pbuf = (((
unsigned

long
)&buf[0]) & ~(PAGE_SIZE
-
1)) + PAGE_SIZE;

18

fork();

19

pbuf[PAGE_SIZE] =
'x'
;

20

pbuf[0] =
'x'
;

21

return

0;

22
}


At line 10 a buffer of five pages is allocated, that means only 5 pages of virtual memory are
allocated. The physical pages are allocated one by one from the line 12 to line 16. For each of these

lines a demand paging event is generated.

At the line 18 a new process is created. Five virtual pages are allocated for the child process and the
physical memory

pages are marked read
-
only and shared between the two processes. At the lines
19 and 20 when
the first process writes the second and the first page, two Copy
-
on
-
Write events
occur and two more physical pages are allocated for the child process.

5.2.2

File Descriptor Table

In order to test our File Descriptor feature

we use the following program:


1
#include

<stdio.h>


2
#include

<fcntl.h>


3
#include

<sys/stat.h>


4
#include

<sys/types.h>


5


6
int


7 main(
void
)


8 {


9

int

fd1, fd2, fd3, fd4, fd5;

10

11

fd1 = open(
"/tmp/a"
, O_RDONLY);

12

fd2 = dup(fd1);

13

close(0);

14

fd3 = open(
"/tmp/b"
, O_RDONLY);
/* /tmp/b is a hard link to /tmp/a */

15

close(1);

16

close(2);

17

fd5 = open(
"/tmp/a"
, O_WRONLY);

18

fd4 = open(
"/tmp/c"
, O_WRONLY);
/* /tmp/c is a soft link to /tmp/b */

19

20

return

0;

21 }

When the program starts
,

the descriptors 0 (standard input), 1 (standard output) and 2 (standard
error) are open. At line 11 the file descriptor 4 is
being
open
ed
. At line 12 a new descriptor (5) points
to the file open
ed

at line 11. Line 13 closes the standard input file descrip
tor. Line 14 opens file
descriptor 0 to point to
/tmp/b
. Lines 15 and 16 close the descriptors 1 and 2 (the standard output
and error). Line 17 opens file descriptor 1 to point to new file structure that point to
/tmp/a
. Line 18
Alexandru Șutîi


opens the file descriptor
2

to write to
/tmp/c
. This means the standard output and error will be
redirected to
/tmp/a
, respectively to
/tmp/c
. Standard input is redirected from
/tmp/a
.

5.2.3

Asynchronous Commands

In order to test that the GDB asynchronous commands work as expected and that

the application
realizes when there is blocking instruction
,

we use the following unit test.

The main process creates
a pipe. At the line 18 it creates a new process. The child process is being stepped till the line 23
where it should block as it wants to

read from the pipe. Its “
Next”

button has to show
B
locked

until
the parent process gets to write in the pipe at the line 33. When the parent gets to the line 35 its
button has to show
Blocked

until the child process exits at the line 26.


1
#include

<unis
td.h>


2
#include

<stdio.h>


3
#include

<stdlib.h>


4
#include

<string.h>


5
#include

<sys/wait.h>


6


7
int

main()


8 {


9
int

pipefd[2];

10
int

pid, status;

11
int

dummy = 0;

12
char

hello[] =
"Hello"
;

13
char

recv[20];

14

15
if

(pipe(p
ipefd))

16 perror(
"create pipe"
);

17

18 pid = fork();

19
if

(pid == 0)

20 {

21 dummy = 2;

22 close(pipefd[1]);

23 read(pipefd[0], recv, 19);

24 printf(
"Got %s
\
n"
, recv);

25 close(pipefd[0]);

26
exit(0);

27 }

28
else

if

(pid < 0)

29 perror(
"fork"
);

30

31 dummy = 3;

32 close(pipefd[0]);

33 write(pipefd[1], hello, strlen(hello)+1);

34 close(pipefd[1]);

35 waitpid(pid, &status, 0);

36

37
return

0;

38 }


5.2.4

Thread Support

As I have explained earlier
,

the main purpose of implementing the thread support was to illustrate
the Producer
-
Consumer problem using threads. Therefore
,

we created a test example that creates
two threads, a producer and a consumer, that share a global bu
ffer
of five elements. The producer
InspectOS


Extending Features for Operating System Inspection


inserts randomly generated numbers and the consumer consumes them.

This example is good for
testing the asynchronous commands and for displaying the variable values. In this problem the user
is interested to see when the

threads stop at the synchronization points and the values from the
buffer.


1
#include

<pthread.h>


2
#include

<unistd.h>


3
#include

<stdio.h>


4
#include

<stdlib.h>


5
#include

<string.h>


6


7
#define

BUFFER_SIZE 5


8
#define

NR_ITERATIONS 30


9


10
typedef

struct

{


11

int

buff[BUFFER_SIZE];


12

int

first;


13

int

count;


14
} buffer_t;


15


16
void

init_buffer(buffer_t* b) {


17

memset(b
-
>buff, 0,
sizeof
(
int
) * BUFFER_SIZE);


18

b
-
>count = 0;


19

b
-
>
first = 0;


20
}


21


22
void

insert_item(buffer_t* b,
int

item) {


23

b
-
>buff[(b
-
>first + b
-
>count)%BUFFER_SIZE] = item;


24

b
-
>count++;


25
}


26


27
int

remove_item(buffer_t* b) {


28

int

item = b
-
>buff[b
-
>first];


29

b
-
>first = (b
-
>
first + 1)%BUFFER_SIZE;


30

b
-
>count
--
;


31

return

item;


32
}


33


34
int

is_buffer_full(buffer_t* b) {


35

return

b
-
>count == BUFFER_SIZE;


36
}


37


38
int

is_buffer_empty(buffer_t* b) {


39

return

b
-
>count == 0;


40
}


41


42
/*


43

* the

buffer where the producer will place items


44


* and from which the consumer will take items


45

*/


46
buffer_t common_area;


47
pthread_cond_t buffer_not_full = PTHREAD_COND_INITIALIZER;


48
pthread_cond_t buffer_not_empty

= PTHREAD_COND_INITIALIZER;


49
pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;


50


51
void
* producer_fn (
void
* arg) {


52

int

item_to_insert, i;


53


54

for
(i = 0; i < NR_ITERATIONS; i++) {

Alexandru Șutîi



55

item_to_insert = rand() % 100;


56

p
thread_mutex_lock(&mutex);


57

// Wait the buffer not to be full


58

if

(is_buffer_full(&common_area))


59

pthread_cond_wait(&buffer_not_full, &mutex);


60

// Insert the item


61

insert_item(&common_area, item_to_inse
rt);


62

// Signal buffer not empty


63

if

(common_area.count == 1)


64

pthread_cond_signal(&buffer_not_empty);


65

pthread_mutex_unlock(&mutex);


66

}


67

return

NULL;


68
}


69


70
void
* consumer_fn (
void
* arg) {


71

int

item_consumed, i;


72

for
(i = 0; i < NR_ITERATIONS; i++) {


73

pthread_mutex_lock(&mutex);


74

// Wait the buffer not to be empty


75

if

(is_buffer_empty(&common_area))


76

pthread_cond_wait(&buffer_not_empty, &mu
tex);


77

// Consume the item


78

item_consumed = remove_item(&common_area);


79

// Signal buffer not full


80

if

(common_area.count == BUFFER_SIZE
-
1)


81

pthread_cond_signal(&buffer_not_full);


82

pthread_mute
x_unlock(&mutex);


83

}


84


85

return

NULL;


86
}


87


88
int

main ()


89
{


90

pthread_t producer_th, consumer_th;


91

int

rc;


92


93


/* initialization */


94


init_buffer(&common_area);


95


srand(time(NULL));


96


97


/* create the
threads */


98



rc = pthread_create(&producer_th, NULL, producer_fn, NULL);

100
if

(rc != 0)

101

perror(
"pthread_create"
);

102

rc = pthread_create(&consumer_th, NULL, consumer_fn, NULL);

103

if

(rc != 0)

104

perror(
"pthread_create"
);

105

106

/* wait for the threads to finish execution */

107

rc = pthread_join(producer_th, NULL);

108

if

(rc != 0)

109

perror(
"pthread_join"
);

110

rc = pthread_join(consumer_th, NULL);

111

if

(rc != 0)

112

perror(
"pthread_join"
);

113

114

return

0;

115
}


InspectOS


Extending Features for Operating System Inspection


5.2.5

Process Output

The unit test for the output feature is not a hard one. The traced process creates a child process.
Both of them print text to the output. The GUI has to know how to separate the output of each
process and show it in its output frame. Also the processes ha
ve to synchronize.


1
#include

<stdio.h>


2
#include

<sys/wait.h>


3


4
int

main()


5 {


6
int

pid;


7 printf(
"Started the process
\
n"
);


8


9 pid = fork();

10
if

(pid == 0)

11 {

12 printf(
"The child process is created
\
n"
);

13
exit(0);

14 }

15
else

if

(pid < 0)

16 perror(
"fork"
);

17

18 printf(
"Child process created
\
n"
);

19 waitpid(pid, &status, 0);

20 printf(
"Child process exited
\
n"
);

21

22
return

0;

23 }

5.2.6

Process Input

In order to test the process input feature we use a test program that creates a child process and
both of
them read integers from the input. This example shows the problem I have iterated in the
implementation section. If you enter more integers at a time
,

one process reads them all even if
it
does not need all of them (it flushes the input pipe).


1
#include

<stdio.h>


2
#include

<stdlib.h>


3
#include

<unistd.h>


4


5
int

main() {


6
int

read_value;


7
int

nr_steps;


8
int

i;


9

10 printf(
"Give the number of steps
\
n"
);

11 scanf(
"%d"
, &nr_steps);

12

13

if

(fork() < 0)

14


perror(
"fork"
);

15
for

(i = 0; i < nr_steps; i++) {

16 printf(
"Give a value to read.
\
n"
);

17 scanf(
"%d"
, &read_value);

18 printf(
"Read %d.
\
n"
, re
ad_value);

19 }

20

printf(
"Exiting...
\
n"
);

21

return

0;

22

}

Alexandru Șutîi


6

U
ser Feedback


6.1

InspectOS Developers’ F
eedback

Before we started the development, we decided to put ourselves in the position of a project’s user,
i.e. the student that has pour
operating system concepts knowledge. Without looking over the
source code we had to compile the project, run it and come with some objective impressions. This
was a productive thing to do, as we came with a list of bugs and issues that were to be discussed

[2].

First of all we did not like the graphical user interface organization. It took some time to figure out
what every widget mean. So we decided to redesign the GUI as the first thing.

We also looked again over the operating systems labs and tried to id
entify new concepts that could
be
visualize
d

in InspectOS. This way we came with the ideas to implement the thread support, as it
is one of the hardest concepts to comprehend by the students. Also we decided to implement the
visualization of the process st
andard output and error and the created process tree. The idea to
implement a way to send text to
the standard in
put came later.

When tracing processes in OS
-
Tool we often

needed to see how the memory or the

file table looke
d
at a previous moment. This le
d

us to the idea to implement the “Back” option that would create
snapshots of the process resources and be able to display them on demand.

6.2

Feedback from the OS
-
Tool Developers and our
S
upervisors

After finishing the proposed ideas list, we had a meeting wi
th Andrei, Lucian and Vlad, the OS
-
Tool
developers, and presented them the list and the way we wanted to approach the implementation of
that features.

Although we had read their bachelor’s thesis and learn
ed

about the problems they have confronted
with whi
le developing OS
-
Tool, this meeting helped us a lot. Lucian brought to our knowledge that
new Linux kernels provide the same information about a process without the need of the OS
-
Tool
kernel module. So we had been given the idea to renounce the kernel mod
ule. We did not do this
,

as there were features more important to implement and to improve, such as the GUI redesign.
However, this could be accomplished in future development. Also Andrei and Vlad shared some
thoughts of how to redesign the GUI and what t
ools exactly to use.

A very objective feedback came from our supervisor Razvan Deaconescu, whom we presented our
work weekly. He came with
a lot of

great ideas regarding the thread support, GUI redesign, “Back”
feature.


InspectOS


Extending Features for Operating System Inspection


6.3

Feedback from

S
tudents

As InspectOS

is an educational application, the most important feedback is the one from its future
users, the students. We have sent a presentation email to a list of students and asked them to try
InspectOS and respond with a valuable feedback.

The feedback was a rea
lly good one. We have received some improvement idea, and bug reports,
that we have solved. There were some small issues related to the GUI, but in great the users had a
good opinion about the application and found it helpful in understanding the operating

systems
concepts. From this feedback we like to think that InspectOS is a stable and useful application at the
moment.



Alexandru Șutîi


7

Conclusions and Future Plans


From the beginning we planned to realize some new features and improvements. We are confident
that we

ha
ve

realized the most part of what we intended. The GUI redesign and the refactoring of
the GUI class hierarchy are one of

our greatest realizations

i
n this project. I believe that we

have
created a better environment for our other f
eatures development and
it will open

new
opportunities for who is going to continue this project in the future.

The thread support feature seemed to me hard to realize, but after a lot of research and many tries I
feel confident that we have done a great job. The redesigning of t
he tracer to support asynchronous
GDB commands was also a great challenge that needed some research as well. Although the
implementation has some known issues, it works well for most of the cases.

Another feature we planned and realized is the “Back” featu
re. It suffered a lot of modifications until
it got to the current state, but the final result is a great one. I congratulate Radu for accomplishing
this goal of InspectOS.

Although we believe we accomplished most of our goals, there have remained some ide
as that we
didn’t implement. Some of them are:



Create a full documentation for the GUI and add a Help menu entry.



Create a configuration file.



Create an installer (although we could implement this in the near future).



Although the file table shows the file

types descriptors point to, we wanted to add more
details to this.



Improve the process stepping. This feature should allow placing break points and adding
more buttons (besides the
Next

button) that a normal debugger has.

These ideas can be found in our p
roject’s issue list, too [
2
].
They

can serve as a good starting point
for future development. Also the OS
-
To
ol developers’ ideas that we have
n’t accomplished can be
implemented by
the future developers.

I feel very happy to have worked with my colleague Radu Velea. I want to thank
him and our
supervisors for the interesting collaboration, as it was

a

successful one. Although we have
encountered several obstacles during our development, we have managed to surpass most of them.

I have learned a lot of new things and improved my knowledge and skills.


InspectOS


Extending Features for Operating System Inspection


A
.
Compiling, Installing and Running
InspectOS


System Requirements

In order to run InspectOS you need a Linux operating system with a 2.6 kernel.
The kernel must
support Kprobes [
15
]. You also need python 2.6 to run the appli
cation.

Installation

Cloning the repository

To clone the InspectOS repository you have to have Git [
16
] installed on your computer. You can
create a read
-
only clone by en
tering the following command in the shell:

> git
clone
http://ixlabs.cs.pub.ro/git/inspectos.git

Installing auxiliary packages

There are

a couple
of third
-
party libraries and tools that have to be
installed for InspectOS to run.

o

Python
-
rsvg

[
17
]

> sudo apt
-
get install python
-
rsvg

o

RBTree module

[
18
]

> wget
http://newcenturycomputers.net/projects/download.cgi/RBTree
-
1.6.zip

> unzip RBTree
-
1.6.zip

> cd RBTree
-
1.6

> sudo python RBTree.py install

o

Graphviz

[
19
]

>
sudo apt
-
get install Graphviz

o

GDB

[
20
]

> sudo apt
-
get install gdb


Running InspectOS

First of all you need to compile the kernel module and insert it. For this you need to enter the
following commands:

>
cd $INSPECTOS_ROOT/fault

> make

> sudo insmod os
-
mon.ko

Alexandru Șutîi


Now you are ready to run the application. You simply need to go to InspectOS root directory and run
the script
inspectOS
.sh
.

> cd $INSPECTOS_ROOT

> sudo bash inspectOS.sh



Tracing a process

When InspectOS

starts you are displayed a form that lets you choose the program you want to
trace
. You have to choose a binary that was compiled with debugging symbols and resides in the
same directory with its source code. You also can trace more processes by accessing

Menu
-
>File
-
>Open.

When you are running InspectOS you are displayed a container for each process. Every process

is
identified by its process id
. You can find it in the
Next

button, in the container name, etc. You can
select what process to visualize by checking its checkbox in the up left corner of the window. You
can visualize maximum two processes at the same time.

In the right of the main window you can see the physical m
emory. Each process container contains
Next

buttons for each thread, a source code frame, an input and an output frame, a

virtual

memory
frame, a VMA frame and a File Table frame. You can choose the frame to view by selecting the
appropriate

radio button.
There is also a
Print
button for printing the variables’ value.





InspectOS


Extending Features for Operating System Inspection


B. InspectOS Repository Description


I will briefly describe some of the directories that form our repository.



fault/

-

This is the directory containing the source code for the kernel mod
ule. This is where
to compile the module and insert it.



examples/

-

All our unit tests can be found here. This is a good starting poin
t for the users
that
use our application

for the first time
.



ixia
-
slides/

-

This directory contains the presentation slide
s, logos, images.



styles/

-

In order to give a custom color (yellow) to the current line in the source code frame
I had to extend a style. This style resides in this directory.



src/

-

This is the source directory for the user space modules. The main direct
ories and files
are:

o

src/mai
n.py



This file defines the
Main

class, the one that creates the window and
starts the application.

o

src/ostool
and
src/os_tool/event



The processing module files reside in these
directories. Here are the classes that read the VMA, memory tables and file table
information from
procfs

and present it to the GUI.

o

s
rc/os
_
to
ol/gui



The GUI classes can be found in this directory
.

The file
p
rocess_view.py
defines the
ProcessView
class that enca
psulates all the information
about

a process. It gets the VMA, memory and file descriptor frames from the
classes
COW, VMA
and
FileTableTab

that are defined in the files
cow.py, vma.py
and
file_table_ta
b.py

respectively.

o

src/os_tool/tracer
.py

-

This file defines the Tracer module.






Alexandru Șutîi


References


[1] Wikipedia. Operating system
-

Wikipedia, the free encyclopedia.
http://en.wikipedia.org/wiki/Operating_system

, 2011. [Online; accessed 1
-
July
-
2011].

[2] InspectOS issue list
-

https://ixlabs.cs.pub.ro/redmine/projects/inspectos/issues
.

[3] Wik
ipedia. Process (Computing)
-

Wikipedia, the free encyclopedia.
http://en.wikipedia.org/wiki/Process_%28computing%29
, 2011. [Online; accessed 1
-
July
-
2011].

[4] Wikipedia. Thread (Computer

Science)
-

Wikipedia, the free encyclopedia.
http://en.wikipedia.org/wiki/Thread_%28computer_science%29
, 2011. [Online; accessed 1
-
July
-
2011].

[5
] Wikipedia. Virtual Memory
-

Wikip
edia, the free encyclopedia.

http://en.wikipedia.org/wiki/Virtual_memory

, 2011. [Online; accessed 1
-
July
-
2011].

[6] Wikipedia. File descriptor
-

Wikipedia, the free encyclopedia.
http://en.wikipedia.org/wiki/File_descriptor

, 2011. [Online; accessed 1
-
July
-
2011].

[7] Memory Management in

Linux,
http://www.makelinux.net/ldd3/chp
-
15
-
sect
-
1

, 2011. [Online;
accessed 1
-
July
-
2011].

[8 ] Wikipedia. Demand Paging
-

Wikipedia, the free encyclopedia.
http://en.wikipedia.org/wiki/Demand_paging

, 2
011. [Online; accessed 1
-
July
-
2011].

[9] Wikipedia. Copy
-
on
-
write
-

Wikipedia, the free encyclopedia.
http://en.wikipedia.org/wiki/Copy
-
on
-
write

, 2011. [Online; accessed 1
-
July
-
2011].

[10]
R. Stal
lman, R. Pesch, S. Shebs,
“Debugging with GDB”.
http://www.tutok.sk/fastgl/download/books/DebuggingWithGDB.pdf

[11]
InspectOS Git Repository
http://ixlabs.cs.pub.ro/git/inspectos.git

[12]

A. Silberschatz, P.B. Galvin, G. Gagne, “Operating Systems Concepts


7
th

Edition”.

[13] R. Love, “Linux Kernel Development”.
stid.googlecode.com/files/Linux.Kernel.Development.3rd.Edition.pdf

[14] GDB Mailing L
ist,
http://sourceware.org/ml/gdb/

[15
] Kernel Probes Home Page,
http://sourceware.org/systemtap/kprobes/

[16] Git Home Page,
http://git
-
scm.com/

[17] Python rsvg Home Page,
http://cairographics.org/pyrsvg/

InspectOS


Extending Features for Operating System Inspection


[18] RBTree Home Page,
http://newcenturycomputers.net/projects/rbtree.html

[19] Graphviz Home Page,
http://www.graphviz.org/

[20]
GDB Home Page,
http://www.gnu.org/software/gdb/