AJAX Geography Project Final Report Supervisor ...

looneyvillestaticSoftware and s/w Development

Aug 15, 2012 (5 years and 2 months ago)

302 views




AJAX Geography Project

Final Report




Supervisor:

Amichai Shulman

Students:

Vitaly Timofeev

Eyal Shemesh



Project goal:



Display security logs geographical visualization and various geographical
statistics

about it
.



Constraints:



Save internal
data at the client side as much as possible



Simplify requirements from clients



Simplify update
mechanism



Development
environment
:



Web server


Apache tomcat 6.0



Database Engine


MySQL Server

5.0



Server side programming language


JAVA, JSP



Client side
programming language


JavaScript



IDE


NetBeans 6.1



IP mapping database


MaxMind (
http://www.maxmind.com
)



Geolocation service


Google (
http://maps.google.com/
)



Chart service



Google charts API (
http://code.google.com/apis/chart/
)



Mainly used technologies


HTML, DHTML, XML, JSON and AJAX.

All the server technologies used on both servers which were used in the project



System architecture
:

In the process of development some
system architectures were implemented and
rejected due to the project constraints and difficulties which raised by technologies
restrictions.

The final system architecture consists of 2 servers:



Global server



Local server

On the both servers we have
same d
evelopment environment but Local server does
not require specific environment (due to the project constraint).

Local server

implies two main roles:



Local database



Location of the HTML file, which is run by the user.

Global server

implies two main roles too:



Global database



Location of scripts that client side is using

The way the system works:

User opens a main HTML file on his local server. In this file we have a directive that
downloads a script from the global server that after

the downloading manages all
the process.

All data processing goes without page refresh by using AJAX (and AJAX like
technologies) and DHTML. For example we will describe the service “local group
mapping”
:



the client choose this option on the screen (the o
ption will appear only after
registration of the service on global server


for each service and client pair we
have a data on the server that contains URL of the service and its transfer
technology) the request to the local server will be initiated. Reque
st will be for
the service “getLocalGroups”, the local server must return a list of the local
groups which exists on the server.



When t
he application will have the list it will be displayed to the user for
choosing the group he wants to display.



When use
r
choos
es

the group
,

a new request to the local server is initiated
,
now it will be for service “getLocalGroupData”. The service must return list of
IPs which the given group consists of.



When the list is available to the system, another request will be in
itiated but
this time to the global server to “mapIP” service. This service provides us with
coordinates of the IP (here we use a MaxMind database).



When the coordinates are available they are displayed through Google Maps
API on the screen to the user

The
re are some more things in the process like for example determining if the IP
is a proxy or not, but we will not describe them here.


Now when we have a look on the systems lets describe the requirements from its
parts:

From the local server we require a w
eb server for 2 features:




Answering service requests



like we said, the local server must give us
some services examples of them are: “getLocalGroups”,
“getLocalGroupData” and so on.

These services are registred on the global
server. The data which the g
lobal server need to know about local (or
global) service is:

o

client id


for which client the service is given

o

service name


what service is given

o

service url


from which url we may get the data

o

service transfer type


technology used to deliver the da
ta



technology may be json, xml and plain text



other technologies may be added if needed (for example csv
or MS Excel file format)



mainly we use xml as default transfer type for the local server





Storing main HTML file



this file is the only thing we supply to the client.
The file is very simple in all manners:


<body onload=”start(‘name’)”>


<script src=”http://server/main.js”></script>

</body>


There is some more lines that we need for the page to be a standard HTML
page but these are the main. The script we include is the main system
script and name is client name that we need to load the system
configuration for the given client

The requirements to the local server are that there will be a web server installed. We
d
o not need a special web server, or server side technology here. We need only to
store there the HTML page and get URLs from the server that will give us the data to
the client requests.


From the global

server we require a web server for 2 features

too
:



Answering service requests


like the local server the global server will answer
some request. We have seen some of them before in the example (“mapIP”)
,
there are others of course, mainly there are many parallel services given by
the local and global serv
ers. When the local server gives to the client local
picture, the global server will give approximate values for the picture. So by
comparing this pictures the system administrator will have an opportunity to
locate anomalies.



On the global server we store

the main scripts of the system, system
configuration, and data.

The global server administration is under our control, so here there will be more
requirements to the technology. For example all the services given by the global
server have transfer type “j
son”, because of cross domain security policy.



Constraints fulfilling



Save internal data at the client side as much as possible


o

The system architecture we implemented is using HTML page on the
clients server, so there is no data sent over the internet
where we
request some service from the local server, so all the request to the
local server are safe.


o

There is no possibility to store all the data locally, so global requests are
needed, but
no data is stored on the global server without user
permission.

The request results are not stored on the server, so if the
global server got hacked the user data will not be compromised (the
data stored in the clients networks and protected by the clients like they
want it to be protected).




Simplify requirements fro
m clients


o

There is only 1 requirement of the clients servers. They need to run a
web server, there is no requirement which web server they will use,
which database they use (if they use it at all). The system may run
without local requests at all, so the
user will only see the global
statistics.


o

When user implements
services

we also simplified the process, he may
choose the technology for the data transfer


will he use JSON or XML
or any other implemented data transfer type (for example CSV may be
simply

added to the system)
.



o

When the client implemented a new service it just need to be registered
in the database and the client will have the possibility to use the service.
There is no need to update anything by us on the client side.




Simplify update mec
hanism


o

The updates are simple to the user and to us. User do not need to do
anything. When he open the page the new version of the script will be
just loaded by the browser, and for us we just change the main script on
our web server. All the logic may be

simply changed. The data transfers
are not rely on the scripts so we may change the scripts without
changing the requests from the services that clients are implemented.


o

New services may be added, and when clients will want to add them
they will need onl
y to register the services
. If there is no
implementation of the new service for the client it will just not appear
as option on the clients page, there is no disruption in the system work.


The problems and solutions

For the first look the system
architecture is more complicated than most of the web
services that are implemented today. Let’s see why the standard ways to implement
web applications will not be applicable here and this will lead us to the solutions we
used.

Until la
st years the web te
chnology consisted of 2 main parts: server and client when
the client side was passive and only provide the user interface. With evolution of
JavaScript, the client part became active too, and it came to the peak with AJAX
when the page may get the new dat
a without reloading of the page.

The simplest web architecture doesn’t give us the security, the data is on our server.
The client doesn’t know or may be doesn’t rely on our security. All the requests are
sent over the internet and may be intercepted. All
the data needed to be updated on
our server. If the client wants a new service we need to do code changes and client
need to deliver the data to us. So the architecture doesn’t give us the possibility to
fulfill the needed constraints.

If we use the new te
chnologies like AJAX with standard web architecture it from the
first look give us some solutions, but when trying to implement we will see
difficulties
. Standard architecture assumes only 1 server. If we locate it in the client

s
network the updates will
be more than problematic, we will need to go to each
client and install the updates, the new services will require updates and so on. If we
locate it on our side of the network will still suffer from the security issues that were
described before.



The next step is to use two servers that will give us solution to the security issues,
but there are new problems
-

cross domain policies
. T
here is no possibility to send
AJAX request to the domain other than a page’s.
So we can’t simply use 2 servers


AJ
AX will not work with one of them. And the second part of the new problems is
how we split the system logic to fulfill the constraints? If it will be on the client’s side
there will be problems with updates, and on our side will be problems with security.


So we placed the main page on the client

s side so the client may use AJAX requests
to his server
, all the logic are stored on the server but runs on the client so there is
no security issues, and there are simple updates because when we change the logic
the client will automatically get the new version. The requests to the global service
we do with AJAX like DHTML technology (we define a new dynamic script which is
requests needed data from our server). This technology overrides the cross domain
policy pr
oblem because we may use scripts that are defined on other domains, like
for example we use Google Maps API.




Some words about services we use to give the needed possibilities. We needed
3

services to be imported:

o

IP mapping to coordinates.

o

Mapping coord
inates to the map.

o

Displaying graphs.

Each of these services is a system by itself
,

and implementation of these wasn’t our
goal. So, we needed to find an appropriate way to import them.

o

IP mapping to coordinates


here we choose MaxMind database.

o

It’s fre
e


very important for academic works.

o

It gives
a
ccuracy 99.3% on a country level

and
76%

accuracy

on a city
level



enough for our work.

o

It’s updated regularly


monthly updates.

o

It may be downloaded to our server


we will know where the clients
data is
going and it will not be sent to 3
rd

party companies

o

It may be relatively simply imported into any database


given in CSV
format

o

Mapping coordinates to the map


we choose Google Maps service.

o

It’s free


very important for academic works.

o

There is no ser
ver side logic needed


it’s very important because we
use static HTML page on the client side

o

Customizable

environment


we may add

to the map

our own markers
,

display icons and so on.

o

Chart drawing


we choose Google charts API

o

Free

o

No server side logic

o

There is no much possibilities but enough for us.


In the end all the constraints are fulfilled, and the system gives the possibilities we
wanted to.



The system installation
:

Global server:

After installing all of the underlying technologies (tomcat, mys
ql, JSP), place the
project files to the appropriate places on web server. Files includes all the web files
and some JAVA classes (in our case IDE was integrated with tomcat so all the files
were placed automatically). Install provided database
image to my
sql (with
command line or GUI tools that may be downloaded from official MySQL site).

Local server:

In case of real use need only to put HTML page to the web server and configure the
client name in the global database.

In case of testing there is a need to

repeat the process with restoring DB image and
placing the project files to appropriate place on the server.

Updating MaxMind database
:

There is a monthly update to MaxMind IP locations DB. The install routine is:

1)

Download CSV from
http://www.maxmind.com/
download/geoip/database/GeoLiteCity_CSV/GeoLiteCity_20080801.zip

2)

Create DB projserver

3)

Create table by following code: CREATE TABLE `projserver`.`locations` (


`lid` int(10) unsigned NOT NULL auto_increment,


`country` char(2) NOT NULL,


`region` char(
2) NOT NULL,


`city` varchar(45) NOT NULL,


`pcode` varchar(45) NULL DEFAULT NULL,


`lat` float NOT NULL,


`long` float NOT NULL,


`mcode` varchar(45) NULL DEFAULT NULL,


`acode` varchar(45) NULL DEFAULT NULL,


PRIMARY KEY (`lid`)

) ENGINE=InnoDB

DEFAULT CHARSET=latin1;

4)

Unpack the files into mysql DB directory

5)

Enter the command line client from Querry browser

LOAD DATA INFILE 'loc.csv' INTO TABLE projserver.locations FIELDS TERMINATED BY ',' ENCLOSED BY '"'
ESCAPED BY '
\
\
' LINES TERMINATED BY '
\
n'

IGNORE 2 LINES;

6)

Remove not needed columns from the table

7)

Create table by following code: CREATE TABLE `projserver`.`ipcity` (


`start` int(10) unsigned NOT NULL,


`end` int(10) unsigned NOT NULL,


`lid` int(10) unsigned NOT NULL default '0'

) ENGIN
E=InnoDB DEFAULT CHARSET=latin1;

8)

Enter command line

LOAD DATA INFILE 'ipcity.csv' INTO TABLE projserver.ipcity FIELDS TERMINATED BY ',' ENCLOSED BY '"'
ESCAPED BY '
\
\
' LINES TERMINATED BY '
\
n' IGNORE 2 LINES;

Services and clients configuration
:

There are 2

tables in the DB that needs to be update to add clients and services.

Clients:

o

table clients


just add new client name.

Services:

o

table serviceNodes

o

cid


client ID

o

surl


url that provide the service

o

transferType


json

for server services, any implemented type for client
services

o

sid


id of the service (service to sid pairing in services table)