Connectors: Visualizing all remote data as local files to ...

moodusroundoSoftware and s/w Development

Aug 15, 2012 (5 years and 3 months ago)

230 views






Connectors
: Visualizing all remote
data as local files to data analyzing jobs.

Computing & Software Systems University of Washington Bothell





Jose Melchor

Summer 2010























Introduction


Work expected for this summer consisted on
implementing the following functionality for the Connector
project:


1.

Connector as an FTP client.

2.

Connector as and HTTP client.

3.

Connector as stdin/stdout using a front
-
end process.

4.

Test programs.


All of these

tasks were achieved and
are
explained

in the fo
llowing sections.


Overview of the project


Connector consists of several classes. The following is an overview of the classes that are part of
Connector.

Please see javadoc pages attached for more details.


Connector Package



This is the package that contains the essential classes for connector. It includes:




Connector Class



A
llows a remote/local application to use standard input, output, and error
using a ConnectorPan
el.
Constructors

first instantiate and connect

to a Con
nector
Daemon

process

in order to transfer data from and to the ConnectorPanel. Once a connection is
established, interaction may begin.



ConnectorDaemon Class

-

Daemon process that manages I/O operations from applications. A

ConnectorDaemon starts automatic
ally once a Connector object is instantiated. Every I/O
request made is handled in a separate thread. ConnectorDaemon accepts requests from
Connector.java, FileInputStream.java, and FileOutput.java.



DataPacket Class



Provides a way to move data from ConnectorDaemon an
d ConnectorPanel.
Since retrieving and storing data from servers is prone errors, DataPacket is a way to
differentiate between actual data or an error (or exception).



FileInputStream Class



Allows user

applications to read/display data from remote file servers.



FileOutputStream Class



Allows user applications to store data to remote file servers.


ConnectorPanel Class



This is the front end application required to reroute standard input, output, and
e
rror to
a machine local to the user. This ConnectorPanel connects to a ConnectorDaemon to manage
I/O operations launched by Connector, FileInputStream, and FileOutputStream Objects from the
Connector package.


TestPrograms
directory
-

I
ncludes several
small programs to test Connectors’

classes. This directory

includes:



ConnectorOutTest
.java
:
Simple program that tests the Connector's out, in, and err methods.

A ConnectorDaemon must be running.



ConnectorOutTest2
.java
: Same as ConnectorOutTest. The only d
ifference is that different data is
being transferred.



HttpTest
.java
:
Tests reading from an http file and displaying contents to a ConnectorPanel.

A ConnectorPanel m
ust be running. Connector uses local “files.txt” to initiate
ConnectorDaemon.



HttpTest2
.ja
va
:
Test reading every 5 seconds data from an http file. ConnectorDaemon must b
e
running. ConnectorDaemon booted using args from "
files.txt
".



MultiThread
.java
:
Tests the behavior of ConnectorDaemon when multiple I/O requests ar
e
performed. A ConnectorPanel

must be running.



MutiThread2
.java
: Same behavior as MultiThread. The only difference is that this program
stores different data at the specified servers.



TestFileInputStream
.java
: Verifies the correct implementation of FileInputStream by trying to
read
two different files and displaying them through a ConnectorPanel.



TestFileOutputStream
.java
: Verifies that files get stored correctly at remote servers. A
Connector object must be used in order to boot a ConnectorDaemon.


Implementation key points


Connect
or as an FTP client

The connection to a remote server occurs inside ConnectorDaemon. Whenever an application requires a
file to be read, the file is retrie
ved depending on the type of

file itself. The library Commons
-
net
-
ftp is
used to retrieve from a regu
lar FTP server. When a file is stored at a secure FTP server, then the library
jsch
-
0.1.42 is used. The same libraries are used
i
f an application requires something

to be stored.


Connector as an HTTP client

As with FTP connections whenever an http file
needs to be read, the operations occur inside
ConnectorDaemon. The library used in this case is htmlparser.


Connector as standard input, output, and error using a front
-
end process

In order to use FileInputStream, FileOutputStream, Connector’s out, in, an
d err methods, first a
Connector object needs to be instantiated in the application desiring to use s
uch objects. The reason

for
requiring a Connector object first is because its

constructor initializes and boots a ConnectorDaemon
process. Such ConnectorDa
emon process is essential since this process spawns threads needed to
perform I/O operations requested.


In order to instantiate a ConnectorDaemon, a Connector constructor uses a “file_map”. A file_map is a
text document where th
e arguments to boot a Conn
ectorDaemon are stated and the name of remote
files and their URL locations are defined. The format of a file_map is explained below.


Once a ConnectorDaemon is initialized, a Connector constructor also initializes a connection with a
ConnectorPanel. In or
der for the connection to take place, a ConnectorPanel must be already running
and listening to the correct IP address and port. In order to establish a ConnectorPanel connection, the
Connector constructor sends a signal to the ConnectorDaemon previously i
nstantiated.
ConnectorDaemon then spawns a thread that manages I/O interactions until close( ) is called.

Javadoc
pages have more information about each constructor available.


File_map

As said before, in order for a ConnectorDaemon to boot properly, it ne
ed
s a text document. The
following is the format required for the file_map.


To define a Connector argument, a line in the
file_map must appear as follows:


#
-
i 127.0.0.1


#
-
p 12345

A pound sign indicates that the line is an argument necessary to boo
t ConnectorDaemon. After a space,
the argument option must be defined. As of now there are only two options “
-
i" indicates which IP
address a ConnectorDaemon is going to use to connect to a ConnectorPanel and “
-
p” indicates the port
to use.

After the optio
n is defined, the actual IP or port can be entered after a space.


If there are no lines with pound signs in the file_map, the ConnectorDaemon
will boot using the
localhost

address
as IP and the

port

11111 as default port.


After arguments to boot C
onnecto
rDaemon are defined,

file names and URLs can be defined. The
format is:

[file name] [URL]

example:
test1.txt


ftp://css432:UWB0th3ll@ftp.tripod.com/project



The format is very important since ConnectorDaemon will use this file to build a hash table with

it. So,
notice that the name of the file itself is not included at the end of the URL. Even though the complete
path should include the name of the file at the end of the URL, the libraries used to retrieve files do not
need
it.


How to use

the Connector
system


As of now, all

test programs use the local file_map called “files.txt” to initialize a Connector object.
files.txt does not include any
IP or port parameters. So, if a test program wants to be connected to a
remote ConnectorPanel, then files.txt
must include the correct IP from where ConnectorPanel will be
running.


In order to connect to initiate a Connector panel, do the following while logged in
at
the dslab:



$ cd SensorGrid/Connector/GUI


$ java ConnectorPanel


*NOTE:
I had problems initiating a GUI from a
Linux

shell. So in order to start the ConnectorPanel GUI
it’s necessary to put additional arguments when running ConnectorPanel. Unfortunately, I don’t know
the arguments
needed

at this point,
but I will update this a
s soon as I can.


In terms of running a test program, the follow
ing should be done while logged in
at the dslab account:



$ cd SensorGrid/Connector/TestPrograms


$ java

cp
Connector.jar:. [
test
-
program
-
name
]


The test programs available at the dslab
’s

fo
lder “TestPrograms” use a ConnectorPanel to display
output. So, a ConnectorPanel must be running before initiating a test program. There is also some
output displaying at the terminal’s console. However, displays at the console are intended for debugging
a
nd testing reasons.


Known Issues

and future work


I programmed and debugged in windows using Netbeans. However, when I transferred everything to
the dslab and tested on Linux, a few issues came up. One issue is about compiling and jarring all the
necessar
y files. I had problems trying to compile everything on the dslab. In the dslab I had to compile
everything manually. Because of that, the Connector folder in the dslab account might contain
unnecessary files. So, a thorough clean up of the Connector’s dir
ectory is necessary.


At this moment
test programs initialize connector objects using the string constructor only. Test
programs can only read
a file_map
from their local directory in order to instantiate a ConnectorDaemon.
The no
-
parameter constructor for

connector objects is supposed to read from the file
“tmp/Connector.config” for instantiation, however, that file has not been set up yet at the dslab. For
that reason, test programs only use their local file_map to construct connector objects.


Another pr
oblem came up when
reading large files. Using my own windows machine, I encounter no
problems reading a large remote file. However, once I start using a Linux machine, the file doesn’t
transfer in its entirety.


The last problem I encountered has to do wit
h running a ConnectorPanel outside the local network.
When testing running a test program from a UWB machine while running a ConnectorPanel outside the
local network, the test program failed to commun
icate with the Panel. So at this moment, test programs
c
an only be tested while running a ConnectorPanel in the same Local Network.