Connectors: Visualizing all remote data as local files to ...

moodusroundoSoftware and s/w Development

Aug 15, 2012 (4 years and 8 months ago)


: Visualizing all remote
data as local files to data analyzing jobs.

Computing & Software Systems University of Washington Bothell

Jose Melchor

Summer 2010


Work expected for this summer consisted on
implementing the following functionality for the Connector


Connector as an FTP client.


Connector as and HTTP client.


Connector as stdin/stdout using a front
end process.


Test programs.

All of these

tasks were achieved and

in the fo
llowing sections.

Overview of the project

Connector consists of several classes. The following is an overview of the classes that are part of

Please see javadoc pages attached for more details.

Connector Package

This is the package that contains the essential classes for connector. It includes:

Connector Class

llows a remote/local application to use standard input, output, and error
using a ConnectorPan

first instantiate and connect

to a Con


in order to transfer data from and to the ConnectorPanel. Once a connection is
established, interaction may begin.

ConnectorDaemon Class


Daemon process that manages I/O operations from applications. A

ConnectorDaemon starts automatic
ally once a Connector object is instantiated. Every I/O
request made is handled in a separate thread. ConnectorDaemon accepts requests from,, and

DataPacket Class

Provides a way to move data from ConnectorDaemon an
d ConnectorPanel.
Since retrieving and storing data from servers is prone errors, DataPacket is a way to
differentiate between actual data or an error (or exception).

FileInputStream Class

Allows user

applications to read/display data from remote file servers.

FileOutputStream Class

Allows user applications to store data to remote file servers.

ConnectorPanel Class

This is the front end application required to reroute standard input, output, and
rror to
a machine local to the user. This ConnectorPanel connects to a ConnectorDaemon to manage
I/O operations launched by Connector, FileInputStream, and FileOutputStream Objects from the
Connector package.


ncludes several
small programs to test Connectors’

classes. This directory


Simple program that tests the Connector's out, in, and err methods.

A ConnectorDaemon must be running.

: Same as ConnectorOutTest. The only d
ifference is that different data is
being transferred.

Tests reading from an http file and displaying contents to a ConnectorPanel.

A ConnectorPanel m
ust be running. Connector uses local “files.txt” to initiate

Test reading every 5 seconds data from an http file. ConnectorDaemon must b
running. ConnectorDaemon booted using args from "

Tests the behavior of ConnectorDaemon when multiple I/O requests ar
performed. A ConnectorPanel

must be running.

: Same behavior as MultiThread. The only difference is that this program
stores different data at the specified servers.

: Verifies the correct implementation of FileInputStream by trying to
two different files and displaying them through a ConnectorPanel.

: Verifies that files get stored correctly at remote servers. A
Connector object must be used in order to boot a ConnectorDaemon.

Implementation key points

or as an FTP client

The connection to a remote server occurs inside ConnectorDaemon. Whenever an application requires a
file to be read, the file is retrie
ved depending on the type of

file itself. The library Commons
ftp is
used to retrieve from a regu
lar FTP server. When a file is stored at a secure FTP server, then the library
0.1.42 is used. The same libraries are used
f an application requires something

to be stored.

Connector as an HTTP client

As with FTP connections whenever an http file
needs to be read, the operations occur inside
ConnectorDaemon. The library used in this case is htmlparser.

Connector as standard input, output, and error using a front
end process

In order to use FileInputStream, FileOutputStream, Connector’s out, in, an
d err methods, first a
Connector object needs to be instantiated in the application desiring to use s
uch objects. The reason

requiring a Connector object first is because its

constructor initializes and boots a ConnectorDaemon
process. Such ConnectorDa
emon process is essential since this process spawns threads needed to
perform I/O operations requested.

In order to instantiate a ConnectorDaemon, a Connector constructor uses a “file_map”. A file_map is a
text document where th
e arguments to boot a Conn
ectorDaemon are stated and the name of remote
files and their URL locations are defined. The format of a file_map is explained below.

Once a ConnectorDaemon is initialized, a Connector constructor also initializes a connection with a
ConnectorPanel. In or
der for the connection to take place, a ConnectorPanel must be already running
and listening to the correct IP address and port. In order to establish a ConnectorPanel connection, the
Connector constructor sends a signal to the ConnectorDaemon previously i
ConnectorDaemon then spawns a thread that manages I/O interactions until close( ) is called.

pages have more information about each constructor available.


As said before, in order for a ConnectorDaemon to boot properly, it ne
s a text document. The
following is the format required for the file_map.

To define a Connector argument, a line in the
file_map must appear as follows:


p 12345

A pound sign indicates that the line is an argument necessary to boo
t ConnectorDaemon. After a space,
the argument option must be defined. As of now there are only two options “
i" indicates which IP
address a ConnectorDaemon is going to use to connect to a ConnectorPanel and “
p” indicates the port
to use.

After the optio
n is defined, the actual IP or port can be entered after a space.

If there are no lines with pound signs in the file_map, the ConnectorDaemon
will boot using the

as IP and the


11111 as default port.

After arguments to boot C
rDaemon are defined,

file names and URLs can be defined. The
format is:

[file name] [URL]


The format is very important since ConnectorDaemon will use this file to build a hash table with

it. So,
notice that the name of the file itself is not included at the end of the URL. Even though the complete
path should include the name of the file at the end of the URL, the libraries used to retrieve files do not

How to use

the Connector

As of now, all

test programs use the local file_map called “files.txt” to initialize a Connector object.
files.txt does not include any
IP or port parameters. So, if a test program wants to be connected to a
remote ConnectorPanel, then files.txt
must include the correct IP from where ConnectorPanel will be

In order to connect to initiate a Connector panel, do the following while logged in
the dslab:

$ cd SensorGrid/Connector/GUI

$ java ConnectorPanel

I had problems initiating a GUI from a

shell. So in order to start the ConnectorPanel GUI
it’s necessary to put additional arguments when running ConnectorPanel. Unfortunately, I don’t know
the arguments

at this point,
but I will update this a
s soon as I can.

In terms of running a test program, the follow
ing should be done while logged in
at the dslab account:

$ cd SensorGrid/Connector/TestPrograms

$ java

Connector.jar:. [

The test programs available at the dslab

lder “TestPrograms” use a ConnectorPanel to display
output. So, a ConnectorPanel must be running before initiating a test program. There is also some
output displaying at the terminal’s console. However, displays at the console are intended for debugging
nd testing reasons.

Known Issues

and future work

I programmed and debugged in windows using Netbeans. However, when I transferred everything to
the dslab and tested on Linux, a few issues came up. One issue is about compiling and jarring all the
y files. I had problems trying to compile everything on the dslab. In the dslab I had to compile
everything manually. Because of that, the Connector folder in the dslab account might contain
unnecessary files. So, a thorough clean up of the Connector’s dir
ectory is necessary.

At this moment
test programs initialize connector objects using the string constructor only. Test
programs can only read
a file_map
from their local directory in order to instantiate a ConnectorDaemon.
The no
parameter constructor for

connector objects is supposed to read from the file
“tmp/Connector.config” for instantiation, however, that file has not been set up yet at the dslab. For
that reason, test programs only use their local file_map to construct connector objects.

Another pr
oblem came up when
reading large files. Using my own windows machine, I encounter no
problems reading a large remote file. However, once I start using a Linux machine, the file doesn’t
transfer in its entirety.

The last problem I encountered has to do wit
h running a ConnectorPanel outside the local network.
When testing running a test program from a UWB machine while running a ConnectorPanel outside the
local network, the test program failed to commun
icate with the Panel. So at this moment, test programs
an only be tested while running a ConnectorPanel in the same Local Network.