: Visualizing all remote
data as local files to data analyzing jobs.
Computing & Software Systems University of Washington Bothell
Work expected for this summer consisted on
implementing the following functionality for the Connector
Connector as an FTP client.
Connector as and HTTP client.
Connector as stdin/stdout using a front
All of these
tasks were achieved and
in the fo
Overview of the project
Connector consists of several classes. The following is an overview of the classes that are part of
Please see javadoc pages attached for more details.
This is the package that contains the essential classes for connector. It includes:
llows a remote/local application to use standard input, output, and error
using a ConnectorPan
first instantiate and connect
to a Con
in order to transfer data from and to the ConnectorPanel. Once a connection is
established, interaction may begin.
Daemon process that manages I/O operations from applications. A
ConnectorDaemon starts automatic
ally once a Connector object is instantiated. Every I/O
request made is handled in a separate thread. ConnectorDaemon accepts requests from
Connector.java, FileInputStream.java, and FileOutput.java.
Provides a way to move data from ConnectorDaemon an
Since retrieving and storing data from servers is prone errors, DataPacket is a way to
differentiate between actual data or an error (or exception).
applications to read/display data from remote file servers.
Allows user applications to store data to remote file servers.
This is the front end application required to reroute standard input, output, and
a machine local to the user. This ConnectorPanel connects to a ConnectorDaemon to manage
I/O operations launched by Connector, FileInputStream, and FileOutputStream Objects from the
small programs to test Connectors’
classes. This directory
Simple program that tests the Connector's out, in, and err methods.
A ConnectorDaemon must be running.
: Same as ConnectorOutTest. The only d
ifference is that different data is
Tests reading from an http file and displaying contents to a ConnectorPanel.
A ConnectorPanel m
ust be running. Connector uses local “files.txt” to initiate
Test reading every 5 seconds data from an http file. ConnectorDaemon must b
running. ConnectorDaemon booted using args from "
Tests the behavior of ConnectorDaemon when multiple I/O requests ar
performed. A ConnectorPanel
must be running.
: Same behavior as MultiThread. The only difference is that this program
stores different data at the specified servers.
: Verifies the correct implementation of FileInputStream by trying to
two different files and displaying them through a ConnectorPanel.
: Verifies that files get stored correctly at remote servers. A
Connector object must be used in order to boot a ConnectorDaemon.
Implementation key points
or as an FTP client
The connection to a remote server occurs inside ConnectorDaemon. Whenever an application requires a
file to be read, the file is retrie
ved depending on the type of
file itself. The library Commons
used to retrieve from a regu
lar FTP server. When a file is stored at a secure FTP server, then the library
0.1.42 is used. The same libraries are used
f an application requires something
to be stored.
Connector as an HTTP client
As with FTP connections whenever an http file
needs to be read, the operations occur inside
ConnectorDaemon. The library used in this case is htmlparser.
Connector as standard input, output, and error using a front
In order to use FileInputStream, FileOutputStream, Connector’s out, in, an
d err methods, first a
Connector object needs to be instantiated in the application desiring to use s
uch objects. The reason
requiring a Connector object first is because its
constructor initializes and boots a ConnectorDaemon
process. Such ConnectorDa
emon process is essential since this process spawns threads needed to
perform I/O operations requested.
In order to instantiate a ConnectorDaemon, a Connector constructor uses a “file_map”. A file_map is a
text document where th
e arguments to boot a Conn
ectorDaemon are stated and the name of remote
files and their URL locations are defined. The format of a file_map is explained below.
Once a ConnectorDaemon is initialized, a Connector constructor also initializes a connection with a
ConnectorPanel. In or
der for the connection to take place, a ConnectorPanel must be already running
and listening to the correct IP address and port. In order to establish a ConnectorPanel connection, the
Connector constructor sends a signal to the ConnectorDaemon previously i
ConnectorDaemon then spawns a thread that manages I/O interactions until close( ) is called.
pages have more information about each constructor available.
As said before, in order for a ConnectorDaemon to boot properly, it ne
s a text document. The
following is the format required for the file_map.
To define a Connector argument, a line in the
file_map must appear as follows:
A pound sign indicates that the line is an argument necessary to boo
t ConnectorDaemon. After a space,
the argument option must be defined. As of now there are only two options “
i" indicates which IP
address a ConnectorDaemon is going to use to connect to a ConnectorPanel and “
p” indicates the port
After the optio
n is defined, the actual IP or port can be entered after a space.
If there are no lines with pound signs in the file_map, the ConnectorDaemon
will boot using the
as IP and the
11111 as default port.
After arguments to boot C
rDaemon are defined,
file names and URLs can be defined. The
[file name] [URL]
The format is very important since ConnectorDaemon will use this file to build a hash table with
notice that the name of the file itself is not included at the end of the URL. Even though the complete
path should include the name of the file at the end of the URL, the libraries used to retrieve files do not
How to use
As of now, all
test programs use the local file_map called “files.txt” to initialize a Connector object.
files.txt does not include any
IP or port parameters. So, if a test program wants to be connected to a
remote ConnectorPanel, then files.txt
must include the correct IP from where ConnectorPanel will be
In order to connect to initiate a Connector panel, do the following while logged in
$ cd SensorGrid/Connector/GUI
$ java ConnectorPanel
I had problems initiating a GUI from a
shell. So in order to start the ConnectorPanel GUI
it’s necessary to put additional arguments when running ConnectorPanel. Unfortunately, I don’t know
at this point,
but I will update this a
s soon as I can.
In terms of running a test program, the follow
ing should be done while logged in
at the dslab account:
$ cd SensorGrid/Connector/TestPrograms
The test programs available at the dslab
lder “TestPrograms” use a ConnectorPanel to display
output. So, a ConnectorPanel must be running before initiating a test program. There is also some
output displaying at the terminal’s console. However, displays at the console are intended for debugging
nd testing reasons.
and future work
I programmed and debugged in windows using Netbeans. However, when I transferred everything to
the dslab and tested on Linux, a few issues came up. One issue is about compiling and jarring all the
y files. I had problems trying to compile everything on the dslab. In the dslab I had to compile
everything manually. Because of that, the Connector folder in the dslab account might contain
unnecessary files. So, a thorough clean up of the Connector’s dir
ectory is necessary.
At this moment
test programs initialize connector objects using the string constructor only. Test
programs can only read
from their local directory in order to instantiate a ConnectorDaemon.
parameter constructor for
connector objects is supposed to read from the file
“tmp/Connector.config” for instantiation, however, that file has not been set up yet at the dslab. For
that reason, test programs only use their local file_map to construct connector objects.
oblem came up when
reading large files. Using my own windows machine, I encounter no
problems reading a large remote file. However, once I start using a Linux machine, the file doesn’t
transfer in its entirety.
The last problem I encountered has to do wit
h running a ConnectorPanel outside the local network.
When testing running a test program from a UWB machine while running a ConnectorPanel outside the
local network, the test program failed to commun
icate with the Panel. So at this moment, test programs
an only be tested while running a ConnectorPanel in the same Local Network.