Small AMI - iSites

righteousgaggleData Management

Jan 31, 2013 (4 years and 4 months ago)

441 views

Matt Bertrand



Project Goal



Compare expected turnout to actual turnout per voting ward on
election day


Prioritize last
-
minute get
-
out
-
the vote efforts.


Assess reliability of expected turnout estimates.


Analyze effectiveness of campaign’s get
-
out
-
the vote effort.






Requirements


Enter, edit, and view expected and actual votes
cast per ward


Display data on a map and table simultaneously


Import and export vote data


Click a ward on the map to bring up its data for
editing/viewing


Click a ward in the table to zoom in & highlight that
ward on map


Refresh map as soon as data is edited








Requirements


Choose between displaying expected turnout,
actual turnout and expected:actual turnout ratio
on the map


Automatically calculate ratio from entered data


Security: require login to the site. Two separate
login accounts
-

one for editing, one for viewing
only


Support ??? users (scalable).

Components


Virtual Server Machines


Amazon Web Services


Open Source GIS tools


PostgreSQL & PostGIS


OpenLayers


GeoServer


Other Open Source Tools


Apache web server


Tomcat Java application server


PHP DataGrid



Amazon Web Services


Elastic Compute Cloud (EC2)


Provides resizable computing capacity


Designed to make web
-
scale computing easy &
accessible to developers


Create and use “Amazon Machine Images” (AMI)


Virtual server machines


Many templates available


Choose your operating system (Linux, Windows)


Choose your horsepower (RAM, processor, etc).

Amazon Web Services


Elastic Block Store


Persistent storage for Amazon EC2 instances
(AMI’s).


Ideal for databases


Easily switch an EBS from one AMI to another


Use to swap database from a small AMI to a faster,
more powerful AMI

PostgreSQL and PostGIS


PostgreSQL


Relational database system



PostGIS


Spatial objects for PostgreSQL


Store, query, and manipulate spatial data objects





GeoServer


Open source geospatial server


Written in Java and the J2EE platform


Includes a graphical user interface for
configuration


Web Mapping Service


Web Feature Service

Web Map & Feature Services


Web Mapping Service (WMS)


Request: specifies the geographic layer(s) and area of interest to
be processed.


Response: returns geo
-
registered map images (JPEG, PNG, etc),
and/or layer attributes.


Web Feature Service (WFS)


Request: specifies the geographic layer(s) and feature subset of
interest based on spatial and/or non
-
spatial queries


Response: return description and/or attributes of the specified
layer features.


Also capable of updating, creating, and deleting features



OpenLayers


JavaScript library for displaying map data


Provides an API (Application Programming Interface) for
building web
-
based geographic applications.


Makes use of map “tiles”


Pre
-
rendered small sections of map, improves performance


Can load data from many sources:


Google Maps, Yahoo Maps, Virtual Earth


GeoServer, MapServer


Any Web Map Service or Web Feature Service


PHP DataGrid


Free data
-
bound grid control written in PHP


View, edit, and export data


Customizable


select which database fields to display,
turn editing on/off for each field, etc.


Putting it all together



AWS

GeoServer

OpenLayers

Apache/

Tomcat

PostGIS

?

=

PHP

DataGrid

Architectural Overview

Client


Servers

Data

PostgreSQL
/

PostGIS

GeoServer

WMS
Request

WFS

Request

Apache

Web Server

Standard
Web Request

Amazon Web Services

Tomcat

OpenLayers

AWS Scalability

Data

PostGreSQL

GeoServer

Apache

Data

PostGreSQL

GeoServer

Apache

EBS

Small AMI

EBS

X
-
Large AMI

X
-
Large AMI

Light Load Setup

Potential Heavy Load Setup

Small AMI:

1.7 GB RAM

1 32
-
bit Virtual Core =

1 EC2 Compute Unit

Extra Large AMI:

15 GB RAM

4 64
-
bit Virtual Cores =

8 EC2 Compute Units

AWS Setup

1.
Register for an account


Not free
-

cost is based on use of AMI’s and storage


~ $2/day for a small running AMI instance

2.
Pick a management console


ElasticFox addon for Firefox


Amazon now provides its own online console


Manage AMI and EBS instances

AWS Setup

3.
Select and start an AMI


Operating System: Ubuntu Linux


Size: small instance (1.7 GB RAM, 160 GB drive, 1 virtual
CPU equivalent to 32
-
bit 1.2 GHz Intel Xeon)

4.
Log in to the AMI using SSH (secure shell)

5.
Install and configure required software


Tomcat, Apache, Apache
-
Tomcat Connector, GeoServer,
PostgreSQL, PostGIS, PHP

AWS Setup

6.
Create an Elastic Block Store (EBS)


1 GB


plenty enough to hold NH ward data


Maximum size: 1 TB

7.
Attach the EBS to the running AMI instance

8.
Mount the EBS as a volume

9.
Create a PostGIS database on the mounted volume



Database Setup


A GIS shapefile of ward boundaries was provided by the
NH campaign staff


The shapefile was imported into PostgreSQL using a built
-
in PostGIS command (shp2pgsql)


Added new fields (turnout_expected, turnout_actual,
turnout_ratio)


Created indexes


Created a trigger to automatically calculate the ratio
whenever expected or actual turnout values changed


Database Setup


PostgreSQL UPDATE trigger


CREATE OR REPLACE FUNCTION updateratio()


RETURNS trigger AS



IF NEW.turnout_expected > 0 THEN




NEW.turnout_ratio =




NEW.turnout_actual / NEW.turnout_expected;



ELSE




NEW.turnout_ratio = 0;



END IF;



RETURN NEW;

GeoServer Configuration


Create a new DataStore


Stores connection information on PostgreSQL database


Create a new FeatureType


Stores information on a geographic vector feature stored in
the database (projection, boundaries, fields, etc).


Create Styled Layer Descriptors


XML format for specifying how to draw features on a map









SLD Rule Example


<sld:Rule>


<sld:Name>1.6
-

1.8</sld:Name>


<sld:Title>1.6
-

1.8</sld:Title>


<ogc:Filter>


<ogc:PropertyIsBetween>


<ogc:PropertyName>turnout_ratio</ogc:PropertyName>


<ogc:LowerBoundary>


<ogc:Literal>1.600001</ogc:Literal>


</ogc:LowerBoundary>


<ogc:UpperBoundary>


<ogc:Literal>1.8</ogc:Literal>


</ogc:UpperBoundary>


</ogc:PropertyIsBetween>


</ogc:Filter>


<sld:PolygonSymbolizer>


<sld:Fill>


<sld:CssParameter name="fill">#FF6F00</sld:CssParameter>


<sld:CssParameter name="fill
-
opacity">1</sld:CssParameter>


</sld:Fill>


<sld:Stroke>


<sld:CssParameter name="stroke">#6E6E6E</sld:CssParameter>


<sld:CssParameter name="stroke
-
width">0.4</sld:CssParameter>


<sld:CssParameter name="stroke
-
opacity">1</sld:CssParameter>


</sld:Stroke>


</sld:PolygonSymbolizer>


</sld:Rule>

Web Page Layout


Requirement: Display map and table/grid simultaneously

Map

<iframe>

PHP DataGrid page

(gridedit.php)

Main page (index.html)

OpenLayers


Requirement: Display ward data on a map


var wmsnh = new OpenLayers.Layer.WMS("NH Wards 4326",
“http://[....]
-
amazon.aws.com/geoserver/wms", {


layers: 'topp:nh_wards4326',


styles: sld,


srs: 'EPSG:4326',


format: 'image/png',


tiled: true,


transparent: true


},


{


'opacity': 0.75,


'isBaseLayer': false


});

OpenLayers


Requirement: Click a ward on the map to bring up its
data to edit/view



map.events.register('click', map,


function(e) {


var url =
"/geoserver/wfs?request=GetFeature&version=1.0.0&typeName=topp:
nh_wards4326&outputFormat=json&FILTER=%3CFilter%20xmlns=%22http
://www.opengis.net/ogc%22%20xmlns:gml=%22http://www.opengis.net
/gml%22%3E%3CIntersects%3E%3CPropertyName%3Ethe_geom%3C/Propert
yName%3E%3Cgml:Point%20srsName=%22EPSG:4326%22%3E%3Cgml:coordin
ates%3E" + lonlatGCS.lon + "," + lonlatGCS.lat +
"%3C/gml:coordinates%3E%3C/gml:Point%3E%3C/Intersects%3E%3C/Fil
ter%3E";


OpenLayers.loadURL(url, '', this, getGridPage);



OpenLayers.Event.stop(e);


});

OpenLayers


WFS Request

<Filter xmlns="http://www.opengis.net/ogc"
xmlns:gml="http://www.opengis.net/gml">


<Intersects>


<PropertyName>the_geom</PropertyName>


<gml:Point srsName="EPSG:4326">


<gml:coordinates>
-
75.12, 40.25






</gml:coordinates>


</gml:Point>


</Intersects>

</Filter>

OpenLayers


JSON (JavaScript Object Notation)


GeoJSON: JSON for geographic data

{


"type": "FeatureCollection",


"features": [


{



"type": "Feature",



"id": "nh_wards4326.48",



"properties": {



"objectid": 48,



"fips": 7160,



"name": "Pittsburg",




"twn_ward": null,



"ncec_code": "3300761780",




"turnout_expected": 3000,



"turnout_actual": 1000,



"turnout_ratio": 0.33,



},


"geometry": {


"type": "MultiPolygon",


"coordinates": …………



}



]

}



OpenLayers


Requirement: Highlight selected feature on map


First, create a highlight style and empty vector layer





var highlight_style = {


strokeColor: 'Red',


strokeWidth: 4,


strokeOpacity: 1,


fillOpacity: 0.0


};



//Add highlight vector layer for selected features


hilites = new OpenLayers.Layer.Vector("Highlighted", {


isBaseLayer: false,


features: [],


visibility: true,


style: highlight_style


});

OpenLayers


Add the selected feature to the ‘Highlighted’ vector layer


function highlightFeature(req)

{



var features = new
OpenLayers.Format.GeoJSON(out_options).read(req.responseText);




// have the Vector layer purge its feature list, replace them
with the new one


hilites.destroyFeatures();


hilites.addFeatures(features);


hilites.setVisibility(true);

}

OpenLayers


Requirement:Click a ward in the table to zoom in
& highlight on map

function highlightFeature(req)

{




if (zoomOnSelect)


{



bounds = features[0].geometry.getBounds();



map.zoomToExtent(bounds);


}

OpenLayers


Requirement: switch between maps of actual turnout,
expected turnout, and turnout ratio

function changeSLD(sldChosen) {


saveBounds();


sld = sldChosen;


init();


document.getElementById("legendImg").src =

'/geoserver/wms?REQUEST=GetLegendGraphic&VERSION=

1.0.0&FORMAT=image/png&WIDTH=100&HEIGHT=20&LAYER=

topp:nh_wards4326&sld=/geoserver/styles/'




+ sld + '.sld';

}


Data Import


Requirement: Import voter turnout data from an external
file into the database.


Comma
-
delimited text file


NCEC Code, expected turnout, actual turnout

Data Import


PHP import code snippet


$handle = fopen($uploadfile, "r");

while (($data = fgetcsv($handle)) !== FALSE) {


$result = pg_prepare($connection, "update" . $data[0],

'UPDATE nh_wards SET turnout_expected = $2,

turnout_actual=$3 WHERE ncec_code = $1');


$result = pg_execute($connection, "update" . $data[0],

array($data[0],$data[1],$data[2]));


}


}

fclose($handle);


Security


Requirement: Login with password to enter site


Two login roles


edit and view


Logins and passwords stored in ‘users’ table in database


PHP code used to validate user input against database
values


User forwarded to either data edit page or view page
depending on login role.

Final Step


Bundle the AMI


Saves current AMI setup as a new template


New AMI’s based on this template will be identical with all
software installed and configured.


Avoid having to reinstall PostgreSQL, Geoserver, etc. all over again
whenever a new instance is created.

Development Cost & Effort


Cost: $110.41


Amazon EC2 instance: $110.15


Amazon EBS storage: $0.26


Time: 4 weeks


~ 100 hours total


Energy:



Resources


Amazon Web Services


aws.amazon.com


GeoServer


geoserver.org


OpenLayers


openlayers.org


PostGIS
-

postgis.refractions.net


PostgreSQL


postgresql.org


Demo


openwebmap.com/votemap