Configuring Metadata Stores - IT Services

yompmulligrubsInternet και Εφαρμογές Web

31 Οκτ 2013 (πριν από 3 χρόνια και 11 μήνες)

70 εμφανίσεις







ANU Metadata Stores

System Administrator’s Manual













This work is licensed under a
Creative Commons Attribution
-
NonCommercial
-
ShareAlike 3.0 Australia
License
.



Table of Contents

Overview

................................
................................
................................
................................
.....

3

Licence
................................
................................
................................
................................
.........

3

Acknowledgement
................................
................................
................................
........................

3

Components
................................
................................
................................
................................
.

3

Architecture

................................
................................
................................
................................
.

4

Prerequisites

................................
................................
................................
................................

4

PostgreSQL

................................
................................
................................
...............................

4

Java Runtime Environment

................................
................................
................................
........

4

Tomcat
................................
................................
................................
................................
.....

5

Configuration

................................
................................
................................
...............................

5

Tomcat
................................
................................
................................
................................
.....

5

Set up access to the Manager web application

................................
................................
........

5

Maven
................................
................................
................................
................................
......

5

Configuring Metadata Stores

................................
................................
................................
.....

6

Setup Database

................................
................................
................................
.....................

6

Setup Miscella
neous
................................
................................
................................
..............

9

Building
................................
................................
................................
................................
........

9

Dependencies

................................
................................
................................
...........................

9

Build Process

................................
................................
................................
..............................

10

Clone the source repository
................................
................................
................................
.....

10

Execute Maven Build
................................
................................
................................
...............

10

Deployment

................................
................................
................................
...............................

10

Deployment using Tomcat Manager
................................
................................
.........................

10

Deployment using Maven Tomcat Plugin
................................
................................
..................

10

F.A.Q

................................
................................
................................
................................
.........

11

How do I encrypt a database password?

................................
................................
..................

11

How do I create/modify reports?

................................
................................
.............................

11

Troubleshooting

................................
................................
................................
.........................

11

SSL Exceptions

................................
................................
................................
........................

11




Overview

This document lists and explains the steps required to deploy and maintain an instance of the ANU
Metadata Stores software.

Licence

Use of ANU

Metadata Stores is governed by the GNU GPL3 licence.

Acknowledgement

This project is supported by the Australian National Data Service (ANDS). ANDS is supported by the
Australian Government through the National Collaborative Research Infrastructure Strate
gy Program
and the Education Investment Fund (EIF) Super Science Initiative.

Components


1.

Metadata
Store

This is the primary component of the project. The Metadata Store provides the means to
update, combine, and serve data from various sources. At this
point in time we are
retrieving information from the ANU Aries, LDAP, Data Commons and Digital Collections
systems. The store can be presented as either a standalone jar file or as a component in
web services.

Currently the Standalone JAR file updates inf
ormation via command line options while as a
component of the web services it serves information to clients.

2.

Metadata
Store Web Services

The Metadata Store Web Services provide access for external clients to retrieve information
from the Metadata Store. T
he web service provide
s

the ability to perform some queries

to
find

the data and
serves the

information for clients in JSON or XML format.

Included with the Metadata Store Web Services are some html pages that provide some
administrative function
ality such

as confirming or den
ying possible relationships and to
provide access to Metadata Store reports.

3.

OAI
-
PMH
Harvester

The harvester can harvest data from external OAI
-
PMH systems to allow for processing at a
later date/time. It pulls the data into individua
l items so that they can then be processed by
the Metadata Store. Examples of systems that data may be retrieved from are the Data
Commons and Digital Collections.

Architecture


Prerequisites

Components 1 and 2 are Java applications that require a server

(or virtual machine) capable of
running an operating system that supports Oracle Java. Refer to
http://www.oracle.com/technetwork/java/javase/downloads/index.html

for a lis
t of operating
systems capable of hosting a Java Runtime Environment.

We are running Red Hat Enterprise Linux Server 5.8 running on a virtual machine with 2GB of RAM.

1.

PostgreSQL. Available from
http://www.postgresql.org/download/

2.

Java Runtime Environment

a.

Tomcat 7.x

If using Linux as the ope
rating system some of the afore
mentioned programs may be available in
your distribution’s repository. Check your package manager for more details.

PostgreSQL

The ANU Metadata Store holds information in a PostgreSQL database.

Java Runtime Environment

Installation of the Java Runtime Environment (JRE) is specific to the platform that it is being installed
on. Refer to your operating system’s user manual for deta
ils. The Java Development Kit (JDK) is
required if you intend to perform remote debugging.

Tomcat

Tomcat is an application server that can be used for hosting web applications. Tomcat has a build
-
in
web server that can be used to serve HTTP requests witho
ut the need for a dedicated web server
such as Apache HTTP Server.

We have developed our application using Tomcat 7.0.27
.

Configuration

Tomcat

If you wish to encrypt a password held in configuration files it is advisable to add the java system
property value named
app.pwd
. To do this the option:

-
Dapp.pwd=”AnEncryptionPassword”

S
hould be added to

either the
JAVA_OPTS

or
CATALINA_OPTS

environment

variable

that will be
used when tomcat starts
.

Set

up access to the Manager web application

Accessing the Manager application through the web interface requires a Tomcat user to be set up
with the
manager
-
gui

role. Accessing the same application through
a scripted interface, such as
through the Maven Tomcat Plugin requires a Tomcat user to be set up with the
manager
-
script

role. Refer to
http://tomcat.apache.org/tomcat
-
7.0
-
doc/man
ager
-
howto.html

for details on how to
configure the Manager application.

Maven

For Maven to deploy applications to Tomcat, you will need to create one or more profiles that
include information such as the URL where the Tomcat instance is hosted along with
the credentials
to use to access its manager application. Refer to the section Set up access to the Manager web
application to setup the users whose credentials you’d like to use to deploy applications through
Maven.

To create a profile for a tomcat instan
ce, open the settings.xml file. Refer to
http://maven.apache.org/settings.html

to find the location of settings.xml. In the file, add the
following XML:

<servers>



<server>



<id>INSTANCE_ID</id>



<
username>USERNAME</username>



<password>PASSWORD</password>

</server>



</servers>

<profiles>




<profile>



<id>PROFILE_ID</id>



<properties>




<maven.tomcat.url>
http://HOSTNAME:PORT
/manager/text</maven.tomc
at.url
>




<maven.tomcat.server>INSTANCE_ID</maven.tomcat.server>



</properties>


</profile
>



</profile
s>


Parameter

Value

INSTANCE_ID

An arbitrary ID assigned to the username and
password to be used for sending requests to the
Manager application.

USERNAME

The username to which manager
-
script role is
assigned.

PASSWORD

Password for the username above.

PROFILE_ID

An arbitrary ID assigned to the tomcat instance
to which a Web application will deploy.
Generally, you’d have one p
rofile for 瑨e
developmen琠瑯浣a琠ins瑡nce, one for 瑥s瑩ng
and one for produc瑩on.

HOSTNAMN

The fully qualified hos瑮ame where 瑨e 瑯浣a琠
ins瑡nce is loca瑥d. For exa浰le,
浥瑡da瑡
-
s瑯res
.co洠.

PORT

The por琠on which 瑨e 瑯浣a琠ins瑡nce is
lis瑥ning on. For exa浰le, 8080 .

INSTAN䍅_ID

The ID assigned 瑯 瑨e co浢ina瑩on of userna浥
and password 瑯 be used for deploying
applica瑩ons 瑨rough 瑨e Manager applica瑩on.


Configuring Metadata Stores

Setup
Database

Metadata Stores requires two databases to be created and configured

and one to be configured
.
One database contains the stores information, the other harvested information.

Store Database

The Store database holds the retrieved metadata to serve t
o clients.

The different types of objects
i.e. People, Publications, Grants, etc. are contained within the same set of tables however they are
discriminated via the
ext_system

column

in the
Item

table.


Figure
1
: Metadata Store S
chema

The Stores database is created and initial values filled out via
executing scripts found In the
met
adata
-
stores/store/src/main/sql

directory.

Create the Stores Database in a PostgreSQL instance by executing the following commands:

psql

Upostgres

f
c
reate_database.sql

psql

Umsuser

fcreate_tables.sql


d
metadatastores
db

psql

Umsuser

fcreate_sequences.sql
-

d
metadatastores
db

psql

Umsuser

f3_add_data.sql

d
metadatastores
db

Then execute each of the SQL files in the format YYYYMMDD_NAME.sql in order:

psql

Umsuser

fYYYYMMDD_NAME.sql


d
metadatastores
db

To
establish a connection between the store and the database created, modify the

following
properties in

store.cfg.xml

found in the
store/src/main/resources

directory.

Property

Value

hibernate.connection.driver_class

Change to an appropriate value if using a database other than
PostgreSQL

hibernate.connection.url

The URL of the database. For example,
jdbc:postgresql://hostname:1234/dbname

hibernate.connection.user


Username to use to

connect to the database. For example,
dcuser

hibernate.connection.password

Password to use to connect to the database.

hibernate.dialect

Change to an appropriate value if using a database server other
than PostgreSQL

Harvest Database

The harvest databa
se contains two tables, one to indicate from where harvesting should occur, the
other to hold the content that has been harvested.


Figure
2
: Harvester Schema

The Harvest database is created in initial values filled out via execut
ing sc
ripts found in the
metadata
-
sto
re
s/stores
-
harvester/src/main/sql

directory.

Create the Harvest Database in a PostgreSQL instance by executing the following commands:

psql

U
postgres


f
create_database.sql

psql

Uharvestuser
-
fcreate_tables.sql
-
dharvestdb

To establish a connection between the harvester and the database created, modify the

following
properties in

harvester.cfg.xml

found in the
stores
-
harvester/src/main/resources

directory.

Property

Value

hibernate.connection.driver_class

Change t
o an appropriate value if using a database other than
PostgreSQL

hibernate.connection.url

The URL of the database. For example,
jdbc:postgresql://hostname:1234/dbname

hibernate.connection.user


Username to use to connect to the database. For example,
dcu
ser

hibernate.connection.password

Password to use to connect to the database.

hibernate.dialect

Change to an appropriate value if using a database server other
than PostgreSQL


Aries Database

The Aries database is an existing database that contains research information. To establish a
connection between the harvester and the database created, modify the
aries.cfg.xml

found in
the
aries/src/main/resources

directory.

The user attempting to upda
te information will
require an account that has access to the Aries database.

Property

Value

hibernate.connection.driver_class

Change to an appropriate value if using a database other than
SQL Server

hibernate.connection.url

The URL of the database. For
example,
jdbc:jtds:sqlserver
://hostname:1234/dbname
;instance
=XXXX

Note
: C
onnecting to

the SQL server database

from

an
environment

other than a standard windows login (e.g. from
Linux) you will also need the domain to connect to.

hibernate.connection.user


Username to use to connect to the database. For example,
dcuser

Note:
This prop
erty is not necessary if logged

in
to
a windows
computer
with

a user who has access to the Aries database.

hibernate.connection.password

Password to use to connect to the data
base.

Note:
This prop
erty is not necessary if logged

in
to a windows
computer with

a user who has access to the Aries database.

hibernate.dialect

Change to an appropriate value if using a database server
other than
SQL Server
.


Setup Miscellaneous

LDAP

To configure the retrieval of data from LDAP the following properties will need to be updated in the
file
ldap/src/main/resources/ldap.properties
.

Property

Value

lda
p
.uri

This is the URL of the LDAP service

ldap.baseDN

The base distinguished name for the

LDAP service

ldap.peopleDn

The distinguished name for finding people within the LDAP service

Harvesting Locations

Insert rows into the

location table of the harvest database. This should include the system, the url of
the OAI provider, the metadata pre
fix, how frequently the data should be harvested (in milliseconds)
and an original harvest date to calculate when to next harvest the data It will harvest the nex
t. If for
whatever reason you do not wish to harvest the entire feed but rather from a speci
fic date, this can
be done via setting the last harvest date.

Currently there are two locations that have code that will process their data, Data Commons and
Digital Collections. The default values for these are DATA_COMMONS and DIGITAL_COLLECTIONS
respec
tively. These can be renamed, and for processing to occur the
harvest.location.datacommons

and
harvest.location.digitalcollections

values will
need to be updated in the
store.properties
.
store.properties

is located in the
store/src/main/resources

directo
ry.

Building

Dependencies

As Metadata Stores is a Maven project its dependencies will automatically be pulled from a Maven
repository.

Build Process

ANU Metadata Stores users

Maven as its build tool.

To build the projects from source perform the
following steps:

Clone the source repository

Clon
e the GitHub repository where ANU Metadata Stores source code is hosted
-

https://github.com/anu
-
doi/metadata
-
stores.git

Execute Maven Build

Execute the following command in the directory where the ANU Metadata Stores repository has
been cloned to compile and build the project into JAR and WAR files:

mvn clean package

If any tests fail, run:

mvn clean

package

DskipTests

It is also possible to create a runnable jar for the store. In which case navigate to the store directory
and execute:

mvn clean package assembly:single
-
DskipTests

Deployment

Once the project has been built
, the generated WAR files
can be deployed to Tomcat by using the
Tomcat manager application or using Maven itself.

Deployment using Tomcat Manager

Assuming the Tomcat instance is hosted at
http://localhost:8080/
, Open Tomcat Manager
applicatio
n at
http://localhost:8080/manager
. Scroll down to the section titled ‘Deploy’. Click on the
browse button, select the WAR file for the application you would like to deploy in the file browser
dialog box and s
elect OK. Then Click on the Deploy button to deploy the application to Tomcat.


Deployment using Maven Tomcat Plugin

The applications can be deploy
ed to Tomcat by executing the following in the project directory:

mvn tomcat:deploy
-
only

p PROFILE_ID

where

PROFILE_ID is the profile ID assigned to a tomcat instance. Refer to section Configuring
Maven.

This command only deploys the web applications in the metadata
-
stores project without executing
the package phase. The modules must already be packaged for th
em to be deployed. If you’d like to
package the application and deploy using a single command, execute the following:

mvn tomcat:undeploy

F.A.Q

How do I encrypt a database password?

Download the latest distribution release from jasypt (
http://www.jasypt.org/
) and unpack the zip file.
Navigate to the bin directory and execute the following command:

>

encrypt input=”TheDatabasePassword” password=”the
-
app.pwd
-
value”
algorithm=”PBEWithMD5AndDES”

The encrypted value will then be returned belo
w the line containing ‘
---
OUTPUT
---


This value should then be placed in the appropriate hibernate configuration file surrounded by
‘ENC()’. For example t
he H
ibernate password property becomes:

<property name="hibernate.connection.password">

ENC(lP+z+2cKM
GlM4vRsM5tl9d72FAJTiF6f)

</property>

Note: the app.pwd value is as set up in the
Tomcat

Configuration.

How do I create/modify reports?

The reports have been created wi
th iReport 4.7.0 (
http://community.jaspersoft.com/project/ireport
-
designer

) and are compiled

and run

using the

jasper reports
engine
(
http://community.jaspersoft.com/project/jasperreports
-
library

).

After creating/modifying the report and subreports in iReport to satisfaction place the jrxml file
s

in
the
services
/src/main/webapp/WEB
-
INF/reports

directory. The reports will be
automatically compiled w
hen the web service starts. M
odifications to the ReportResource and
ReportGenerator classes may also be required

to be able to access the reports

from the front end

o
n
the web server
.

Trou
bleshooting

SSL Exceptions

If the tomcat instance is configured for SSL connections from clients, it is vital that the private key
and public certificate are correctly configured. Refer to the section SSL Configuration HOW
-
TO at
http://tomcat.apache.org/tomcat
-
7.0
-
doc/ssl
-
howto.html

.

If using a self
-
signed certificate on the server, all clients connecting to it must have the certificate in
its trusted certs store.
This includes applications within Tomcat that act as clients to other
applications in the same Tomcat instance.