SPNHC 2012 Demo Deployment

From Filtered Push Wiki
Jump to: navigation, search

Prerequisites

Below are the prerequisites prior to deployment.

Servers

1. Tomcat7 on port 8080

sudo apt-get install tomcat7

2. Glassfish 3.1 on port 8081 (download from: http://glassfish.java.net/downloads/3.1-final.html)

wget  http://download.java.net/glassfish/3.1/release/glassfish-3.1-unix.sh

3. Apache2 on port 80

sudo apt-get install apache2)


PHP and Pears

1) PHP5 configured with apache with the following extensions: pear, mysql, mysqli, uuid, curl

sudo apt-get install php5 libapache2-mod-php5 php-pear php5-mysqlnd php5-uuid php5-curl

2) Required PEAR packages (for Morphbank):

sudo pear install Archive_Tar Config Console_Getopt HTTP HTTP_Request2 Log
sudo pear install MDB2 MDB2_Driver_mysql MDB2_Driver_mysqli Mail Net_Socket Net_URL Net_URL2 PEAR Structures_Graph XML_Util


Maven

Once maven is installed, it needs to be configured to use the archiva repository. Copy the settings file from /root/.m2/settings.xml from firuta to the .m2/ directory in your home directory on your machine. This way maven will know where to obtain fp specific dependencies that are defined in the Annotation Processor pom.

Other

Also need MySQL server as well as maven, svn, git

sudo apt-get install mysql-server mysql-client
sudo apt-get maven subversion git


Obtain Sources

1) Morphbank + Symbiota:

git clone git://filteredpush.git.sourceforge.net/gitroot/filteredpush/morphbank
git clone git://filteredpush.git.sourceforge.net/gitroot/filteredpush/symbiota

2) Client Helper:

svn co https://filteredpush.svn.sourceforge.net/svnroot/filteredpush/FP-RdfHandler/trunk FP-RdfHandler
svn co https://filteredpush.svn.sourceforge.net/svnroot/filteredpush/FP-ClientHelper/trunk FP-ClientHelper
svn co https://filteredpush.svn.sourceforge.net/svnroot/filteredpush/FP-Deployments/FP-AnnotationWS FP-AnnotationWS

3) Network Node:

svn co https://filteredpush.svn.sourceforge.net/svnroot/filteredpush/FP-API-Library/trunk FP-API-Library
svn co https://filteredpush.svn.sourceforge.net/svnroot/filteredpush/FP-Deployments/FP-EAR FP-EAR
svn co https://filteredpush.svn.sourceforge.net/svnroot/filteredpush/FP-Node/trunk FP-Node
svn co https://filteredpush.svn.sourceforge.net/svnroot/filteredpush/FP-Messaging/FP-Messaging-KVP/trunk FP-Messaging-KVP

4) Annotation Processor (contains 3 subprojects):

svn co https://filteredpush.svn.sourceforge.net/svnroot/filteredpush/FP-Access/trunk FP-Access

5) SpecifyWeb

svn co https://filteredpush.svn.sourceforge.net/svnroot/filteredpush/FP-Tools/fp-oauth-provider/specifyweb FP-Tools-oauth-specifyweb


Configuration

Here is configuration that needs to be done prior to build and deploy.

Apache

I have added the following to /etc/hosts on my local machine:

127.0.0.1       symbiota
127.0.0.1       morphbank (morphbank main application)
127.0.0.1       mbimages (morphbank image server)

For each of the following I have a corresponding configuration in /etc/apache2/sites-available:

1) The morphbank site should have [MORPHBANK_HOME]/www as its DocumentRoot and Directory, this is where the main app lives
2) The config for morphbank site must have the following line: php_value auto_prepend_file "[MORPHBANK_HOME]/configuration/app.server.php"

3) The mbimgaes site should have [MORPHBANK_HOME]/ImageServer as its DocumentRoot and Directory, this is where the image server lives
4) The mbimages site must have the following line: php_value auto_prepend_file "[MORPHBANK_HOME]/configuration/image.server.php"


Glassfish

1) Download and extract mysql-connector-java-5.1.20.jar from the archive found at http://www.mysql.com/downloads/connector/j/
2) Copy (we will also need this jar when initializing the Annotation Store) mysql-connector-java-5.1.20.jar to .../glassfish3/glassfish/domains/domain1/lib

Steps below are in the admin console: http://localhost:4848

3) On the left select Resources > JDBC > Connection Pools and create a new connection pool named "mysql_messagestore" with the resource set as java.sql.ConnectionPoolDataSource and MySql selected as the vendor. In step two, you should configure the Password, DatabaseName (messagestore) and User properties at a minimum. Also, append the database name to the URL property (like jdbc:mysql://:3306/messagestore). Once you have done this click the finish button. Select the connection pool again from the list and edit it. On the edit page click the ping button to test your configuration.
4) Now on the left select Resources > JDBC > JDBC Resources and create a new resource "jdbc/__mysql_messagestore" with the connection pool you just created.

Note: if you end up using a different name for your connection pool you must edit the persistence.xml file in the src/main/resources/META-INF directory of the FP-Messaging-KVP project.

Node Config

1) The configuration for the network node and components can be found in FP-Node/src/main/resources/config.properties. For a localhost deployment, the jndi names for each of the three components that start with "java:global" should be uncommented and the last three in this file should be commented out.

2) We will configure sdb.ttl later for the Annotation Store (in the [Database Initialization] step)

Annotation Processor

Rename FP-Access/FP-AnnotationProcessor-Web/src/main/filters/filter.properties.sample to filter.properties and make sure that the properties in this file have the correct values for your set up.

Specify Web

Rename specifyweb/src/main/filters/filter.properties.sample to filter.properties and make sure that the properties in this file have the correct values for your set up.

NOTE: the database name will be fpannotationprocessor (we will create it in a later step) the db.url should be:

db.url = jdbc:mysql://localhost:3306/fpannotationprocessor

Also, copy the mysql-connector-java-5.1.20.jar (the Connector/J downloaded from the MySQL site) to your tomcat/lib directory (i.e. /usr/share/tomcat7/lib/).

Morphbank and Symbiota

Morphbank and Symbiota need the correct host name, url and port for both the accesspoint and clienthelper webservice endpoints. These can be configured via FPConfig.php prior to deploy. Edit the following files found in your git checkout dirs (morphbank and symbiota)


NOTE: fp/FPConfig.php not found in git checkouts.

1) in morphbank/mb32/www/includes/fp/FPConfig.php you will find the endpoint urls, configure these for use by the filtered push php libraries.
2) in symbiota/classes/fp/FPConfig.php you will find the endpoint urls, configure these for use by the filtered push php libraries.

Database Initialization

Initialize the schema for the applications that make use of MySQL:

Morphbank and Symbiota

1) Morphbank: http://sourceforge.net/apps/mediawiki/biodivimage/index.php?title=Biodiversity_Image_Repository#Install_MB32_Sample_Database

2) Symbiota (the excerpt below is from http://symbiota.org/tiki/tiki-index.php?page=Installation+Instructions):

  • Create new database schema appropriately named for project
  • Create read-only and read/write users for new database
  • Load schema from scripts
    • Schema definition files are located in <SymbiotaBaseFolder>/config/schema/ and all start with db_schema
    • Database can be installed with ISO-8859-1 or UTF8 character set definitions. Note that source data needs to match database and website character set definitions
    • If your MySQL database is a pre 5.1 version, you must use ver_5.0 schema versions
  • Load Stored Procedures located in <SymbiotaBaseFolder>/config/schema/stored_procedures.sql
  • Run database patch scripts available for upload version. Make sure to run the scripts in the correct order.

Message Store

Find the sql dump inside FP-EAR/messagestore.sql and load it into a new mysql database called messagestore (also create a user and assign appropriate privileges)

Annotation Store

This is the SDB triple store running on top of a MySQL implementation. See http://jena.apache.org/documentation/sdb/installation.html

We need to create the initial table structure in MySQL, use the following instructions:

  • setup the environment for command line (replace the values with correct locations): (Is this just for the sdbconfig command? --Maureen Kelly 11:00, 12 September 2012 (EDT))
$ export SDBROOT="/path/to/sdb"
$ export SDB_JDBC="/path/to/driver.jar"
  • create database fpannotations in mysql:
  • use the sdbconfig to initialize the database with a command like:
bin/sdbconfig --sdb=home/lowery/workspace/FP-Node/src/main/resources/sdb.ttl --create
  • Initialization done, from now on the messaging system will use the sdb.ttl file in the src/main/resources so make sure its settings are correct and that your changes were saved.

Annotation Processor

Create database fpannotationprocessor in MySQL and load the sql dump found in FP-Access/FP-AnnotationProcessor-Web/fpannotationprocessor.sql

Specify Sample Data

Create database specify_fp_sample in MySQL and load the sql dump found on firuta in /home/shared/specify/specify_fp_sample.20120607.sql.gz

Build

Use maven to build all components and create the deployment packages. I've noticed that order of execution matters. (You will probably need to configure your maven settings.xml file to access the Archiva repository on firuta first).


1) Build and install RDF Handler lib (creates FP-RdfHandler/target/FP-RdfHandler-0.0.1-SNAPSHOT.jar):

cd FP-RdfHandler
mvn install

2) Build Client Helper libs and AnnotationWS (creates FP-AnnotationWS/target/FP-AnnotationWS-0.0.1-SNAPSHOT.war):

cd FP-ClientHelper
mvn install -DskipTests
cd ../FP-AnnotationWS
mvn install

3) Build API-Library and FP-EAR (creates FP-AnnotationWS/target/FP-EAR-0.0.1-SNAPSHOT.ear):

cd FP-API-Library
mvn install
cd FP-Messaging-KVP
mvn install
cd FP-Node
mvn install -DskipTests
cd FP-EAR
mvn install

4) Build FP-Access

NOTE: Tests fail but the application works, skip them in order to build the war file successfully (see below).

NOTE: You will need to have a filter.properties file created. This file does not come with the source, but a sample file does. Be sure to edit the filter.properties file to reflect your deployment configuration.

cd FP-Access
cp FP-AnnotationProcessor-Web/src/main/filters/filter.properties.sample FP-AnnotationProcessor-Web/src/main/filters/filter.properties
mvn install -Dmaven.test.skip=true

5) Build specifyweb oauth provider

NOTE: You will need to have a filter.properties file created. This file does not come with the source, but a sample file does. Be sure to edit the filter.properties file to reflect your deployment configuration.

cp src/main/filters/filter.properties.sample src/main/filters/filter.properties
cd FP-Tools-oauth-specifyweb


Deployment

Here are the instructions for deployment.

Client Helper WS

cp FP-AnnotationWS/target/FP-AnnotationWS-0.0.1-SNAPSHOT.war /var/lib/tomcat7/webapps/annotationws.war

FP-EAR

/usr/share/glassfish3/glassfish/bin/asadmin deploy --force FP-EAR/target/FP-EAR-0.0.1-SNAPSHOT.ear

/etc/init.d/glassfish restart

AnnotationProcessor

cp FP-Access/FP-AnnotationProcessor-Web/target/FP-AnnotationProcessor-Web.war /var/lib/tomcat7/webapps

sudo /etc/init.d/tomcat7 restart

Specify Web

Copy and rename the war file for this project to your tomcat webapps directory. Rename it specifyweb.war.

cp specifyweb/target/specifyweb-0.0.1-SNAPSHOT.war /var/lib/tomcat7/webapps/specifyweb.war

Morphbank

1) Copy the mb32 directory from the morphbank git checkout to the apache www folder

cp -r morphbank/mb32 /var/www/

2) Do the following to satisfy the installer that starts when you first view morphbank:

chmod -R a+w /var/www/mb32/log
chmod -R a+w /var/www/mb32/configuration

mkdir /var/www/mb32/www/images/
mkdir /var/www/mb32/www/images/mirrorLogos
mkdir /var/www/mb32/www/images/newsImages
mkdir /var/www/mb32/www/images/userLogos

chmod -R a+w /var/www/mb32/www/images

3) deploy the mb_data directory (found in the tar on firuta) to /var/www (this is where our images are for the specimens)

4) go to http://morphbank/ and fill in the required fields for install, the install.php will create a user and make some changes in the db related to the object procedures

5) make sure that /var/www/mb32/configuration/config.ini matches my config, also change the alt image server to point to morphbank (this way we get all the images to show up)

Symbiota

1) Copy the symbiota directory from the git checkout to the apache www folder.

cp -r /symbiota /var/www/

2) Rename the following files (remove the part at the end "_template"):

mv /var/www/symbiota/index_template.php /var/www/symbiota/index.php
mv /var/www/symbiota/header_template.php /var/www/symbiota/header.php
mv /var/www/symbiota/leftmenu_template.php /var/www/symbiota/leftmenu.php
mv /var/www/symbiota/footer_template.php /var/www/symbiota/footer.php
mv /var/www/symbiota/css/main_template.css /var/www/symbiota/css/main.css
mv /var/www/symbiota/css/jquery-ui_template.css /var/www/symbiota/css/jquery-ui.css 
mv /var/www/symbiota/css/speciesprofile_template.css /var/www/symbiota/css/speciesprofile.css

3) Edit and rename the following to configure:

mv /var/www/symbiota/config/dbconnection_template.php /var/www/symbiota/config/dbconnection.php mv /var/www/symbiota/config/symbini_template.php /var/www/symbiota/config/symbini.php

4) Copy symbiota/images to /var/www/symbiota from the tar on firuta. (contains the icons for the collections)

Filtered Push

  • See my tar archive on firuta for database dumps (morphbank and symbiota should be replaced with our data and we also need the specify data in there)
  • Also compare configurations, there is a list of all the config files in the README in the tar.
  • Lastly, in the floating directory in the tar, run "floating patch" to add the floating div for the demo.

Revision numbers

The components of the project that have tags have been tagged "SPNHC_2012_DEMO" in the SourceForge repository. Since not all do, here are the revision numbers that correspond to the demo deployment:

FP-API-Library1351
FP-Access1402
FP-ClientHelper1252
FP-Deployments1250
FP-Design1329
FP-Drivers1374
FP-Messaging1338
FP-API-Library1201
FP-Network1399
FP-Tools1376

Cleaning out demonstration annotations

To remove the annotations produced when walking through the script for the demonstration using the specimen with barcode number 673, run the following queries:

use messagestore;
delete from kvp;
delete from message;
use fpannotations;
delete from Nodes;
delete from Triples;
use mb32;
delete from DeterminationAnnotation where specimenid = 581553;
delete from Annotation where objectid = 581554;
use symbiota;
delete from omoccurdeterminations where occid = 4922 and detid > 13;
update omoccurrences set family='Rosaceae',sciname='Crataegus barrettiana',tidinterpreted=5394,scientificnameauthorship='Sargent',identifiedby='D. E. Boufford',dateidentified='1984-01-01' where occid = 4922;
use specify_fp_sample
delete from determination where collectionobjectid = 234481 and determinationid > 324854;