Slides

educationafflictedΒιοτεχνολογία

4 Οκτ 2013 (πριν από 4 χρόνια και 1 μήνα)

95 εμφανίσεις

AGRESEARCH

SHOW AND TELL




Greg
Peyroux
, Information Solutions Manager

greg.peyroux@agresearch.co.nz


Alan McCulloch, Software Engineer

alan.mcculloch@agresearch.co.nz


Information Services Software Development

And Specialists Team


Development, Web and Data Services

OVERVIEW




Demonstrations of applications, services and data
of interest, discussion, expressions of interest


Group Structure


Team


Who and What We Do


New Direction and Business Focus


Development and Management Tools


Open source and cross platform tools and
frameworks


Examples

INFORMATION SERVICES GROUP

Information Technology and Knowledge Services



Knowledge


Library


Records management a


Knowledge advisory



IT


Technical


User Support


Development, Web, Data and Bioinformatics




Recent History of
AgResearch
:


Series of Reviews


‘Fit for Purpose’ Initiatives


Significant Team Turnover


Opportunity to innovate and improve


THE TEAM


REBUILT AND REPOSITIONED

How do you know that you add value?



Restructured team: Aligned to Outcomes


Emphasis on “All
AgResearch
” + the Bigger Picture


Focus on strategic goals


Greater Management and Business Awareness


Transparency: Work request portal on intranet





THE TEAM


REBUILD AND REPOSITION

Development Services


Priority on solutions for science and farms


Request Analysis



Determine optimum solution

»
Alternative approach e.g. open source

»
Reuse/repurpose existing system

»
Purchase shrink wrapped S/W or consultancy

»
Develop


Close
-
off, Feedback and Executive Reporting






A NEW DIRECTION

FOR
DEVELOPMENT

THE TEAM


REBUILDING

Is B “Best”?


Desperately

Seeking:


Fresh

Ideas and Concepts


Interns


Effective C
ommunication Skills



Increased

Science Capability:


PHD,
MScs
,
BScs



Statistical, Research + Technician
E
xperience


Critical Mass

THE EXTENDED TEAM

Software Development And Specialists Team (Information Solutions)

Development, Web and Data Services


Developers


Web, SharePoint


Windows, Mobile Devices


Lab (LIMS) and Farm Systems


Bioinformatics


Specialists


GIS


ERP analyst


Reporting Specialist


Database administrator


Records










TEAM IMPACT


Broad Span


Internal


All AgResearch


Science


Support


Farms


External


Farms


Meat processors


Commercial Labs


Accountants

and consultancy firms


Group and Team Structures Create Consistent , Aligned Service

Is B “Best”?

Microsoft Stack



NET framework


Migrating to TFS 2010


SQL Server


BizTalk


SharePoint

2010 (CMS, team, documents, My Sites etc)


Keen to collaborate on:


Public Records Act


Research Data

Management


Mapping Systems


Open Source and Cross
Plaform

Tools (Alan)



DEVELOPMENT TOOLS


Is B “Best”?

Organisation
-
Wide Data Integration




Support



Farm
-
Systems



Science data



Location



Field Data recording





KEY PROJECTS


INTEGRATION


Resolution Farm

Management and Planning


Integration of Information Systems

INTEGRATION
-

LOCATIONZ



Resolution Farm

Management and Planning


INTEGRATION
-

LOCATION



Resolution Farm

Management and Planning


Global Location Integration

and Management




Spatial Location


Silver light application


draw
, annotate, search and view


Import


GPS points


SHP


KML etc



Known


Desktop Application


Primarily for sample tracking



Global Location ID


Web service to guarantee retrieval





OVERVIEW OF FARMING/RESEARCH DATA


Farm

and

Related

Research

Data

Components

Environmental

Animal


Animal


locations

and


movements

Milk

yield

and

vat

Herd

Tests

and

results

etc

Primary data sets

Primary data sets

Location/GIS
-
related

Paddock/Transect

Points, Lines Polygons


Integrate into data warehouse, mine for traits

e.g. Best
calvers

in the wet

SMART DATA RECORDING AND STORAGE


Dynamic task creation and recording


Mobile device


User creates a custom task on their PC


Records task with a range of checks and rich validation


User uploads task data into a database


Excel Import and Metadata annotating




OVERVIEW OF FARMING/RESEARCH DATA


Task Recorder


Integrated Task Creation Framework

TOOLS FOR MANAGING PEOPLE, PRODUCTS,
PROJECTS AND CHANGE

Ugh! Look at his new
recording system, that
won’t last!


Porter’s 5 Forces


PEST


SWOT


Fatal Flaw


Stakeholder mapping


Tipping Point


Use of levers


Incubate advocates


Plan for resistors

Value of Tools

MANAGEMENT TOOLS AND FRAMEWORKS


MANAGEMENT TOOLS

Kotter
, J. (2007). Leading change: Why

transformational efforts fail?

Harvard Business Review

, 85(1), 96
-
103.

Managing Change:
Kotter’s

8 Steps to Transformation

OPEN SOURCE AND CROSS
-
PLATFORM
ENVIRONMENTS AND FRAMEWORKS

Allan McCulloch

WINDOWS AND UNIX HPC RESOURCES


CROSS
-
PLATFORM SOFTWARE DEVELOPMENT
-

STATUS


EXTENDED DATA MODELLING


DESIGN PATTERNS FOR DATA MODELLING


COLLABORATION SUPPORT
-

WIKIS

AND
DRUPAL





WINDOWS AND UNIX HPC RESOURCES


We use the Condor scheduler
-

creates and manages both Windows
and Unix compute pools. Mature, scalable, cross
-
platform



Windows = desktop machines , jobs run in background. Users
approached and volunteer machines to the pool (+ “captive”
machines in training rooms and IS and departmental staff !) as
and when required . Max size so far, simulation run on about
40 machines. Saved $$$ on leased high performance windows
boxes.


Unix = Centos
Redhat

/ Centos based , HP blades (mix of 32 and
64 bit , 2 and 4 core, 2Gb RAM ). B
-
word data pre
-
processing
mainly





“LARGE MEMORY” SMP UNIX MACHINES



1 x
Redhat

Linux database server (
postgres

,
mysql
, Oracle)



Data warehousing


B
-
word


GIS data (
postgres
)



1 x
Redhat

Linux compute server



Data Mining


Stats (R , ASREML)


B
-
word miscellany


Virtual machine provisioning (early days)



CROSS
-
PLATFORM (
LINUX
-
WINDOWS) IN
-
HOUSE
SOFTWARE DEVELOPMENT


Java based middleware supporting various B
-
word web
interfaces (not being further developed)


Small scale python,
perl

and R scripting for specific problems or
pipelines (/ bucket brigades). Unusual for us to develop anything
significant from scratch


huge communities, many packages and
solutions already out there.


Python based Object
-
Relational system supporting mainly B
-
word
databases


Mathematical modellers developing C++ based models



EXTENDED DATA MODELLING

What is it ?



ER : Entity
-
Relationship model : Chen (1976), formalized in Ng
(1981). Standard Terminology.


Extended Entity Relationship (EER) : little standard terminology ,
many contributors


EER only one of a number of semantic models


Developed to meet complex requirements beyond data
-
processing in business and industry


CAD/CAM db, image and
graphics db, cartographic and geological db.


Ref
ElMasri
, R.A. ,
Navathe

, S.B. : Fundamentals of Database
Systems

BRDF FRAMEWORK


SOME
EER

ASPECTS


Inheritance used to provide a core data provenance and record
identification and classification layer for all tables


Automatic date stamps and user ids for updates on any table


Keywords field for search engines


Record status for versioning


Class
typeid

for object
-
relational mapping support


Universal unique record identifier across all tables


really useful !


Heterogenous lists can include member records from many tables


e.g. search engine results; projects manifests (subjects, samples, lab
resources etc)


Normalised free text comments attached to any record


Provenance of all data imports
-

records are linked to the data source
they came from .


Entity types and relationship types treated uniformly







BRDF FRAMEWORK


Postgres

based (….but portable to
dbs

like Oracle,
SQLServer
)


Easy to install
-

single setup script


Around 8 instances mostly b
-
word related but also hosting
genetics and epidemiological data


Python based object
-
relational mapping and web interface


About to begin work on a C# /.net based OR mapping and new
front
-
end


(Custom object relational mappings are cost
-
effective and work
well for data processing , but objects do not usually plug into
standard GUI
componentry

, and so therefore are not cost
effective for UI development

)


(and vice
-
versa !!!


)




DESIGN PATTERNS FOR DATA MODELLING


“Design Patterns” (Gamma, Helm etc al.)


reusable pieces of
design in Object Oriented software engineering (e.g. observer,
visitor, factory method, façade, flyweight,….)



We need Design Patterns for databases to control complexity
and cost



Design patterns easier to define for EER based systems




IMPORT (IN) AND DISPLAY (OUT) DESIGN PATTERNS

STUDY
-
OBSERVATION DESIGN PATTERN IN
BRDF FRAMEWORK

STUDY
-
OBSERVATION DESIGN PATTERN SUGGESTED
DISCRETIZING

CONTINUOUS MS OBSERVATIONS INTO
BINS

DRUPAL

BASED COLLABORATION SUPPORT : MULTI
-
TENANTED DESIGN

APPLICATIONS



Some examples of applications

FARM APPLICATIONS



Resolution Farm

Management and
Planning



ETS Tools
-

Forestry


Resolution Farm

Management and Planning


FARM APPLICATIONS
-

ETS



MAF lookup
tables and rules



Google Map
interface



Extended GPS

support

SUMMARY





Open source and cross platform


Windows and Unix HPC resources


Cross
-
platform Software development


status


Extended Data Modelling


Design patterns for data modelling


Collaboration support
-

wikis and
Drupal


Development and Management Tools


Data integration


Change management


About Us


Team


Who and What We Do


Direction and Business Focus

AGRESEARCH

SHOW AND TELL




Greg
Peyroux
, Information Solutions Manager

greg.peyroux@agresearch.co.nz


Alan McCulloch, Software Engineer

alan.mcculloch@agresearch.co.nz


Thank you for your attention


Questions



Please feel free to contact us