20101015 - IRMIS3 Statusx - Google Drive

greenbeansneedlesΛογισμικό & κατασκευή λογ/κού

13 Δεκ 2013 (πριν από 3 χρόνια και 3 μήνες)

258 εμφανίσεις

IRMIS3 Status

Gabriele Carcassi

15 Oct 2010

IRMIS3 Status


This talk is about:


what is currently downloadable from the website


what is ready for production


what is actively supported

Project infrastructure

Production server at BNL since January 2009

“alpha”
-

AuthZ

and
AuthN

only designed

Likely implemented by the end of the year

Build and test server

Automated tests

IE or
firefox

on Windows/Linux/Mac supported

SourceForge

project (mailing list, bug tracking, wiki).

Monthly release cycle.

IRMIS3 SERVICE
ARCHITECTURE

IRMIS3 architecture

IRMIS DB

(
MySQL
)

Data service

Java

applets

AJAX

components

Scripts and CLI

(
perl
, Python, …)

Browser

(Firefox, IE, …)

Web server

(Glassfish)

Data service enforces business rules

(clients can be dumb)

Data service allows db schema or

business rule changes
while in production

Data service provides a good abstraction

layer on which to integrate

Software stack

XML protocol (REST style WS)

Few database utilities: backup,
consistency check, etc…

Client

Server

Integration with external
tools (i.e.
physcs
)

3
rd

party Perl/
Pyton

scripts

3
rd

party Java apps
Jython

scripts

Web applications

JavaScript bridge

Applets and
Widgets

Java Client API

Data Service layer

Database layer

Getting data

<
componentType

id="90" description="Oscilloscope: 4 Channel @ 500MS/s; 500MHz BW" name="HP54540A">


<manufacturer id="5" name="Agilent (HP)"/>


<
formFactor

id="5" description="Freestanding"/>


<functions>


<function id="46" description="CCMS"/>


<function id="8" description="Instrument"/>


</functions>


<requires>


<interface id="143" description="
GPIB_Slave
"
relType
="control"/>


<interface id="5" description="Freestanding"
relType
="housing"/>


<interface id="78" description="120VAC"
relType
="power"/>


</requires>


<presents>


<interface id="234" description="Port"
relType
="control"/>


</presents>


<ports>


<port name="
Chnl

1">


<
portType

id="13" name="BNC
-
F" group="RF Connectors">


<
pinDesignator

id="303" name="1"/>


</
portType
>


<pin usage="
Chnl

1"
pinDesignatorId
="303">


<
signalType

id="1" direction="IN"/>


</pin>


</port>


<port name="
Chnl

2">...</port>


<port name="
Chnl

3">...</port>


<port name="
Chnl

4">...</port>


<port name="RS232">...</port>



...



</ports>

</
componentType
>

This is an example of what
the data service might
deliver

XML returned

REST style (data service)

Id references

De
-
normalized data

Writing data

<transaction
xmlns:xsi
='http://www.w3.org/2001/XMLSchema
-
instance'


xmlns
='http://xml.bnl.gov/schema/irmis'


xsi:schemaLocation
='http://xml.bnl.gov/schema/irmis irmis.xsd'>


<create>


<manufacturer name="
Acopian
"/>


</create>


<update>


<
componentType

id="2" name="Site" description="Overall facility. Root of housing hierarchy.">


<manufacturer id="1" name="None"/>


<
formFactor

id="1" description="Virtual"/>


<properties/>


<functions/>


<requires/>


<presents/>


<ports/>


</
componentType
>


</update>


<delete>


<
formFactor

id="4" description="
IndustryPack
"/>


</delete>


<connect>


<cable color="blue" label="AA"
portAId
="123"
portBId
="321"/>


<conductor
pinAId
="123"
pinBId
="321"/>


<conductor
pinAId
="124"
pinBId
="322"/>


</connect>

</transaction>

XML describing a transaction sent through a POST

allows remote/disconnected work

no lock “FOR UPDATE”

idempotent actions (error or same result)

different independent “commands”

Implementation

MySQL

JPA

JAXB

SAX

Business

logic

Servlet

Read

Write

Stream out XML as soon as possible

to reduce delay

JAXB to convert transaction request

to Java objects

JPA
vs

JDBC: simpler queries to maintain

6% performance loss

Performance


Dell Precision M4300 laptop, Intel Core Duo T7500, 2.20 GHz
-

APS dataset


Entire component data set streamed in about 5 seconds (31338
components, 63911 relationships)


Most other queries under 100 ms (manufacturers 88ms, interfaces
95ms)


Roughly half of the time is spent in DB access and half in
XML generation


Streaming starts right after DB access


Only tuning done is on the database query (was “good
enough”)


Other areas: hardware/
os

tuning, xml generation improvements,
profiling, caching, …

IRMIS3 CLIENT API

Client API


To access the service, XML and HTTP is all you need


Available in all languages


Two libraries provided that allow to work with objects


PHP API from MSU


Java API from BNL


Only going to talk about the Java API

Java API: design principles


The Java API implement the XML protocol and provides
objects that represent the data


All reads are cached: no need to keep caches in your
code.


All reads are synchronized


you can have reads on multiple threads on the same data and
will not trigger multiple reads on the server (extremely useful for
GUIs)


Java API design principles


The read access is object oriented


Data is exposed through interfaces

-
Component,
ComponentType
, Manufacturer


Some classes function as “finders” for the classes containing the
data

-
Components.getById
(123),
FormFactors.getByName
(“Rack”)


You can follow references from one object to the other

-
component.getComponentType
().
getManufacturer
()

Java API design principles


The write access is protocol oriented


You create a transaction


You call methods to add elements in the transaction

-
Manufacturers.createManufacturer
(transaction, “ACME”)


You save the transaction, which may fail


The read access is refreshed only after the data is saved

Importing components from Excel
(using
Jython
)

from
gov.bnl.irmis.api

import *

from java.net import URI

from java.io import File

from
jxl

import *


workbook =
Workbook.getWorkbook
(File(“components.xls"))

sheet =
workbook.getSheet
(0)


nRows

=
sheet.getRows
()

nColumns

=
sheet.getColumns
()


irmisDB

=
IrmisDB.getInstance
();

irmisDB.setURI
(URI("http://cs
-
build.nsls2.bnl.gov:8080/IRMISDataService/data"))


transaction =
irmisDB.createTransaction
();

for row in range(1,
nRows
):


componentTypeName

=
sheet.getCell
(0, row).
getContents
();


componentType

=
ComponentTypes.getByName
(
componentTypeName
);


serialNumber

=
sheet.getCell
(1, row).
getContents
();


Components.createComponent
(transaction,
componentType
,
serialNumber
)

transaction.save
();

print "Operation successful"


Importing from Excel (
Jython
)

from
gov.bnl.irmis.api

import *

from java.net import URI

from java.io import File

from
jxl

import *


workbook =
Workbook.getWorkbook
(File(“components.xls"))

sheet =
workbook.getSheet
(0)


nRows

=
sheet.getRows
()

nColumns

=
sheet.getColumns
()


irmisDB

=
IrmisDB.getInstance
();

irmisDB.setURI
(URI("http://cs
-
build.nsls2.bnl.gov:8080/IRMISDataService/data"))


transaction =
irmisDB.createTransaction
();

for row in range(1,
nRows
):


componentTypeName

=
sheet.getCell
(0, row).
getContents
();


componentType

=
ComponentTypes.getByName
(
componentTypeName
);


serialNumber

=
sheet.getCell
(1, row).
getContents
();


Components.createComponent
(transaction,
componentType
,
serialNumber
)

transaction.save
();

print "Operation successful"


Load excel file

Importing from Excel (
Jython
)

from
gov.bnl.irmis.api

import *

from java.net import URI

from java.io import File

from
jxl

import *


workbook =
Workbook.getWorkbook
(File(“components.xls"))

sheet =
workbook.getSheet
(0)


nRows

=
sheet.getRows
()

nColumns

=
sheet.getColumns
()


irmisDB

=
IrmisDB.getInstance
();

irmisDB.setURI
(URI("http://cs
-
build.nsls2.bnl.gov:8080/IRMISDataService/data"))


transaction =
irmisDB.createTransaction
();

for row in range(1,
nRows
):


componentTypeName

=
sheet.getCell
(0, row).
getContents
();


componentType

=
ComponentTypes.getByName
(
componentTypeName
);


serialNumber

=
sheet.getCell
(1, row).
getContents
();


Components.createComponent
(transaction,
componentType
,
serialNumber
)

transaction.save
();

print "Operation successful"


Connect to the
service

Importing from Excel (
Jython
)

from
gov.bnl.irmis.api

import *

from java.net import URI

from java.io import File

from
jxl

import *


workbook =
Workbook.getWorkbook
(File(“components.xls"))

sheet =
workbook.getSheet
(0)


nRows

=
sheet.getRows
()

nColumns

=
sheet.getColumns
()


irmisDB

=
IrmisDB.getInstance
();

irmisDB.setURI
(URI("http://cs
-
build.nsls2.bnl.gov:8080/IRMISDataService/data"))


transaction =
irmisDB.createTransaction
();

for row in range(1,
nRows
):


componentTypeName

=
sheet.getCell
(0, row).
getContents
();


componentType

=
ComponentTypes.getByName
(
componentTypeName
);


serialNumber

=
sheet.getCell
(1, row).
getContents
();


Components.createComponent
(transaction,
componentType
,
serialNumber
)

transaction.save
();

print "Operation successful"


Loop over the
rows

Importing from Excel (
Jython
)

from
gov.bnl.irmis.api

import *

from java.net import URI

from java.io import File

from
jxl

import *


workbook =
Workbook.getWorkbook
(File(“components.xls"))

sheet =
workbook.getSheet
(0)


nRows

=
sheet.getRows
()

nColumns

=
sheet.getColumns
()


irmisDB

=
IrmisDB.getInstance
();

irmisDB.setURI
(URI("http://cs
-
build.nsls2.bnl.gov:8080/IRMISDataService/data"))


transaction =
irmisDB.createTransaction
();

for row in range(1,
nRows
):


componentTypeName

=
sheet.getCell
(0, row).
getContents
();


componentType

=
ComponentTypes.getByName
(
componentTypeName
);


serialNumber

=
sheet.getCell
(1, row).
getContents
();


Components.createComponent
(transaction,
componentType
,
serialNumber
)

transaction.save
();

print "Operation successful"


Get data out

Create
components

Fluent API


Navigating through hierarchies can be tedious


On top of the “beans” API there is a fluent API
(internal domain specific language)

Search API

Builders

Change Set API

IRMIS3
USER INTERFACE

UI design


Web interface: uses applets and Web Start (both greatly
improved with Java 6u10)


Uses Swing standard components plus open source
(
SwingX
, Flamingo, JIDE, …)


UI elements can optionally be used in custom pages and
integrated with
javascript


UI available only for Components and Cabling


UI elements

Component hierarchy tree widget (filtering, sorting, search, …)

Component path

widget

Component applications

Cabling applications


STATUS
BY AREA

Components


Support for component and component types is
production level


Data model has been tested extensively through previous use at
other sites


Performance tests were done using APS datasets


Most of the work now is in UI development


Most applications at their 3
rd

iteration


Cabling


Support for cabling is still beta


No considerable amount of data has been put into it


No performance tests done (see lack of data)


UI at their second iteration


Need users!

Lattice


There is preliminary support for lattice


Only through the API, no UI


A separate tree structure of “elements”, each of an
“element type”. Each relationship has alignment
information.


Lattice cannot be modified

in place: copy on write creates

a new version


Right now, provided as
-
is

1

1

1

1

1

2

2

PV


PV service and API is considered alpha. No UI yet. No
crawler scripts provided yet.