Slide 1 - QI-Bench

gayheadtibburInternet and Web Development

Feb 5, 2013 (4 years and 9 months ago)

232 views

Monthly Program Update

January 12, 2012








Andrew J. Buckler, MS

Principal Investigator

WITH FUNDING
SUPPORT
PROVIDED BY
NATIONAL
INSTITUTE OF
STANDARDS
AND
TECHNOLOGY

Agenda


Monthly snapshot in
Jira


(including status of installation at NIST)


QIBA 3A project snapshot


Theoretical development


Architecture and SW stack

2

2

3

BSD
-
2 license

Domain is

www.qi
-
bench.org
.


Landing page
provides


Access
to
prototypes,


Repositories
for
download and
development,


Acknowledgemen
ts
,


Jira

issue tracking,
and


Documentation

3

QIBA 3A PROJECT SNAPSHOT

(
recalling that this is a
testbed

for us)

4

4

4

Basic structure of the challenges

5

Pilot

Pivotal

Investigation 1

Train

Test

Pilot

Pivotal

Investigation

Train

Test

Pilot

Pivotal

Investigation

Train

Test

Pilot

Pivotal

Investigation n

Train

Test


Defined set of data


Defined challenge


Defined test set policy

5

5

5

First one:


Presently in pilot phase,


Using
StudyDescription

method


Used batch scripting with reference method to
aid data
curation


10
-
12 participants (about 20 QI
-
Bench users)


First participant data received


Analysis plan using N
-
way ANOVA in R started



Pivotal phase starting with batch assisted
curation


Will be transitioning to database schema for
metadata (gradually away from spreadsheet)

0
1
2
3
4
5
6
7
8
9
10
Pooled Bias
Pooled Variability
Variability across shapes
Variability across densities
Variability across slice
thicknesses
Method A
Method B
Method C
Method D
Method E
Method F
Group
6

6

6

1.
Relative performance is computed according
to descriptive statistics

2.
We determine a group value for each of the
descriptive statistics, e.g., as the mean plus 1
stdev

(or as wide as we think wise).

3.
Results presented using radar plots

Bias

Variability

Repeatability

cross
-
x
reproducibili
ty

cross
-
y
reproducibili
ty

New Method

7.00

3.00

10.00

11.00

12.00

Group

7.16

6.64

6.96

7.81

9.00

0.00
5.00
10.00
15.00
Bias
Variability
Repeatability
cross-x
reproducibility
cross-y
reproducibility
New Method
Group
In this example, the new proposed method
does not perform well enough to be
considered a valid method since it falls
outside the group values.

7

7

7

In this example, the new proposed method
is seen to perform within group values and
may even help pave the way for an
improved claim.

Bias

Variability

Repeatability

cross
-
x
reproducibility

cross
-
y
reproducibility

New
Method

3.00

3.00

4.00

6.00

8.00

Group

7.16

6.64

6.96

7.81

9.00

0.00
2.00
4.00
6.00
8.00
10.00
Bias
Variability
Repeatability
cross-x
reproducibility
cross-y
reproducibility
New Method
Group
8

8

8

THEORETICAL DEVELOPMENT

progress re: utilization of logical and statistical inference at
each of two levels, technical performance of assay methods,
and qualification of biomarker in specific clinical context


9

9

9

Another way to look at what needs to
happen

10

10

10

10

Formulate

Statistical Analysis
Results (Relation
strength)

Annotation and
Image Markup,
Non
-
imaging
Clinical Data

Primary Data:
Images and other
Raw Data

Reference Data Sets

QIBO

Specify

RDF Triple Store

CT
Volumetry

CT

obtained_by

Tumor
growth

m
easure_of

Therapeutic

Efficacy

used_for

Analyze

Y=β
0..n

1
(QIB)+β
2
T+
e
ij

Execute

Specify
:
Establish a logical specification and
setup terms for mathematical analysis

11

11


Functionality:


Establish means to semantically labeling
imaging biomarker data with emphasis on
representing both the clinical context in which
an imaging biomarker is used as well as the
specifics of the imaging protocol used to acquire
the images.


Set up the logistic regression model:


Precisely specify dependant variable


Account for covariates


Enumerate independent variables and
error terms (sources of variability)


Establish database for collection of terms.


Method:


Provide GUI to traverse the QIBO concepts
according to their relationships and create
statements represented as RDF triples and
stored in an RDF store.


Each set of RDF triples will be stored as a
“profile.”


Relationship strength initialized based on prior
estimates (if available)

QIBO

Specify

RDF Triple Store

CT
Volu
metry

CT

obtained_by

Tumor
growt
h

m
easure_of

Therap
eutic

Efficac
y

used_for

Ontologies

supporting
Specify

12

12


Extend the QIBO to link to existing
established
ontologies

1.
leverage BFO upper ontology to
align different
ontologies

2.
convert portions of BRIDG and
LSDAM to ontology models in
OWL


Automated conversion would done
in two steps:

1.
convert current
Sparx

Enterprise Architect XMI EMF
UML format

2.
export resulting EMF UML into
a RDF/OWL representation
using
TopBraid

Composer


Formulate
: advanced query framework
made possible by
Specify

13

13


allow users to select the profiles (or set of
RDF triplets) created in Specify, execute a
query and retrieve the results in various
forms.


assemble/transform the set of RDF triples
to SPARQL queries:

1.
form an uninterrupted chain linking
the instance of the input class from
the ontology to the desired output
class

2.
formulate/invoke necessary SPARQL
queries against the web services
deployed in SADI framework.


interface with the query engine and will
have offline (asynchronous) query
execution capability.


results to be exportable as serialized
objects (RDF/XML and CSV)

Formulate

Statistical
Analysis
Results
(Relation
strength)

Annotation
and Image
Markup,
Non
-
imaging
Clinical
Data

Primary
Data:
Images and
other Raw
Data

Reference Data Sets

RDF Triple Store

CT
Volu
metry

CT

obtained_by

Tumor
growt
h

m
easure_of

Therap
eutic

Efficac
y

used_for

Data Services supporting
Formulate

14

14


wrap existing data services such as NBIA,
caArray
,
caTissue
, AIM and PODS using
Semantic Automated Discovery and
Integration (SADI)


this is enabled by metadata available
through the UML representations of
the models exposed by these services
and CDE annotations available for
them through
caDSR
.


describe service I/O semantically using the
extended version of QIBO


service registry of SADI will help the
automated composition of computer
-
interpretable queries by the query engine.


example: “there is a service that
returns Biological Subjects that has
undergone certain Biological
Interventions”

Analyze
: Use annotation and image
markup to support statistical inference

15

15


Support Clinical Performance assessment (i.e.,
in addition to current Technical Performance)


Outcome studies


Integrated genomic/proteomic
correlation studies


Group studies for biomarker qualification


(set up a basic multiple regression analysis,
e.g.) Intent to treat analysis of the primary
outcome via covariance model of the general
form (
QIB
t
)=β
0..n

1
(QIB
0
)+β
2
T+
e
ij

where
QIB
t

and
QIB
0

are the QIB at a time after treatment and at
randomization respectively, T is a treatment group indicator,
and β
0..n
, β
1
, and β
2

are model parameters. β
2

represents the
effect of treatment and its estimate is the difference
between group means on the log scale, after adjustment for
any imbalance between the groups in log QIB. The error
terms in the model,
e
ij
, are assumed mutually independent
and normally distributed. Depending on the nature of the
QIB, the log transformation may be used instead of the
direct value. Likewise calculations may be performed using
z scores with corresponding conversion with raw values.

Quantitative Imaging
Specification Language
Batch Analysis
Service
Reference Data Set
Manager
UPICT Protocols, QIBA
Profiles, literature
papers and other
sources
QIBO
-
BatchMake
Scripts
Reference Data Sets, Annotations, and
Analysis Results
(red edges
represent
biostatistical
generalizability)
Source of clinical study
results
Clinical Body of Evidence
(formatted to enable
SDTM and/or other
standardized registrations
4. Output
3. Batch analysis
scripts
UPICT Protocols, QIBA
Profiles, entered with
Ruby on Rails web
service
QIBO
Examples of output at biomarker
(above the assay level)

16

16

From Jack 1999. Note: W
-
score is the relative score of the
measured HC volume corrected for intracranial volume and
compared to age and sex adjusted
normals
.

Hypothetical model of dynamic biomarkers of the
Alzheimer's pathological cascade

Jack et al., 2010

16

16

16

To inform
thresholding

To substantiate surrogacy
(or its weaker form of “activity”)

ARCHITECTURE AND SW STACK

So what is a cohesive architecture that maximizes leverage of
best thinking, existing
touchpoints
, and stays current over
time?


17

17

17

STDM
standard of
CDISC into
repositories
like FDA’s
Janus.

MVT
portion of
AVT, re
-
useable
library of R
scripts.

MIDAS,
BatchMake
,
Condor
Grid;
built
using
Zend

on PHP.

caB2B,
NBIA,
PODS data
elements,
DICOM
query tools.

QIBO, AIM,

RadLex
/
Snomed
/
NCIt
; built
using Ruby
on Rails.


Specify context for
use and assay
methods.


Use consensus terms
in doing so.

Specify


Assemble applicable
reference data sets.


Include both imaging
and non
-
imaging
clinical data.

Formulate


Compose and iterate
batch analyses on
reference data.


Accumulate
quantitative read
-
outs
for analysis.

Execute


Characterize the
method relative to
intended use.


Apply the existing
tools and/or extend
them.

Analyze


Compile evidence for
regulatory filings.


Use standards in
transfer to regulatory
agencies.

Package

18

18

MVT
:
Reasonable framework, but many
gaps

19

19

There are multiple possibilities to deploy it as a web application, some
of which we’ve considered:

1.
Re
-
implement the existing implementation to use GWT in place
of Swing, inclusive of both the
XIPHost

as well as MVT
components, retaining the WG23 concept.

2.
Re
-
implement only those parts necessary to perform the
needed MVT functions using GWT with enough data handling
to do so but without doing everything necessary to retain the
full
XIPHost

capability.

3.
Leverage the GUI design concept but otherwise implement
without starting from the Swing code.



In all cases, there is the secondary design alternative of introducing a
RESTful

web service layer explicitly or not.



(By the way, just for fun, I performed a conversion of the current Swing
code to Ajax using
AjaxSwing
.


I got most of AVT working over the web
with minimal effort, but this isn’t a serious alternative because
AjaxSwing

has a license fee.


I did it because I wanted to see how easy
such a path would be.


It’s an interesting capability! But irrelevant in the
end.)

Pros: optimized for DICOM, works with
workstations

Cons: hard to create web apps, not optimized for
semantic web

20

20

HW

XIP

Application

Inventor

Application Modules

WG 23 System Services PLUG

WG 23 System Services SOCKET

GRID

CLIENT

SERVICES

DICOM

SERVICES

(DCMTK)

OTHER

SERVICES

VTK

ITK

AIMTK

other

OS

NCIA

XIP

IDE

RadLex

AIM

NCI

Protégé

EVS

XIP

MIDDLEWARE

DICOM

DICOM Services

IVI Middleware

caGrid

CaBIG

caDSR, EVS, RadLex, AIM ontology, etc

Client access

Service access

Grid Data
Service

Grid Analytical
Service

AIM Data
Service

XIP App

Service

Host

WG23

DICOM
Image
Sources

20

20

Alternative architectural form…

21

SW Stack


J2SE (J2EE compliant)


MySQL


caGrid


Globus


Application:


JBoss


caCore

21

21

21

With pros and cons “opposite” that of the XIP
based architecture

Functionality view annotated with
architecture

22

22

HW

XIP

Application

Inventor

Application Modules

WG 23 System Services PLUG

WG 23 System Services SOCKET

GRID

CLIENT

SERVICES

DICOM

SERVICES

(DCMTK)

OTHER

SERVICES

VTK

ITK

AIMTK

other

OS

NCIA

XIP

IDE

RadLex

AIM

NCI

Protégé

EVS

XIP

MIDDLEWARE

DICOM

DICOM Services

IVI Middleware

caGrid

CaBIG

caDSR, EVS, RadLex, AIM
ontology, etc

Client access

Service access

Grid
Data
Servi
ce

Grid
Analytica
l Service

AI
M
Dat
a
Ser
vic
e

XIP
Ap
p

Ser
vic
e

Ho
st

W
G2
3

DIC
OM
Imag
e
Sour
ces

22

22

MIDAS Core

Apache

File System

PostGreSQL

Publication
DB

MIDAS Data
Server

MIDAS e
-
journal

MIDAS Compute
Server

MIDAS
Visualization

MIDAS Client

MIDAS C++ API

MIDAS Web API

When annotation and markup has already been done

AIM
-
enabled (e.g.,
ClearCanvas
)
workstation

RIS
worklist

items

DICOM Q/R

First step to rationalizing architecture: mash
them together and see what falls out

23

23

23

23

NCIA

RadLex

AIM

NCI

XIP

MIDDLEWARE

DICOM

DICOM Services

IVI Middleware

SADI framework (e.g., wrapped
caGrid
)

CaBIG

caDSR, EVS, RadLex, AIM ontology, etc

Client access

Service access

Grid Data
Service

Grid Analytical
Service

AIM Data
Service

HW

XIP

Application

Inventor

Application Modules

WG 23 System Services PLUG

WG 23 System Services SOCKET

GRID

CLIENT

SERVICES

DICOM

SERVICES

(DCMTK)

OTHER

SERVICES

VTK

ITK

AIMTK

other

OS

XIP

IDE

Protégé

EVS

XIP App

Service

Host

WG23

DICOM
Image
Sources

This is an ongoing discussion. More to come!

24

24

Value proposition of QI
-
Bench


Efficiently collect and exploit evidence establishing
standards for optimized quantitative imaging:


Users want confidence in the read
-
outs


Pharma

wants to use them as endpoints


Device/SW companies want to market products that produce them
without huge costs


Public wants to trust the decisions that they contribute to


By providing a verification framework to develop
precompetitive specifications and support test
harnesses to curate and utilize reference data


Doing so as an accessible and open resource facilitates
collaboration among diverse stakeholders

25

25

Summary:

QI
-
Bench Contributions


We make it practical to increase the magnitude of data for increased
statistical significance.


We provide practical means to grapple with massive data sets.


We address the problem of efficient use of resources to assess limits of
generalizability
.


We make formal specification accessible to diverse groups of experts that are
not skilled or interested in knowledge engineering.


We map both medical as well as technical domain expertise into
representations well suited to emerging capabilities of the semantic web.


We enable a mechanism to assess compliance with standards or
requirements within specific contexts for use.


We take a “toolbox” approach to statistical analysis.


We provide the capability in a manner which is accessible to varying levels of
collaborative models, from individual companies or institutions to larger
consortia or public
-
private partnerships to fully open public access.

26

26

QI
-
Bench

Structure / Acknowledgements


Prime: BBMSC
(Andrew Buckler, Gary Wernsing, Mike Sperling, Matt Ouellette)


Co
-
Investigators


Kitware

(Rick Avila, Patrick Reynolds,
Julien

Jomier
, Mike Grauer)


Stanford (David Paik, Tiffany Ting Liu)


Financial support as well as technical content: NIST
(Mary Brady, Alden Dima, Guillaume
Radde
)


Collaborators / Colleagues / Idea Contributors


FDA (Nick Petrick, Marios Gavrielides)


UCLA (Grace Kim)


UMD (Eliot Siegel, Joe Chen, Ganesh Saiprasad)


VUmc

(Otto Hoekstra)


Northwestern (Pat Mongkolwat)


Georgetown (Baris Suzek)


Industry


Pharma
: Novartis (Stefan Baumann), Merck (Richard Baumgartner)


Device/Software:
Definiens

(Maria Athelogou),
Claron

Technologies (Ingmar Bitter)


Coordinating Programs


RSNA QIBA (e.g., Dan Sullivan, Binsheng Zhao)


Under consideration: CTMM
TraIT

(Andre Dekker,
Jeroen

Belien
)

27

27