Deployment of a Gridbased Medical Imaging Application

HealthGrid05 Conference, Oxford UK April 2005 Richard McClatchey MammoGrid Technical Coordinator, UWE

Contents
MammoGrid challenges Approach and technolgies adopted MammoGrid prototyping Services and Grid solutions Deployment and future work Lessons learned and conclusions
7th April 2005 HealthGrid'05 Conference, Oxford The MammoGrid Collaboration 2

About Mammogrid
EU FP5 funded project, start: 2002, end:2005

To build a grid-powered pan-european mammography database and evaluate its use in a clinical environment
Participants:
Medical Institutions: Udine University Hospital; Addenbrookes Hospital, Cambridge; Oxford University Academic Institutions, commercial partners: UWE, CERN, Mirada Solutions Ltd, Pisa, Sassari
HealthGrid'05 Conference, Oxford The MammoGrid Collaboration 3

7th April 2005

Key Objectives:
Acquisition of large sample of mammograms Standardization of mammograms Annotation of mammograms by humans as well as CADe software Distributed data management system, crossinstitute, cross-country queries Sharing of computing resources for the purpose of optimizing data storage and execution of computing-intensive algorithms Proof of concept with active clinical participation
7th April 2005 HealthGrid'05 Conference, Oxford The MammoGrid Collaboration 4

Technical Challenges
Make Grid work in practice and usable in hospitals (execute clinical queries, run SMF & CADe, share diagnoses) Provide a distributed and federated clinical data management system Deliver a secure system that could be integrated into a hospital information system Investigate how the medical application can be isolated from the (still evolving) Grid as new Grid flavours emerge (e.g. OGSI->Web Services/Grid Services). Provide persistency for data models and meta-data models to handle heterogeneity
7th April 2005 HealthGrid'05 Conference, Oxford The MammoGrid Collaboration 5

Grid challenges: interoperability

Legal restrictions on access to data Clinicians, researchers, developers, Govt, Data resides in hospitals Existing systems, firewall protected Combining several databases Heterogeneity Secure file transfer essential Large images to be transferred Approx 100Mb per patient APIs needed for third party applications
7th April 2005 HealthGrid'05 Conference, Oxford The MammoGrid Collaboration 6

Contents
MammoGrid challenges Approach and technolgies adopted MammoGrid prototyping Services and Grid solutions Deployment and future work Lessons learned and conclusions
7th April 2005 HealthGrid'05 Conference, Oxford The MammoGrid Collaboration 7

Apply emerging GRID technology rather than develop it. A lightweight GRID, study its usage in hospitals A prototype federated database of mammograms in hospitals in the UK and Italy It will investigate : meta-data for resolving queries standardised mammograms to resolve image variability Health data security using a novel Grid box the infrastructure needed for CADe It will provide feedback from the Hospital community To inform the next generation of HealthGrids
7th April 2005 HealthGrid'05 Conference, Oxford The MammoGrid Collaboration 8

Implementation Approach

Steps in Project

EDG AliEn
AliEn Stack of glite

EGEE

MammoGrid

P1

P1

P1.5

P1.5

P2

Centralised DB Distributed DB Web services (Oct 2003) (April 2004) (Nov 2004)

GRID-enabled Multiple VOs services (Post MammoGrid)

Time
7th April 2005 HealthGrid'05 Conference, Oxford

we are here
The MammoGrid Collaboration 9

Underpinning technologies
Distributed computations
Web Services (SOAP, XML,WSDL); Service-Oriented Architecture (SOA); AliEn Grid Middleware EGEE gLite, the emerging Grids standard DICOM:

Mammography

SMF:

DICOM files are transferred over SCP-SCU; DICOM files are stored as immutable objects. Standardized Mammography Form (Mirada) Computer-Aided Detection algorithms (CALMA)
HealthGrid'05 Conference, Oxford The MammoGrid Collaboration 10

CADe:

7th April 2005

Contents
MammoGrid challenges Approach and technolgies adopted MammoGrid prototyping Services and Grid solutions Deployment and future work Lessons learned and conclusions
7th April 2005 HealthGrid'05 Conference, Oxford The MammoGrid Collaboration 11

Mammogrid Prototypes
Prototype 1.0
No metadata Centralized data Patient data stored alongside files Services are instantiated inside AliEn

Prototype 1.5
Decentralized metadata and data File storage separated from semi-structured data Query semantics Query Performance Portal and QueryManager: static webservices gLite interface: static, insecure webservices

Prototype 2.0
GAS factory to secure gLite UI Portal and QueryManager: proper grid services (OGSI) [Authentication to gLite via delegation] Query optimization As time/resources permit

7th April 2005

HealthGrid'05 Conference, Oxford

The MammoGrid Collaboration

12

MII: P1.5 Overview

Oxford Node
Scanner MAS MWS MG Services Add Dicom File Query Data Retrieve Dicom File Job Execution Local gLite Services Storage Element Computing Element File Transfer Monitoring Central gLite Services Authentication Database Proxy Transfer Scheduler Job Scheduler Grid data File Catalogue gLite Repository

Clinician

Cambridge Node

MG data MG meta-data Cambridge Repository

Virtual Repository

MG data MG meta-data Udine Repository Scanner MAS MWS MG Services Add Dicom File Query Data Retrieve Dicom File Job Execution

Clinician

Local gLite Services Storage Element Computing Element File Transfer Monitoring

Udine 7th April 2005 Node

HealthGrid'05 Conference, Oxford

The MammoGrid Collaboration

13

What has been delivered (P1.5)

A set of tested and configured Gridboxes deployed at the hospitals. Near-final version of the Mirada Acquisition System and Mammogram Workstation Successful implementation and testing of MammoGrid Services across deployed sites SMF & CADe algorithms deployed, tested (SMF & CADe compatibility) P1.5 gLite/AliEn-based, distributed database system Web-service implementation with persistently stored data and meta-data models Grid for job and file distribution, database outside the Grid middleware Test VO at CERN: gLite-based, MG services are Grid services
7th April 2005 HealthGrid'05 Conference, Oxford The MammoGrid Collaboration 14

Gridboxes
Design, construction and delivery completed by subcontractor All units tested and deployed In use at Udine, Oxford Univ, CERN and Cambridge Connected to the Grid and available for clinical evaluation.

7th April 2005

HealthGrid'05 Conference, Oxford

The MammoGrid Collaboration

15

MammoGrid Node
MammoGrid deployed in Hospitals & running Acquisition Station Interface of clinicians w/ system

Gridbox

built & tested hosts Grid engine and MG medical services

Grid

7th April 2005

HealthGrid'05 Conference, Oxford

The MammoGrid Collaboration

16

Cambridge Clinical Site

7th April 2005

HealthGrid'05 Conference, Oxford

The MammoGrid Collaboration

17

Contents
MammoGrid challenges Approach and technolgies adopted MammoGrid prototyping Services and Grid solutions Deployment and future work Lessons learned and conclusions
7th April 2005 HealthGrid'05 Conference, Oxford The MammoGrid Collaboration 18

Mammogrid Services
GridBox to Client: Portal DICOM SCP HTTP(G) (retrieve) GridBox to peers: Query Manager SE CE FTD Central to GridBox: LDAP (configure) Proxy (Database) TransferManager JobManager Optimizers
7th April 2005
DICOM SCU/SCP

DICOM SCP Data

SOAP over HTTPG PORTAL

QUERY MANAGER

gLite/AliEn Services

CLIENT

HTTPG/HTTP

GRIDBOX

QUERY MANAGER

gLite/AliEn Services

CENTRAL SERVICES LDAP

PORTAL

DICOM SCP Data

GRIDBOX

FILE CATALOGUE

CERN

HealthGrid'05 Conference, Oxford

The MammoGrid Collaboration

19

Data Design Schema

Single Entry point: Portal Distributed patient and pathology data Distributed metadata (data description)
gLite CENTRAL
PORTAL MAMMO META MAMMO DATA PORTAL MAMMO META MAMMO DATA

SE

SE

Data and schema autonomy (wrappers, mediators) Centralized File Catalogue

SE

FILE CATALOGUE

SE

Distributed File Repository (Storage Elements)

MAMMO META MAMMO DATA PORTAL

MAMMO META MAMMO DATA PORTAL

7th April 2005

HealthGrid'05 Conference, Oxford

The MammoGrid Collaboration

20

Example: add new Mammogram

1. 2. 3. 4. 5. 6. Clinician Scans Mammogram Client Sends DICOM file to SCP buffer in Gridbox Client contacts portal, requests add()
6.
5.

MAMMO WORK STATION

2.

DICOM SCP

3. add()

PORTAL

4.

Portal contacts gLite, inserts DICOM file into catalogue Portal reads Mammogrid metadata structure
Scanner

File Catalogue, SE

1.

MAMMOGRID META-DATA

MAMMOGRID DATA

Portal sends data to DBBackend (XML), data goes to MySQL DB

7th April 2005 HealthGrid'05 Conference, Oxford The MammoGrid Collaboration 21

Example: query
1. Clinician constructs query using GUI 2. Client sends query XML to Portal 3. Portal sends query to a Query Manager 4. Query Manager plans query distribution 5. Query Manager distributes the Query 6. Query Handlers receive the query 7. Query Handler resolves against metadata DB 8. Query Handler executes query on the Mammogrid DB (wrappers, mediators omitted)
7th April 2005

AGE > 50

1. query()

PORTAL

QUERY HANDLER
5.

2.

PROBABILITY OF CANCER > 80%

4.

MAMMO WORK STATION

FINDING = MASS MALIGNACY=HIGH

MAMMOGRID META-DATA

QUERY MANAGER

MAMMOGRID DATA

3.

QUERY MANAGER

3.

QUERY MANAGER

HealthGrid'05 Conference, Oxford

The MammoGrid Collaboration

6.

22

gLite/AliEn services overview

PORTAL

SOAP HTTP

gLite/AliEn CENTRAL SERVICES

mgadmin MG Interface UI udine udine MG CA certificate

admin

FTD SSH key mgprod AliEn::Service CE/ClusterMonitor SSH key mgprod AliEn::Service::Interface SE aliprod Roles:
admin aliprod udine cambridge oxford

aliprod

AliEn::UI

7th April 2005

HealthGrid'05 Conference, Oxford

(HTTP)

+ls() +whereis() +rm() +submit() +mkdir() +mirror() +get()

SSH key mgprod aliprod

gLite/AliEn site services

People:
mgadmin mgprod (each site ) udine cambridge oxford

The MammoGrid Collaboration

23

Algorithms, Job Execn.

Algorithms are deployed centrally Presently two different kinds of algorithms: SMF: Mammogram standardization DICOM image into DICOM image CADe Feature detection (calcification, mass) DICOM image into text output (later into DICOM SR) Workflow: User selects from available algorithms User selects input files (usually result of a query) User specifies additional algorithm parameters Portal creates jdl, submits job (batch, splitting) Monitoring of job status User may examine stdout, stderr, view and register resulting DICOM Jobs running on grid do not access the database
7th April 2005 HealthGrid'05 Conference, Oxford The MammoGrid Collaboration 24

Mammogrid and gLite

Data Management:
Semi-structured data management is limited, nonperformant. For our purposes data management is provided by a separate layer Links between file-based and semi-structured data: LFNs Unacceptable performance for generic use-cases (1 min for actual transfer with ~5 min overhead) A method installed to bypass FTD transfers Problems with splitting (mostly resolved) Job status reporting not reliable
HealthGrid'05 Conference, Oxford The MammoGrid Collaboration 25

File Transfers:

Job Execution:

7th April 2005

Contents
MammoGrid challenges Approach and technolgies adopted MammoGrid prototyping Services and Grid solutions Deployment and future work Lessons learned and conclusions
7th April 2005 HealthGrid'05 Conference, Oxford The MammoGrid Collaboration 26

MammoGrid solution
Service-Oriented Infrastructure that federates autonomous mammogram databases on top of a Grid middleware.
Medical imaging (MI) service layer:
Generic services for handling image-related data (parsing, transforming, storing etc); Specific imaging services (image analysis, query services etc.) Mediates between MI services and underlying Grid middleware; Examples: storing/retrieving files on the Grid, authentication, job submission and execution

Clients

SOA
Medical-Image Services Grid-aware Services

Grid-aware service layer:

Grid Middleware e.g. gLite

The MammoGrid Collaboration 27

7th April 2005

HealthGrid'05 Conference, Oxford

Prototype Mammography Database

Addenbrooks Hospital, Cambridge
IDC

MAMMOGRID VO
Oxford University Udine University Hospital

7th April 2005

HealthGrid'05 Conference, Oxford

The MammoGrid Collaboration

28

Mammogrid VO status
Production P1.5 deployed end of 2004 with empty data store P1.5 demonstrated at the 2004 EU Annual Technical Review Presently ~2000 images uploaded from previously acquired data (~20GB) Batch upload (and batch SMF) is ongoing, live acquisition started Clinical trials are starting. Final activities: bugfixing, clinical trials, dissemination, commercialization
7th April 2005 HealthGrid'05 Conference, Oxford The MammoGrid Collaboration 29

Contents
MammoGrid challenges Approach and technolgies adopted MammoGrid prototyping Services and Grid solutions Deployment and future work Lessons learned and conclusions
7th April 2005 HealthGrid'05 Conference, Oxford The MammoGrid Collaboration 30

MammoGrid Future Plans

P1.5 Architecture delivered & deployed
P1.5 already present in Oxford and CERN Final deployment in Cambridge & Udine hospitals 2004-2005 testing phase

P2 Architecture under development (resources limited)

New Services to provide enhanced security and confidentiality legal and ethical conformance Facility to add further image and non-image handling services MammoGrid dedicated Meta-Data Database heterogeneous schema discovery / query services

Framework 6 project(s) to research biomedicine See http://mammogrid.vitamib.com

7th April 2005 HealthGrid'05 Conference, Oxford The MammoGrid Collaboration 31

Lessons Learned - Grids

GRID is new : new skills required and few with them. GRID is slow : therefore initially concentrate on
functionality rather than performance.

GRID is changing : have to hit a moving target. GRID is complex : many layers of software, developed
by different groups.

GRID is still insecure / unreliable / over-hyped GRID is multi-flavoured, needs standards

7th April 2005 HealthGrid'05 Conference, Oxford The MammoGrid Collaboration 32

Lessons for HealthGrids

Clinician community is highly risk-averse Medical data an ethical and legal minefield Grid software is still highly volatile Companies not yet ready to invest in Healthgrids No real cross-over with bio/genetic information Research needed in biomedical applications with companies (recent FP6 call?) Progress on Grid standardisation essential to enable out-of-the-box capability and stability Ethical issues must be resolved before we get clinician and company buy-in.
7th April 2005 HealthGrid'05 Conference, Oxford The MammoGrid Collaboration 33

So