CORE ESSnet (COmmon Reference Environment) progress report

Carlo Vaccari Istat - Italy

MSIS Meeting - Luxembourg May 23-25 2011

Outline
Introduction and history CORE objectives CORE where we are
Architecture implementation Information model

CORE and SDMX CORE and GSIM

MSIS Meeting - Luxembourg May 23-25 2011 2

CORA ESSnet
Financed by Eurostat under 2009 Statistical Workprogramme Countries involved: it (coordinator), ch, dk, lv, nl, no, se Duration: October 2009 - October 2010
MSIS Meeting - Luxembourg May 23-25 2011 3

CORA Technical Architecture CORA Model: two dimensions

Functional dimension Construction dimension

Functional dimension
Adoption of GSBPM 4.0 9 subprocesses of level 2
1 Specify Needs 2 Design 3 Build 4 Collect 5 Process 6 Analyse 7 Disseminate 8 Archive 9 Evaluate

MSIS Meeting - Luxembourg May 23-25 2011

Construction Dimension: Layers

Figures Time Series Statistic Population Unit Variable Value A domain of interest documented by statistical products Statistical series over time Integrated or simple statistical product for a given time A population at a given time A statistical unit at a given time A statistical variable at a given time A logical representation of the value of a variable

MSIS Meeting - Luxembourg May 23-25 2011

CORA Model Grid

Statistical processes compliant to CORA model are intended to be designed by statisticians

MSIS Meeting - Luxembourg May 23-25 2011 6

After CORACORE!
COmmon Reference Environment (CORE), financed by Eurostat under 2010 Statistical Workprogramme Countries involved: it (coordinator), fr, nl, no, pt, se Duration: December 2010 - January 2012

MSIS Meeting - Luxembourg May 23-25 2011

CORE Principal Outcomes

Environment for the definition and execution of statistical processes
Definition of a process in terms of services selected from an available repository Execution of the composed workflow
MSIS Meeting - Luxembourg May 23-25 2011 8

CORE Outcomes: Design

CORA model CORA information model Design of CORE services and processes

MSIS Meeting - Luxembourg May 23-25 2011

CORE Outcomes: Implementation

Selection of available middleware solutions for process execution Realization of an environment able to permit the execution of processes:
Interfaces (GUIs) for defining CORE processes for statistical users Integration APIs Repository of integration layers
MSIS Meeting - Luxembourg May 23-25 2011 10

CORE Outcomes: Testing

Realization of processes starting from services implementing some GSBPM phase Evaluation of costs related to integration Prototype implementation (to be engineered)

MSIS Meeting - Luxembourg May 23-25 2011

11

CORE Architecture (1)

GUIs to support modelling of CORE processes according to the CORA grid
Modeling & control flow constructs Drag & drop facilities for process design Global schema Implementation: we are evaluating the usage of an open process editor tool (Oryx http://bpt.hpi.uni-potsdam.de/Oryx/WebHome )
MSIS Meeting - Luxembourg May 23-25 2011 12

CORE Architecture (2)

Process Runtime Controlled execution of services Implementation: integration of existing workflow solutions, currently in evaluation phase Service Runtime Integration APIs (in-out data transformation) Service execution Implementation: CSV and SQL data transformations are currently being implemented Service Repository Deployment of services
MSIS Meeting - Luxembourg May 23-25 2011 13

CORE Information Model (1) First draft of CORE information model

Del. 2.1 released: requirements for the model of the interface through which statistical services will communicate Information Model to be released, currently in discussion phase

MSIS Meeting - Luxembourg May 23-25 2011

14

CORE Information Model (2) Design Principles (in discussion):

Rectangular data sets (rows & columns) Strong typing (data, rules, parameters) Dataset kinds (eg micro/aggreg) Free-style arguments (eg scripts tool dependent) Other (service arguments and infos)
MSIS Meeting - Luxembourg May 23-25 2011 15

CORE and SDMX

Both initiatives foster standardization CORE
Focus on standardization of processes and data exchanges (mainly) intra-NSI

SDMX
Focus on standardization of processes and data exchanges (mainly) inter-NSIs (or between NSIs and international organizations)
MSIS Meeting - Luxembourg May 23-25 2011 16

CORE and SDMX - 2

CORE
Focus on all phases of statistical processes Both micro and macro data considered

SDMX
Focus (mainly) on dissemination phase Mainly macro data considered

MSIS Meeting - Luxembourg May 23-25 2011

17

Information Model
Both propose an information model CORE information model
Takes explicitly process dimension into account through GSBPM Data dimension

SDMX information model

Mainly focused on data dimension
MSIS Meeting - Luxembourg May 23-25 2011 18

CORA Information Model

Service
1 n

+belongs_to Layer level +contains +belongs_to +has Element

n

+implements

Constructor
prescript

+input
n output.belongs_to.level = input.belongs_to.level + 1

+output n Construct +represented_by

1

Object

Figure

Time series

Statistic

Population

Unit

Variable

MSIS Meeting - Luxembourg May 23-25 2011

19

SDMX Data & Metadata Information Model

Data or Metadata Structure Definition
uses specific data or metadata structure

Data or Metadata Set

conforms to business rules of the data or metadata flow

Data or Metadata Flow

can be linked with categories from multiple category schemes

can have child categories

Category

can get data from multiple data providers

comprised of subject or reporting categories

Data Provider

Provision Agreement

Category Scheme

MSIS Meeting - Luxembourg May 23-25 2011

20

On Information Models
Different abstraction levels CORE
Higher modelling level E.g.: statistics as tabular data

SDMX
Lower modeling level E.g.: aggregated data set with dimensions, attributes and measures
MSIS Meeting - Luxembourg May 23-25 2011 21

Open Issues - 1
Can we use SDMX for micro and macro data exchanges in a CORE process?
Need for mapping of information models

MSIS Meeting - Luxembourg May 23-25 2011

22

On-Going Work - 1
CORE implementation scenario within Istat Main phases: Sample selection and allocation CORE wrapping of available SAS and R procedures
MSIS Meeting - Luxembourg May 23-25 2011 23

On-Going Work - 2
Design and implementation of CORE Integration APIs Possible in/out SDMX translations
CORE TOOL

SDMX

IAPI

TOOL

IAPI

SDMX

MSIS Meeting - Luxembourg May 23-25 2011

24

Open Issues - 2
What about metadata?
CORE: Data and metadata managed in the same way SDMX:
Distinction between structural metadata and reference metadata Dedicated effort for metadata management
MSIS Meeting - Luxembourg May 23-25 2011 25

Collaboration between CORE/SDMX ESSnets

CORE planned deliverable on Feedbacks on SDMX Usage in CORE Periodical meetings inside Istat between coordinators of the two ESSnets Exchanges of resources between the two ESSnets

MSIS Meeting - Luxembourg May 23-25 2011

26

CORE and GSIM

GSIM: Generic Statistical Information Model deliverable from OCMIMF Operationalising a Common Metadata/Information Management Framework activity inside Statistical Network Ambiguity on the acronym: reference to generic statistical information model in CORE ESSnet proposal In March started activity to clarify relationships (thanks to J.P.Kent and A.Hamilton)
MSIS Meeting - Luxembourg May 23-25 2011 27

CORE and GSIM

First analysis and discussions: the deliverables from the two initiatives are complementary in intent and do not overlap in concept

Necessary to avoid gaps and/or duplications and ensure the complementary relationship

MSIS Meeting - Luxembourg May 23-25 2011

28

CORE Information Model

CORE will define a very generic information model (CORE-IM) for the interface through which statistical services will communicate with each other within the framework of the CORA model As a communication protocol, CORE-IM focuses on the postal envelope used when passing information between services, rather than focusing in detail on the information being communicated (ie what is inside)
MSIS Meeting - Luxembourg May 23-25 2011 29

CORE and GSIM

CORE-IM current hypothesis: to support a flag to indicate if the information being communicated is described within GSIM without claiming to align the semantics of the content (eg, classification), but only to alert a consuming service which understands GSIM that it can relate the content to GSIM

MSIS Meeting - Luxembourg May 23-25 2011

30

Complementary nature
CORE-IM supporting semantic interoperability at a very high, abstract level (here is an information object, along with the envelope information about it) where GSIM can provide greater semantic precision to a subset of information objects communicated using CORE CORE supporting communication between services substantial interoperability benefits Information aligned with GSIM semantics
MSIS further level of interoperability Meeting - Luxembourg May 23-25 2011 31

Complementary nature
GSBPM reference model for statistical business processes GSIM reference model for information input to, used by and produced by those processes Models are independent it's possible to use one without the other CORE-IM recognizes and uses GSBPM and (hopefully) will do the same with regard to GSIM, giving them a potential contact point
MSIS Meeting - Luxembourg May 23-25 2011 32

Coordination in practice
Need to maximize the extent to which these synergies are achieved in practice: Members in common (no, se) ABS leader for the OCMIMF and observer in CORE CORE members external reviewers for GSIM material CORE WP2 co-ordination input from the OCMIMF collaboration team in regard to deliverables Half day session at METIS workshop (October) presenting CORE and OCMIMF works to external metadata specialists Common documents in preparation ...
MSIS Meeting - Luxembourg May 23-25 2011 33