Joint Vienna Institute August 5 - 16, 2013 L-6: Processing System Setup Reproductions of this material or any parts of it should refer to the IMF Statistics Department as the source Lecture Outline Processing system: guidelines Processing system: guidelines Software Database-based computer system Use of spreadsheet JVI/QNA/L6 : 2 IMF Statistics Department 2 Processing system: guidelines Time series oriented Time series oriented Data input separate from calculations Annual data separate from quarterly data JVI/QNA/L6 : 3 IMF Statistics Department Processing system: guidelines Clear documentation of each step of the Clear documentation of each step of the compilation process Adjustments documented By electronic/hard-copy notes By using explicit adjustment factors Data input in original form Li ki d f i i id ifi bl d Linking and re-referencing in separate identifiable and reproducible steps JVI/QNA/L6 : 4 IMF Statistics Department 3 Processing system: guidelines Standardized set up across all areas Standardized set up across all areas Same format Same organization Full classification lists Allow empty cells JVI/QNA/L6 : 5 IMF Statistics Department Software No ready made QNA compilation software exist y Q p Spreadsheets are good for small systems and experimental work Time-series databases best for storage of large volumes of data Large compilation systems: Programs for batch handling of data stored in time series databases Procedures for interactive handling of data stored in time series databases JVI/QNA/L6 : 6 IMF Statistics Department 4 Software Two groups of commercially available database software Time series databases Fame, Dbank, Aremos, Modeleasy Treat all data objects (data arrays or data vectors) as time series Relational database packages Access, oracle, Sysbase, dbase, paradox Specialized for cross-sectional operations A custom-made interface to the database may be needed, to ease data exchange between the database and other software packages, as well as smaller tailor-made compilation modules JVI/QNA/L6 : 7 IMF Statistics Department Database-based computer system: Elements Databases for data input Databases for data input Databases for storage of monthly, quarterly and annual source data Database for storage of annual national accounts estimates Databases for storage of annual source data JVI/QNA/L6 : 8 IMF Statistics Department 5 Database-based computer system: Elements Compilation routines p Benchmarking of indicators to annual data Deflation / reflation Source data assessment procedures tracking on a detailed level, editing Compilation system assessment procedures simulations on historical data/tracking on an aggregated level agg egated e e Reconciliation/comparison of GDP estimates from the production, expenditure, and income sides Seasonal adjustment (link to X-12-Arima or Tramo- Seats) JVI/QNA/L6 : 9 IMF Statistics Department Database-based computer system: Elements Databases for storage of compiled QNA data Databases for storage of compiled QNA data Database(s) for official published data Archived copies of previous quarters published data Working databases for unpublished estimates JVI/QNA/L6 : 10 IMF Statistics Department 6 Database-based computer system: Elements Standardized naming conventions for time series Standardized naming conventions for time series identifiers capturing all dimensions of the data- room Structure that is easy to understand Follows the classification system Shows the type of data - frequency, value/price index, Shows the stage of processing, etc. JVI/QNA/L6 : 11 IMF Statistics Department Organization of spreadsheets: things to consider Name in a meaningful way Name in a meaningful way Store where they can be easily found Date whenever revisions are made Fully document General statement of what is contained in the file and the data sources used the data sources used More detailed statement on any assumptions made and any adjustments made to the data. JVI/QNA/L6 : 12 IMF Statistics Department 7 Organization of spreadsheets: things to consider Titles and headings within a file Titles and headings within a file Tables should have proper titles to indicate the contents, with units clearly stated Columns should be clearly headed, again with units clearly stated Give each layer a sensible name - not sheet1, sheet2 etc JVI/QNA/L6 : 13 IMF Statistics Department Organization of spreadsheets: things to consider References and formulas within a file References and formulas within a file All numbers imported from outside should have a clear statement of the source Any numbers imported from outside should be entered only once and thereafter be linked to other cells by formula formula All formula must be double checked for accuracy Check that they actually do what they are supposed to do JVI/QNA/L6 : 14 IMF Statistics Department 8 Organization of spreadsheets: things to consider Spreadsheet layout within a file p y Use different colors for input and output data Whole time series should always be visible, not one year's data Use split/freeze frames to make the row column heading visible. Standardize the layout across all areas so it can be read and understood by someone without assistance Use comment facility in excel to explain derivation adjustments Use comment facility in excel to explain derivation, adjustments Today() function to put date on printouts Use header/footer to put date and filename New folders for archived material No formatting until you are ready to tabulate JVI/QNA/L6 : 15 IMF Statistics Department Organization of spreadsheets: things to consider Data management within a file Data management within a file Each edition of compilation should be on a separately named spreadsheet, to have an archived version (if something goes wrong or to follow through revisions) Use the multiple layers of spreadsheet to separate out different processes Make a convention that the time dimension is either Make a convention that the time dimension is either always across the rows or down the columns Make your system rebase-friendly by identifying every price or volume index as BY=100 and in a set color JVI/QNA/L6 : 16 IMF Statistics Department