CASA Developers Guide

Justo Gonzalez, ESO, 9-12-2016

Scope of CASA

 CASA (Common Astronomy Software Applications) is a suite of C++/Python SW components

used in radio astronomy. NRAO (National Radio Astronomy Observatory) has the lead for
CASA, and the CASA group lead is Jeff Kern (See https://science.nrao.edu/enews/casa_001/).

 NRAO uses CASA to support ALMA and EVLA. ESO contributes with 2FTEs as stakeholder
for ALMA for development and maintenance of CASA.

 CASA has a core (casacore) which is used by various radio-telescopes (e.g. LOFAR, SKA,
ALMA, EVLA), and modifications to it have to be compatible with all of them.

 ESO has contributed in the following areas of CASA during the last few years:
 Flagging (i.e. mark bad data)
 Transformation of visibilities prior to imaging
 Parallelization Framework (MPI)

 Useful links:
 CASA Homepage (NRAO): https://casa.nrao.edu/
 CASA Cookbook: https://casa.nrao.edu/docs/cookbook/

High-Level architecture of CASA

There are 4 well defined levels of components in CASA. It is possible to inspect the code as reading
this guide using web-browser access to the CASA svn repository:

https://svn.cv.nrao.edu/svn/casa/trunk/

 casacore: Set of low level C++ libraries including:

 CASA containers (similar to c++ standard library)

 I/O and iterator support for the Measurement Set (MS) format
 Low level science-related code (coordinates, measures, fitting, images, …)
 Useful links:
 ASTRON casacore docs: http://casacore.github.io/casacore/
 CASA MS format: https://casa.nrao.edu/Memos/229.html
  code: Set of mid level C++ libraries, including the following areas (under maintenance of ESO)

 flagging: Set of flagging agents to flagdata covering various algorithms like clip, shadow, or
RFI detection. It consists of one top level orchestrator, and separated classes (agents) for
each flagging algorithm and the I/O access layer.

 code/flagging/Flagging
 AgentFlagger: Top level orchestration with state-like methods like open, configure,
run, done..
 FlagDataHandler, FlagMSHandler, FlagCalTableHandler: I/O layer which basically
relies on the Visibility Iterator / Visibility Buffer interface:
 code/msvis/VisBuffer2.h
 code/msvis/VisibilityIterator2.h
 FlagAgentBase,
FlagAgent<Manual/Clip/Shadow/Extention/Quack/Summary/Rflag>: Set of
flagging agents to flagdata covering various algorithms like clip, shadow, or RFI
detection.

 Flagdata useful links:

 Urvashi Rao (NRAO) html pages on flagdata
http://www.aoc.nrao.edu/~rurvashi/FlaggerDocs/FlaggerDocs.html
 CASA flagging cookbook pages
https://casa.nrao.edu/docs/cookbook/casa_cookbook004.html#sec178

 mstransform: Common framework to perform transformations in visibility data such as time

averaging, channel averaging, hanning smooth, re-gridding, reference frame transformation,
Spectral Window combination/separation or simply data splitting. There are two separated
areas as this module is going trough a refactoring:

 code/mstransform/MSTransform: Old design (although currently in use) where there are

only 3 main classes involved:
 MSTransform: Top level orchestration with state-like methods like open, configure,
run, done.
 MSTransformManager: Main class implementing all data transformations
 MSTransformDataHandler: I/O handling class implementing methods to create the
basic structure of a Measurement Set, and modify subtables as necessary.
 MSTransformRegridder: Class containing frequency (channel) re-gridding
algorithms.

 code/mstransform/TVI: New design where each transformation is encapsulated in a

“Transform Visibility Iterator”. For each TVI there is a Google unit test framework
(Gtest) that verifies the output of each TVI vs the old framework
 <ChannelAverage/Convolution/HanningSmooth/PhaseShifting/Regridding>TVI
 test/t<ChannelAverage/Convolution/HanningSmooth/...>TVI
  mstransform useful links:
 Sandra Castro html pages on mstransform
http://www.eso.org/~scastro/ALMA/casa/MST/MSTransformDocs/MSTransformDocs.html
 CASA cookbook mstransform pages
https://casa.nrao.edu/docs/cookbook/casa_cookbook005.html#sec271
 MSTransform original design document
http://www.eso.org/~jagonzal/CASA-Documentation/mstransform/desing/CASA-4.1-Transform-
Visibility-Framework.pdf
 Transform Visibility Iterator Development Guide
http://www.eso.org/~jagonzal/CASA-Documentation/mstransform/TVI/TVI-Development-Guide.pdf
 MSTransform framework article in the CASA Newsletter
https://science.nrao.edu/enews/casa_003/

 gcwrap: This area contains:

 Components: High level C++ components which are exposed to the python layer.
 Tasks: Python scripts using the before mentioned components to perform standard
operations (kind of recipes).
 Infrastructure scripts: Like the parallelization framework, regressions, tests, etc

 components: These are still C++ components that mainly resort to the orchestrator defined
in code. The interface files (.h headers) are automatically generated by SWIG using the
definitions contained in an xml file. This xml file also contains the in-line documentation.
The following components are maintained by ESO:

 gcwrap/tools/flagging/agentflagger_cmpt.cc: flagging framework component using the

code/flagging/Flagging/AgentFlagger.cc orchestrator, which is forward defined in
agentflagger_forward.h and declare as private member in agentflagger_private.h. The .h
interface file is generated from mstransform.xml, the methods described in this file are
exposed to the python layer.

 gcwrap/tools/mstransformer/mstransformer_cmpt.cc: mstransform framework

component using the code/mstransform/MSTransform/MSTransform.cc orchestrator,
which is forward defined in mstransformer_forward.h and declare as private member in
mstransformer_private.h. The .h interface file is generated from mstransform.xml, the
methods described in this file are exposed to the python layer.

 tasks: These are python scripts that call different components to perform standard
operations. Each task also has an xml file that provides constrains for the parameters and
defines in-line documentation. For each task there is a set of functional tests, which are ran
routinely to make sure that daily developments don't break existing functionality. The
following tasks are maintained by ESO.

 gcwrap/python/scripts/task_flagdata: flagging framework task, using the agentflagger

 component (defined in gcwrap/tools/flagging/agentflagger_cmpt.cc). Documentation
and constrains are defined in gcwrap/tasks/flagdata.xml, and functional tests defined in
gcwrap/python/scripts/tests/test_flagdata.

 gcwrap/python/scripts/task_mstransform: mstransform framework task using the

mstransformer component, with documentation and constrains defined in
gcwrap/tasks/mstransform.xml, and functional tests defined in
gcwrap/python/scripts/tests/test_mstransfrom. Additionally the following tasks also
resort to the mstransform component:
 task_split
 task_hanningsmooth
 task_cvel2
 task_partition

 Infrastructure scripts: Here there are 3 areas of particular interest:

 gcwrap/python/scripts/regressions: These are regressions containing end-to-end test

cases where many CASA tasks are exercised. At the end of the regressions there is
typically a check, on the final image peak or rms values which are the criteria to make
sure that the regression has passed successfully. Some of these regressions ran routinely
on the testing servers maintained by NRAO (see TODO).

 gcwrap/python/scripts/tests: These are the task-level functional tests, that exercise a

number of task-level use cases. Typically when a new bug is detected and fixed a new
functional test is added to make sure it does not reproduce again. These tests are run
using the Python testing framework “nose”:

http://pythontesting.net/framework/nose/nose-introduction/

 gcwrap/python/scripts/mpi4casa: This is the python-level parallelization framework

based on MPI (in particular mpi4py). Documentation is available here (see TODO)

https://svn.cv.nrao.edu/svn/casa/trunk/gcwrap/python/scripts/mpi4casa/CASA-4.3-MPI-Parallel-
Processing-Framework-Installation-and-advance-user-guide.pdf

 asap: This is the container for the ALMa Single Dish code developed by NAOJ. It is currently
migrating from an old data format (Scan Table) to the standard data format of CASA
(Measurement Set), thus it is required that all CASA code supports Single Dish data.

CASA Data Format (Measurement Set)

  Measurement Set: The CASA data format is know as Measurement Set (a specialization of the
generic CASA Table format), which is organized as a relational database, with a main table
containing the visibility data and a series of Ids pointing to meta-data in other Sub-tables:

https://casa.nrao.edu/Memos/229.html#SECTION00041000000000000000

 Tile & Layout: The data cubes (visibilities, spectral weights, channelized flags) are organized
with store managers using a tiling approach – that is, the data is read and cached using blocks
called 'tiles' which are defined the first time a Measurement Set is created.

http://casacore.github.io/casacore/group__Tables__module.html#Tables_TiledStMan

 MSSelection: There is a defined syntax to select rows on a Measurement set based on various
selection axis (spw,time,scan,baseline,uvrange, etc)

http://www.aoc.nrao.edu/~sbhatnag/misc/msselection/msselection.html#GeneralSyntax.html

 TaQL: The Table Query Language is query-like language to select rows from a Measurement
Set (actually a generic CASA table):

http://www.astron.nl/casacore/trunk/casacore/doc/notes/199.html

 Visibility Iterator / Visibility Buffer: CASA resorts to an iterator/buffer patter to transverse

the data and access to it. Typically the iterator defines chunks of data by sorting and grouping
all rows with the same Metadata Identifiers (e.g. SCAN, STATE, FIELD, SPW), and then there
is a second level of iterator per time-stamp (integration). At this level the Visibility Buffer
interface provides access to a complete integration (all baselines) for a given time-stamp.

https://casa.nrao.edu/Release4.3/docs/doxygen/html/classcasa_1_1vi_1_1VisibilityIterator2.html#detail
s
https://casa.nrao.edu/Release4.3/docs/doxygen/html/classcasa_1_1vi_1_1VisBuffer2.html#details

 Lustre Cluster: The clusters typically used for processing ALMA/EVLA data with CASA are
based on Lustre+Inifiniband connection. For more details about this HW architecture please
contact the NRAO Lustre Cluster responsible (jrobnett@nrao.edu).

Mstransform Framework
  MSTransform is a common framework to perform transformations in visibility data such as
time averaging, channel averaging, Hanning smooth, re-gridding, reference frame
transformation, Spectral Window combination/separation or simply data splitting.

http://www.eso.org/~jagonzal/CASA-Documentation/mstransform/desing/CASA-4.1-Transform-
Visibility-Framework.pdf
https://casa.nrao.edu/docs/cookbook/casa_cookbook005.html#sec271

 The key aspect of the mstransform framework is that all transformations are performed in
memory w/o requiring intermediate I/O steps:

https://science.nrao.edu/enews/casa_003/

 A first version of the MSTransform framework contained all transformations in a single class
(MSTransformManager) with resorted to function pointers to configure all transformations at
the various levels (cube/matrix/row).

https://svn.cv.nrao.edu/svn/casa/trunk/code/mstransform/MSTransform/MSTransformManager.cc

 A second version is on the way, which resorts to different classes (transforming iterators as
originally designed) to concatenate all operations. Transforming Visibility Iterators (TVI) have
the same public interface as a normal Visibility Iterator used to transverse the data, but perform
some transformations in between. It is possible to concatenate transforming iterators in an
arbitrary order:

https://open-jira.nrao.edu/browse/CAS-8347
https://open-jira.nrao.edu/browse/CAS-8219
http://www.eso.org/~jagonzal/CASA-Documentation/mstransform/TVI/TVI-Development-Guide.pdf

 At the time being what is pending is the “SPW Combination” transforming iterator, and
migrating MSTransformManager to use TVIs instead of function pointers:

https://open-jira.nrao.edu/browse/CAS-8352
https://open-jira.nrao.edu/browse/CAS-8353

 In parallel other applications like flagging (ESO) plotms (Pam Ford) and calibration (George
Moellenbrock) are using TVIs from the mstransform framework for averaging or phase shifting,
so it is important to keep an eye on their needs and developments:

https://open-jira.nrao.edu/browse/CAS-4619
https://open-jira.nrao.edu/browse/CAS-6215
https://open-jira.nrao.edu/browse/CAS-8269

MPI Parallelization framework

  Since CASA 4.3 we have been developing a MPI-based parallelization framework oriented for
job farming / trivial parallelization (no interaction between the subprocesses, only sub-process
master).

http://www.eso.org/~jagonzal/CASA-Documentation/HPC/Poster/Python-parallelization-poster-
sciops2015.pdf

https://svn.cv.nrao.edu/svn/casa/trunk/gcwrap/python/scripts/mpi4casa/CASA-4.3-MPI-Parallel-
Processing-Framework-Installation-and-advance-user-guide.pdf

http://www.eso.org/~jagonzal/CASA-Documentation/HPC/mpi4casa/CASA-Parallel-Processing-
Framework.pdf

 Current design is implemented at python-level using mpi4py backed up by OpenMPI:

https://www.duo.uio.no/handle/10852/10848

http://pythonhosted.org/mpi4py/usrman/index.html

http://pythonhosted.org/mpi4py/apiref/index.html

 And uses a Multi-Measurement Set approach (split MS by scan/spw) to break down the
processing in various independent sub-jobs (aka trivial parallelization).

https://svn.cv.nrao.edu/svn/casa/trunk/gcwrap/python/scripts/mpi4casa/CASA-4.3-Multi-MS-Structure-
doc.pdf

http://www.eso.org/~jagonzal/CASA-Documentation/HPC/TrivialParallelization/Not-So-Trivial-
Parallelization.pdf

 The parallelization framework is stable, however it normally necessary to perform some

maintenance work to migrate to new versions or adapt it to other changes in the python
infrastructure of CASA:

https://open-jira.nrao.edu/browse/CAS-9426
https://open-jira.nrao.edu/browse/CAS-9287

Flagging framework
  The CASA flagging framework was restructured already some years ago and is currently quite
stable. Each flagging algorithm is encapsulated in a separated class (Flagging Agent) and all I/O
is handled in a separated class (MSTransformDataHandler) with two specializations, one for
Measurement sets (MSDataHandler) and another for Calibration Tables (FlagCalTableHandler):

http://www.aoc.nrao.edu/~rurvashi/FlaggerDocs/FlaggerDocs.html

https://casa.nrao.edu/active/docs/doxygen/html/classcasa_1_1FlagAgentBase.html

https://casa.nrao.edu/active/docs/doxygen/html/classcasa_1_1AgentFlagger.html#details

https://casa.nrao.edu/active/docs/doxygen/html/classcasa_1_1FlagDataHandler.html

https://casa.nrao.edu/active/docs/doxygen/html/classcasa_1_1FlagDataHandler.html#details

 Currently it is quite stable, but there are some interesting developments specially in the area of
automatic flagging (RFI), and also in connection with the Transform Visibility Iterator
Framework, namely to apply some transformations on the fly before applying flag algorithms:

https://open-jira.nrao.edu/browse/CAS-6215
https://open-jira.nrao.edu/browse/CAS-5808

Setting up development environment

  Code repository: Control revision is based mainly on SVN (servers located at NRAO),
although we resort to an external reference for casacore (hosted in GitHub). It is not necessary
to have an account to check out and/or update, but it is necessary in order to make modifications
to the repository. In the case of casacore (Git) there are some administrators handling the pull
requests

 General repository (everything can be checked out from here):

https://svn.cv.nrao.edu/svn/casa/trunk/

 casacore repository (necessary to make pull requests):

https://github.com/casacore/casacore

* Some key people taking care of casacore (can resolve questions, help with pull requests,
etc)
- Ger van Diepen (ASTRON): diepen@astron.nl
- Jim Jacobs (NRAO): jjacobs@nrao.edu

 Data repository: There is also a repository of testing datasets containing small Measurement
Sets for the task-level functional tests, and also some bigger Measurement Sets for regressions.
Additionally it contains ephemeris and geodetic data which is necessary to start up CASA.

https://svn.cv.nrao.edu/svn/casa-data/trunk

NOTE: Since the data repository is quite big, we typically have only one copy per development
machine, and we add a link in the build area pointing to it (see “Build process” section).

 Build process: Currently CASA is supported for Red Hat 6/7 and MAC OX 10.10/10.11.

 Build instructions for MAC 10.10 and MAC 10.11 include installing all the 3 rd party
dependencies using MacPorts, and then configure using cmake as described here:

https://safe.nrao.edu/wiki/bin/view/Software/CASA/CasaEnvSetupOSX1010
https://safe.nrao.edu/wiki/bin/view/Software/CASA/CasaCMakeFlagsOSX1010

https://safe.nrao.edu/wiki/bin/view/Software/CASA/CasaEnvSetupOSX1011
https://safe.nrao.edu/wiki/bin/view/Software/CASA/CasaCMakeFlagsOSX1011

 There are also build instructions for RedHat 6/7, including configuration of a yum
repository for the 3rd party dependencies:

https://safe.nrao.edu/wiki/bin/view/Software/CASA/HowToSetupAnEL6CasaDevelopmentComputer
https://safe.nrao.edu/wiki/bin/view/Software/CASA/CasaCMakeFlagsRH6

https://safe.nrao.edu/wiki/bin/view/Software/CASA/HowToSetupAnEL7CasaDevelopmentComputer
https://safe.nrao.edu/wiki/bin/view/Software/CASA/CasaCMakeFlagsRH7

But the ESO CASA development machines already have this configuration (CASA development 3 rd
party package dependencies) available in the following machines (using Centos instead of RedHat, but
they are almost identical).

Host Cores RAM Container OS Local Storage Network Area Data repository
Machine (Dockers) (No backup!) Storage (read-only, updated
(domain (No backup!) daily)
hq.eso.org)
almahost01 8 16G almahpc01 Centos 6 /data/users /data/data_casa
/diska (local)
almahost02 12 12G almahpc02 Centos 6 /diska /data/nas (btrfs) /data/data_casa
/data1 (mounted from (mounted from
/data/users (solid sagan5 with sagan4 with NFS)
state disk – don't use NFS)
for storage)
sagan5 16 64G almahpc03 Centos 6 /data/users (ext4) /data/nas (btrfs) /data/data_casa
/data/nas2 (ext4) (mounted from
almahpc04 Centos 7
sagan4 with NFS)

NOTE 1: These machines are accessible using the same ESO login credentials as for the rest of the system (e.g. email,
Navision, etc). ESO user accounts are also auto-mounted via NFS under /home/
NOTE 2: The 3rd party dependencies should be upgraded using yum as announced via the casa-staff mailing list.

In any of these machines the sequence is typically

 Check out code in same separated compartment (CASA_CODE_DIR)

svn co https://svn.cv.nrao.edu/svn/casa/trunk/

 Create a build area (CASA_BUILD_AREA), containing and installation directory

(CASA_INSTALL_DIR), separated build areas casacore, code, gcwrap, asap and a link
to the data repository (as explained previously there should be only one per development
machine, say located at CASA_DATA_DIR).
mkdir casa-trunk #This is CASA_BUILD_AREA
cd casa-trunk
mkdir linux_64b casacore code gcwrap build #linux_64b is CASA_INSTALL_DIR
mkdir casacore/build code/build gcwrap/build asap/build
ln -s <CASA_DATA_DIR> data #Common data repository is located at CASA_DATA_DIR

 Create a file with the following common cmake options (CASA_CMAKE_OPTS)

 For CentOS 6 / RedHat 6

# build casacore for CASA
set(CASA_BUILD TRUEON CACHE BOOL "" FORCE)
# build casacore python bindings
set(BUILD_PYTHON ON CACHE BOOL "" FORCE)
set(PYTHON_EXECUTABLE "/usr/lib64/casa/01/bin/python" CACHE FILEPATH "" FORCE)
set(PYTHON_INCLUDE_DIR "/usr/lib64/casa/01/include/python2.7" CACHE PATH "" FORCE)
set(PYTHON_LIBRARY "/usr/lib64/casa/01/lib/libpython2.7.so" CACHE FILEPATH "" FORCE)
set(SWIG_EXECUTABLE "/usr/lib64/casa/01/bin/swig" CACHE FILEPATH "" FORCE)
set(CASA_VERSION_FEATURE 1 CACHE STRING "" FORCE)

# no library versioning to avoid need to relink everything on update

set(NO_SOVERSION TRUEON CACHE BOOL "" FORCE)

# workaround for el6 cmake bugs

set(Boost_NO_BOOST_CMAKE ON CACHE BOOL "" FORCE)

 For CentOS 7 / RedHat 7

 # build casacore for CASA
set(CASA_BUILD TRUEON CACHE BOOL "" FORCE)
# build casacore python bindings
set(BUILD_PYTHON ON CACHE BOOL "" FORCE)
set(PYTHON_EXECUTABLE "/opt/casa/01/bin/python" CACHE FILEPATH "" FORCE)
set(PYTHON_INCLUDE_DIR "/opt/casa/01/include/python2.7" CACHE PATH "" FORCE)
set(PYTHON_LIBRARY "/opt/casa/01/lib/libpython2.7.so" CACHE FILEPATH "" FORCE)
set(CASA_VERSION_FEATURE 1 CACHE STRING "" FORCE)

# no library versioning to avoid need to relink everything on update

set(NO_SOVERSION TRUEON CACHE BOOL "" FORCE)

# workaround for el6 cmake bugs

set(Boost_NO_BOOST_CMAKE ON CACHE BOOL "" FORCE)

 Configure and build in order casacore, code, gcwrap, asap (from their build directories)
cmake -C <CASA_CMAKE_OPTS> -DCMAKE_INSTALL_PREFIX=CASA_INSTALL_DIR <CASA_CODE_DIR>/casacore
make all -j8
make install (not automatic for casacore)

 Source casainit.sh script which should have been created automatically inside
<CASA_BUILD_AREA> . After that all paths are properly set and it is should be
possible to start-up CASA. By default a logger gui pops-up but it is possible to disable it
(and have logs showin at the terminal) with the options --nologger --log2term --nogui
source <CASA_BUILD_AREA>/casainit.sh
casa --nologger --log2term --nogui

 Now you can inspect the inline documentation of tasks like flagdata and mstransform by
typing “help flagdata” or “help mstransform”. You can also take a look at the input
parameters of each tasks typing “inp flagdata”/”inp mstransform”.

 Additionally CASA take into account the following configuration files and
environmental variables:

 $HOME/.casarc: This is a file that may contain some specially configuration to

switch on/off low level features, specially in connection with I/O like cache sizes,
asyncrnous I/O or memory resident subtable. It looks like this (but Caution!!! It is
recommended not to switch on any of these settings).
table.tsm.option: cache
table.tsm.maxcachesizemb: 64
table.tsm.option: buffer
table.tsm.buffersize: 65536
table.tsm.option: mmap

MemoryResidentSubtables.enable: true

 $HOME/.casa/init.py: This file contains some python-level imports that are required
at init time. It can also include settings of global variables to control e.g. exception
handling.
NOTE: This setting is actually recommended: __rethrow_casa_exceptions=True

 OpenMP Environmental Variables: OpenMP is used in various places and the

number of threads can be control by setting the environmental variable
OPM_NUM_THREADS.
Testing environment
  C++ unit tests (Gtests)
 CASA relies on the Gtest (Google test framework) for low-level testing (e.g. individual
classes). Unit tests therefore have to include <gtest/gtest.h> and derive from
::testing::Test. There are several examples in the code/mstransform/TVI/test module,
where a “Transforming Iterator” is compared vs the old code “MSTransformIterator”, for
example tChannelAverageTVI is a good example.

 These unit tests are compiled as small executables, as defined in the module's CmakeList,
using the Cmake macro “casa_add_google_test”: but sometimes it happens that there are
dependencies on other modules that come later on in the compilation chain, therefore it is
necessary to add the unit test executable to some other module. This happens for instance
with many tests under code/mstransform/TVI/test, which are actually defined in
code/synthesis/CMakeList:
casa_add_google_test( MODULES mstransform SOURCES ../mstransform/TVI/test/tChannelAverageTVI.cc ../mstransform/TVI/test/TestUtilsTVI.cc)
casa_add_google_test( MODULES mstransform SOURCES ../mstransform/TVI/test/tHanningSmoothTVI.cc ../mstransform/TVI/test/TestUtilsTVI.cc)

 Once a Gtest has been added to an appropiate CmakeList, then it is possible to compile it,
and it will be installed under the module in which the defining CmakeList belongs, for
instance:
 tUVContSUbTVI has its code under code/mstransform/TVI/test
 But it is add es a test in code/synthesis/CMakeList
 It is installed under code/build/synthesis/tUVContSubTVI

 C++ unit tests are the first line of tests that ran after a commit, so if something went wrong
the developer responsible of the offending commit gets a notification email.

 Python tasks-level functional tests

 As described in the previous sections there is a collection of functional tests that exercise
each CASA task. These unit tests are based on Python NOSE framework and in order to use
it have to import the NOSE's unittest module, and create unit test classes derived from
unittest.TestCase with a setUp method (typically copy over test files) a tearDown
method (delete test files) and various unit tests. There are many good examples for the
flagging and mstransform tasks:
 gcwrap/python/scripts/tests/test_flagdata
 mstransform (gcwrap/python/scripts/tests/test_mstransform

 In order to run these tests there is an utility script called “runUnitTests” in which it is
possible to specify in brackets an entire class of tests to be run or only one particular test.
 Run all flagdata tests:
casa --nogui --log2term -c <INSTALL_PATH>/linux_64b/python/2.7/runUnitTest.py test_flagdata

 Run all flagdata 'shadow' mode tests:

casa --nogui --log2term -c <INSTALL_PATH>/linux_64b/python/2.7/runUnitTest.py test_flagdata[test_shadow]

 Run only the flagdata test related to ticket CAS-2399

casa --nogui --log2term -c <INSTALL_PATH>/linux_64b/python/2.7/runUnitTest.py test_flagdata[test_CAS2399]

 NOTE: It is fundamental (actually mandatory) to run the task-level unit tests of all tasks
affected by modifications in the corresponding code base.

 Complete information about the task-level unit tests is available here:

 http://www.eso.org/~scastro/ALMA/CASAUnitTests.htm
http://www.eso.org/~scastro/ALMA/CASAUnitTests.pdf
http://www.eso.org/~scastro/ALMA/CASAUnitTests.doc

 Regressions

 Additionally there are a number of regressions defined under

gcwrap/python/scripts/regressions which are routinely ran by the test environment at
NRAO:

https://svn.cv.nrao.edu/test-summary/

 In case of a failing regressions and email is sent to the case-staff mailing list, in which case
it is necessary to inspect which regressions and/or functional tests may have failed. This
email points to a Bamboo job, which is a web tool integrated with Jira that shows the failing
tests and associated commits and Jira-Issues (See Software Engineering section).

 CASA guides

 Finally there is a set of CASA Guides available online, describing with lots of details a
complete data reduction process using CASA. It is highly recommend to go trough at least
one of this guides as an introduction to CASA:

https://casaguides.nrao.edu/index.php/ALMAguides

Software engineering

 Atlassian tools: CASA Uses the Atlassian ecosystem tools, which are very well integrated
among themselves and with svn/git

 Jira: Ticketing system, which is used for all work items (bug reports and engineering tasks).
It shows the svn/git commits.
https://open-jira.nrao.edu

 Confluence: Team collaboration tool, which shows all tickets and their scheduled start/stop
dates automatically. It also shows vacations (license), travels and meetings but not like the
Jira tickets, the corresponding dates have to be manually entered by each developer.

https://open-confluence.nrao.edu

 Bamboo: Test reporting system that shows for each test job the associated tickets and svn/git
commits.
https://open-bamboo.nrao.edu

 Ticket life-cycle: CASA tickets have a life-cycle with the following phases
  Open: This is just ticket creation, it can be a bug open by users and/or other CASA team
members, or an engineering tasks open by CASA leads or developers. In general is good to
have tickets for all relevant work pieces (including documentation), collected under an
umbrella ticket (or “Epic” using the Atlassian terms) and separated individually whenever it
makes sense.
 Scheduled: At this phase is when it is estimated how much effort a ticket will take (start/stop
dates). This is mandatory so that the ticket shows up in the Confluence team collaboration
tool. Also it is mandatory to specify the target CASA release.
 Ready to verify: This is the verification phase when the main developers of the ticket verify
the functionality (using C++ Gtests, Python task tests or similar).
 Ready to validate: After the ticket has been verified by developers it goes to user validation
phase (typically science validation) which is conducted by an independent team of
users/scientists.
 Priorities: Typically CASA tickets have priority “High”. Priority “Blocked” is reserved only
to issues affecting coming releases. Which cannot be circumvented by a workaround.

 Release cycle:

 Official releases: CASA has typically 2 major releases per year, but there can be various
patch releases (e.g. CASA 4.5, 4.5.1, 4.5.2, 4.5.3). For ALMA there is usually a new cycle
starting in October-November so a release is produced in September. These are the release
dates of the latests CASA releases:

Version Release date

4.3 12/30/99
4.3.1 12/30/99
4.4 16-Jun-2015
4.5 29-Oct-2015
4.5.1 02-Feb-201
4.5.2 11-Feb-2016
4.5.3 06-Apr-2016
4.6.0 04-Apr-2016
4.7.0 19-Oct-2016

 Feature freeze: Typically there is a feature freeze 1 month before the target release data, and at
that point a release branch is created and commits are quite restricted.

 Release criteria: The release is only produce after all regression tests are passing successfully. In
case of failing regression a test summary is sent to the casa-staff mailing list and developers
have to inspect this list to check if some of their tests are failing.

 Continuous integration: Apart from the official releases, there is a continuous integration
 process, handled by a Jenkins Job, which produces tarballs available for download:

 Red Hat: https://svn.cv.nrao.edu/casa/linux_distro/stable/

 MAC OX: https://svn.cv.nrao.edu/casa/osx_distro/stable/
 Jenkins Overview: https://casa-jenkins.nrao.edu/job/casa-trunk/
 Continuous Integration summary: https://svn.cv.nrao.edu/test-summary/

 Documentation: There are 3 levels of Documentation

 Developer documentation: It is generated automatically from Doxygen, using the headers

documentation.

 Top-Level view

https://casa.nrao.edu/active/docs/doxygen/html/index.html

 Example AgentFlagger:

https://casa.nrao.edu/active/docs/doxygen/html/classcasa_1_1AgentFlagger.html

 Example MSTransformDatahandler:

https://casa.nrao.edu/active/docs/doxygen/html/classcasa_1_1SubMS.html#a2ff8
8a50ca24c19b2aeda5a705ced1a4

 Task and component in-line documentation: This is the documentation defined via xml and
exposed to the python-level thanks to SWIG.

 gcwrap/tools/mstransformer/mstransformer.xml
 gcwrap/tools/flagging/agentflagger.xml
 gcwrap/tasks/flagdata.xml
 gcwrap/tasks/mstransform.xml

 User manual: Up to release 4.7 CASA documentation has been compiled in the so-called
“CASA Cookbook” which is generated via Latex. But as of today we are migrating to Plone
a content management system.

 CASA Cookbook Latex sources:

 Latest CASA Cookbook: https://casa.nrao.edu/docs/cookbook/
 Plone (New system to be used): https://casa.nrao.edu/plonedocs

CASA team and community

  casa-staff email list: The casa-staff mailing list (casa-staff@nrao.edu) is the fundamental
vehicle for communication within the CASA team. Typically developers post questions,
concerns, or announcements about new infrastructure changes or upcoming developments.

 CASA Twiky: This is the collaborative web area where developers post documentation to share
with other team members: https://safe.nrao.edu/wiki/bin/view/Software/CASA/WebHome

 Monday Telecons: There is a weekly Teleconference happening regularly every Monday at

15:00 UTC. Typically each developer fills a weekly report a few days before:

 DIAL-IN NUMBERS & PASSCODES

 IP: 192.33.117.12##8110
 Phone: (434) 817-6524
 Usual Topics
 Build, Release
 Verification Testing
 Validation Testing
 Architecture
 HPC
 Development

 List of CASA Telecon Minutes-of-Meetings (MoM):

https://safe.nrao.edu/wiki/bin/view/Software/CASA/CASAMeetingPage#Monday_Morning_Meetings

 CASA Users Committee: There is a stable CASA users Committee that holds a regular
meeting each year to give recommendations about CASA, details can be found here:

https://safe.nrao.edu/wiki/bin/view/Software/CASA/CASAUsersCommittee

 CASA Staff: This is a partial list of CASA staff which not at all complete (sorry for whoever I
may have missed!!) but contains the most regular contacts in the context of the ESO
contributions to CASA. I highlight the following groups:
 in green the members of the TVI (Transform Visibility Iterator) development group which is
likely to be very active in the future.
 In blue the members of the CASA HPC Group who have been working closely with MPI.
 In orange the members of the flagdata development group.

Name Role EMail

Jeff Kern CASA Group Lead jkern@nrao.edu
Jared Anand Crossley CASA Community liaison jcrossle@nrao.edu
Juergen Ott CASA Project scientist jott@nrao.edu
Crystal Brogan ALMA CASA subsystem scientist cbrogan@nrao.edu
Jennifer Donovan Meyer CASA Scientific Testing Lead jmeyer@nrao.edu
Lindsey Davis ALMA Pipeline lead ldavis@nrao.edu
Brian Butler EVLA Pipeline Lead bbutler@nrao.edu
Ger van Diepen casacore responsible diepen@astron.nl
Kanako Sugimoto Single Dish Lead kana.sugi@nao.ac.jp
James Jacobs CASA Architect jjacobs@nrao.edu
casacore responsible
TVI development group
Takeshi Nakazato TVI Development Group takeshi.nakazato@nao.ac.jp
MSTransform development group
Pam Ford CASA Plotms responsible pford@nrao.edu
(Phasing out?) TVI Development group
George Moellenbrock CASA Calibration responsible gmoellen@nrao.edu
TVI development group
UVContsub development group
Sandra Castro CASA HPC Group lead (MPI)
Flagdata development group
MSTransform development group
Stewart Williams ALMA Pipeline Group stewart.williams@stfc.ac.uk
CASA HPC lead (MPI)
Urvashi Rao CASA HPC Group (MPI) rurvashi@nrao.edu
Flagdata development group
Imaging Development group
Imaging Parallelization group
Sanjay Bhatnagar Imaging Development group bhatnagar.sanjay@gmail.com
Imaging Parallelization group
CASA HPC Group (MPI)
Martin Pokorny Imaging Parallelization group mpokorny@nrao.edu
I/O Improvements group
 CASA HPC Group (MPI)
Kumar Golap Imaging Development group kgolap@aoc.nrao.edu
CASA HPC Group
Takahiro Tsutsumi UVContsub development group ttsutsum@nrao.edu
Imaging Development group
Imaging Parallelization group
CASA HPC Group
James Robnett NRAO Lustre cluster responsible jrobnett@nrao.edu
CASA HPC Group
Andy Hale CASA build & test lead ahale@nrao.edu

Darrell Schiebel CASA build & test darrell@schiebel.us

CASA Python interface darrell.schiebel@gmail.com
Akeem Wells CASA build & test awells@nrao.edu
Ville Souranta CASA build & test vsuorant@nrao.edu

CASA Accounts

The following accounts are necessary for CASA developers:

Domain Server
NRAO CASA Atlassian Tools https://open-jira.nrao.edu
(Jira, Confluence, Bamboo) https://open-confluence.nrao.edu
https://open-bamboo.nrao.edu
NRAO CASA Twiky https://safe.nrao.edu/wiki/bin/view/Software/CASA/WebHome
NRAO Linux domain login.aoc.nrao.edu
(Socorro and Charlottesville cluster) ssh.cv.nrao.edu
NRAO Windows domain https://casa.nrao.edu/plonedocs
(Plone documentation)

List of pending tickets

 Task Ticket Description Effort
flagging CAS-5808 Changes in rflag threshold scaling defaults Minor
CAS-6215 Generalized pre-averaging in flagdata Minor (follow up validation)
mstransform CAS-8352 SPW Combination TVI Major (New development)
features CAS-6835 Allow combination of SPWs with different Major (May require low level
number of channels infrastructure changes)
CAS-8353 Migrate MSTransformManager Major (Significant
to use new modular TVIs infrastructure changes)
CAS-9452 Migrate denoising lib to new version of GSL Medium-Major (depending
on how much robust fit
functionality is used)
CAS-9451 Put Recep's re-indexing package into Medium (Requires in-depth
production verification)
CAS-8094 conversion to SOURCE frame Major (Significant
for non-ephemeris sources infrastructure changes)
CAS-7929 Allow phase center re-assigment Major (New development)
independent of reference frame
transformation in mstransform
CAS-7930 Improve OTF phase center shift in Major (New development)
mstransform o match fixvix
mstransform CAS-9414 mstransform apparently doesn't Medium (Recently open)
bugs per-average correctly
CAS-9441 Issue with channel averaging Medium (Recently open)
TVI with selection
mpi4casa CAS-9426 Migrate mpi4casa to adapt to new Medium (Recently open)
CASA python initialization schema
CAS-9287 Migrate mpi4casa to Open MPI 2.0.1 Minor (follow up
verification)