GIS Seminar, September 21

Raster Formats

Data source vs MXD layer vs layer file

 In a map document, a layer is a pointer to data with instructions for ArcMap about how to display the data
(the layer properties).
 More than one layer can point to the same data. Different properties are allowed for each layer.
 The layer properties you configure are saved in the mxd or can be exported to a separate Layer File (.lyr
extension)
 The Layer File does not contain the data, only the properties and is only usable by ArcMap.
 The Layer File has to be able to find the data when added to another map. Use relative paths.
o Check Map Document Properties > ‘Store relative pathnames to data sources’ checkbox
o Or, use Save to Layer File tool
 Not necessary, but save the .lyr file in the same folder as the data source when possible so that they stay
together

Raster layers and Image functions

 Raster layers can be used as inputs to an image function, an on-the-fly calculated raster based on the input
 Alternative to writing a raster out to disk based on a calculation
 Example is putting a hillshade function on top of a dem
o Open Image Analysis window
o Select raster as input to function and click the Add Function button
o Right-click on the name of the raster, select Insert Function and choose the function
o Fill out the parameters and close the Function Template Editor.
o There is a new raster layer in your map document that you can treat as any other raster layer,
including as input to another raster process.
o Change function parameters by going to the Function tab in the layer properties dialog and double-
clicking on the function name to open the Raster Function Properties dialog.
Raster basics
 A raster is an ordered matrix/array/grid of values, with instructions to a program for how to display the
values:

Example pseudo file, header and values

This raster is 15 columns x 5 rows
Start displaying these values in the upper left of your display:
4,4,4,4,4,7,7,7,7,7,3,3,4,4,4,4,4,4,4,4,4,7,7,7,3,4,4,4,5,5,4,4,4,4,4,7,7,7,4,4,4,5,5,5,5
,4,4,4,4,7,7,7,4,4,4,4,3,3,3,4,4,4,4,7,7,7,7,7,3,3,3,3,4,4,4
Result:
4,4,4,4,4,7,7,7,7,7,3,3,4,4,4
4,4,4,4,4,4,7,7,7,3,4,4,4,5,5
4,4,4,4,4,7,7,7,4,4,4,5,5,5,5
4,4,4,4,7,7,7,4,4,4,4,3,3,3,4
4,4,4,7,7,7,7,7,3,3,3,3,4,4,4
 Values can be elevation, wind speed, land cover category, gravity measurement, color (RGB stacked
rasters) etc.
 When displayed on a screen, the values are represented by a color within a pixel which has a width and a
height.
 A non-GIS image viewer will keep track of values using image coordinates

 Image coordinates vs spatial reference coordinate

o In image space, the width and height are defined by the screen resolution
o In coordinate space, the width and height are defined by additional georeferencing information
Georeferencing
 ‘Sidecar’ or ‘buddy’ file: world file (*.tfw, *.jpw, *.bpw, etc). Text file you can open in a regular text editor.
50 - cell size in x-dimension
0 - rotation about y-axis
-0 - rotation about x-axis
-50 - cell size in y-dimension
666907.78621063067 - x-coordinate of the center of the upper left pixel
6795674.9494232498 - y-coordinate of the center of the upper left pixel

 Embedded: eg., ASCII raster header

ncols 407
nrows 404
xllcorner 666882.78621063 (can also specify xllcenter)
yllcorner 6775499.9494232
cellsize 50
NODATA_value -9999
-9999 -9999 -9999 -9999 -9999 -9999 97 97 90 89 95 87 53 56 55 54 57 58 55 57 56 57 58 59
56 59 55 56 58 54 55 59 60 56 58 56 56 56 62 58 52 51 -9999 -9999 -9999 -9999 -9999 -9999
-9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999
-9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999
-9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999

 Note that neither the world file nor the ASCII raster headers have any information about the coordinate
system. You need a .prj file, auxiliary file (.aux.xml) or knowledge of the correct SR.
 .prj file:
Projection UTM
Zone 5
Datum NAD83
Spheroid GRS80
Units METERS
Zunits NO
Parameters

 Auxiliary file (.aux.xml) has SpatialReference XML element:

<SpatialReference xsi:type="typens:ProjectedCoordinateSystem">
<WKT>PROJCS["NAD_1983_Alaska_Albers",GEOGCS["GCS_North_American_1983",DATUM["D_North_Amer
ican_1983",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["De
gree",0.0174532925199433]],PROJECTION["Albers"],PARAMETER["False_Easting",0.0],PARAMETER[
"False_Northing",0.0],PARAMETER["Central_Meridian",-
154.0],PARAMETER["Standard_Parallel_1",55.0],PARAMETER["Standard_Parallel_2",65.0],PARAME
TER["Latitude_Of_Origin",50.0],UNIT["Meter",1.0],AUTHORITY["EPSG",3338]]</WKT>

 Georeferencing can be embedded in the header of a few binary formats, eg, TIFF (which makes it a geotiff),
JPEG 2000, ECW, MrSID.
 Check properties of raster in ArcCatalog or use gdalinfo command to see if a raster has georeferencing
associated with it:
>gdalinfo ascii_raster.asc
Driver: AAIGrid/Arc/Info ASCII Grid
Files: ascii_raster.asc
ascii_raster.prj
Size is 407, 404
Coordinate System is:
PROJCS["UTM Zone 5, Northern Hemisphere",
GEOGCS["NAD83",
DATUM["North_American_Datum_1983",
 SPHEROID["GRS 1980",6378137,298.257222101,
AUTHORITY["EPSG","7019"]],
TOWGS84[0,0,0,0,0,0,0],
AUTHORITY["EPSG","6269"]],
PRIMEM["Greenwich",0,
AUTHORITY["EPSG","8901"]],
UNIT["degree",0.0174532925199433,
AUTHORITY["EPSG","9122"]],
AUTHORITY["EPSG","4269"]],
PROJECTION["Transverse_Mercator"],
PARAMETER["latitude_of_origin",0],
PARAMETER["central_meridian",-153],
PARAMETER["scale_factor",0.9996],
PARAMETER["false_easting",500000],
PARAMETER["false_northing",0],
UNIT["METERS",1]]
Origin = (666882.786210629970000,6795699.949423200500000)
Pixel Size = (50.000000000000000,-50.000000000000000)
Corner Coordinates:
Upper Left ( 666882.786, 6795699.949) (149d53'17.59"W, 61d15'35.26"N)
Lower Left ( 666882.786, 6775499.949) (149d54'21.69"W, 61d 4'43.36"N)
Upper Right ( 687232.786, 6795699.949) (149d30'34.38"W, 61d15' 2.06"N)
Lower Right ( 687232.786, 6775499.949) (149d31'46.24"W, 61d 4'10.40"N)
Center ( 677057.786, 6785599.949) (149d42'29.97"W, 61d 9'53.25"N)
Band 1 Block=407x1 Type=Int32, ColorInterp=Undefined
NoData Value=-9999
Georeferencing demo
 ‘Update georeferencing’ – creates world file, .aux.xml, and internal tags, if applicable.
 Update or rectify?
o Transformations

o Affine (1st order polynomial) - update is fine

o 2nd, 3rd order or spline, eg., any rubber sheeting – rectify or always provide .aux.xml.
o The most honest way to release a georeferenced raster is probably to release the original, un-
georeferenced raster as well as the .aux.xml file which will contain the transformation methods and
 the control links used rather than a rectified raster. From a rectified raster, one cannot go back to
the untransformed original to assess the accuracy of the georeferencing.
 In general, avoid any transformation other than affine. Exception is when you have no reason to believe
that the non-georeferenced image must maintain it’s aspect ratio.
o For example, a scan of a field sheet that is made up of taped together sections of a paper map. Paper
can shrink and warp over time and there are probably gaps in the taped seams. In this case, you
need to provide as many georeferencing links as possible and you can use a rubber sheet
transformation
Pixel/bit depth
 Determines the range of possible values in a cell.
 Resize, if possible, for your range
unsigned 1 bit = 0 to 1
unsigned 2 bit = 0 to 4
unsigned 4 bit = 0 to 16
unsigned 8 bit = 0 to 255
signed 8 bit = -128 to 127
unsigned 16 bit = 0 to 65535
signed 16 bit = -32768 to 32767
unsigned 32 bit = 0 to 4294967295
signed 32 bit = -2147483648 to 2147483647
floating point 32 bit = -3.402823466e+38 to 3.402823466e+38 The default for DEMs
unsigned 64 bit = 0 to 18446744073709551616

 Nominal size of raster is rows * columns * bands * bytes. An 8-bit RGB image measuring 5000 x 5000 is
5,000 (height) x 5,000 (width) x 3 (bands) x 1 (byte) = 75,000,000 bytes = 75 MB
 Size of a floating point 32 bit DEM – 30 MB
 The same DEM reduced to 16 unsigned – 14.5 MB
Compression
 Lossy vs lossless

o Lossy is fine for background images but usually not suitable for data
o Lossless compression does not usually result in much compression unless there are large areas of
NoData or the same value.
Compression Lossless/lossy Notes
LZ77 lossless Used in PNG
PackBits lossless Apple format
LZW lossless Used in GIF
JPEG lossy Choose amount of compression
JPEG 2000 either

 TIFF files can use any of these methods or none.

o An uncompressed tiff – 190 MB
o The same tiff compressed with LZW – 167 MB
o The same tiff compressed with JPEG – 8.5 MB
o From ArcMap raster layer, choose Data > Export Data from right-click menu
o When TIFF is selected as output Format, Compression Type dropdown has all available options.
 Compressed files take up less space on disk, but are uncompressed in RAM when used. The greater the
compression, the longer the decoding takes.
 In general, for background imagery – TIFF with JPEG compression is good compromise between file size,
speed, and accuracy
Resolution
 1987, Waldo Tobler, UCSB cartographer:
o “..divide the denominator of the map scale by 1,000 to get the detectable size in meters. The
resolution is one half of this amount.”
Pyramids

 Downsampled versions of the original raster that speed up panning and zooming.
 At ArcGIS 10, pyramids are stored in .ovr file adjacent to the raster
 Can also be internal in JPEG 2000, ECW, MrSID, and TIFF
 GDAL overview = ESRI pyramid
Formats

GDAL – Geospatial Data Abstraction Library http://www.gdal.org/

 Open source library for raster and vector data formats
 Drivers (read/write access) for 142 raster formats (some have 3rd party dependencies)
o ESRI – 70 formats
o Global Mapper – 65 formats
o QGIS – all GDAL supported formats
o In practice, we are mostly talking about just a handful; TIFF, JPEG, PNG, ESRI GDB, GRID,IMG, ASCII,
NetCDF, NITF
 Raster utilities
o Available as Windows DOS, linux, unix console/terminal commands
o Python bindings (https://pypi.python.org/pypi/GDAL)
o R bindings (https://cran.r-project.org/web/packages/rgdal/index.html)
o GDAL utilities get installed with QGIS, GRASS, OSGeo (which includes a nice console shell so you
don’t have to worry about updating your PATH variable), FWTools, a few other open-source GIS
packages
o QGIS provides GUIs for most of the GDAL utilities.
A few common GIS Raster formats

Format Ownership Notes

TIFF Adobe Ubiquitous; flexible; internal georeferencing, tiles, and pyramids
JPEG/JPEG Joint Photographic JPG – 3 band only, no internal georeferencing.
2000 Experts Group JP2 – single or multiband, internal georeferencing
PNG Open, World Wide Web Internal channel for transparency, very common internet interchange
Consortium (W3C) format, used for tiles in web-served rasters.
FGDBR – ESRI Useful if there is a need to store raster data in the same container as
ESRI file gdb associated vector data.
raster
ESRI binary ESRI Similar to ArcInfo coverage, a set of files grouped in two directories;
GRID grid name and info
ASCII Created by ESRI but Completely open, text-based BUT no compression leading to big files.
functionally open
NetCDF Open, created by Univ. Multi-variable and multi-dimensional (eg., time, height)
Corp for Atmospheric environmental data.
Research (UCAR)

A couple
others
GeoPackage Open Geospatial SQL-Lite based database raster stored in png or jpg tiles.
raster Consortium
GDAL Open XML-based specification for creating raster catalogs, eg., multiple,
Virtual tiled rasters can be viewed and analyzed as a single raster without
Raster loading all of the tiles at once.
(VRT)
GDAL Examples
 Internal tiles and pyramids in geotiff
o SPOTchip_0.tif - 2.6 GB
o Step 1 – build internal tiles and compress with JPEG compression
gdal_translate -co TILED=YES -co COMPRESS=JPEG SPOTchip_0.tif SPOTchip_1.tif

o Step 2 – build internal pyramids. Create 8 levels to open in ArcGIS

gdaladdo -r average --config COMPRESS_OVERVIEW JPEG \
--config PHOTOMETRIC_OVERVIEW YCBCR \
--config INTERLEAVE_OVERVIEW PIXEL \
SPOTchip_2.tif 2 4 8 16 32 64 128 256
o SPOTchip_2.tif – 462 MB

 Virtual Raster (VRT)

o Build one virtual raster from a collection of tiles, eg., a directory of rasters or a list of paths to rasters
– RGB_items_1perline.txt
o Build VRT from list
gdalbuildvrt -input_file RGB_items_1perline.txt
SPOT5.SDMI.AK.ORTHO.RGB.vrt

o Build overviews
o SPOT5.SDMI.AK.ORTHO.RGB.vrt
 4,475 rasters
 2.08 TB of data
 8.3 GB of overviews
 Opens in ArcGIS and QGIS