Asme PTB-12-2017 PDF

AS M E PTB-1 2-201 7
G u i d e l i n e s fo r Ad d re s s i n g D a ta
G a p s a n d R e c o rd k e e p i n g fo r
AS M E B 3 1 . 4 , B 3 1 . 8 a n d B 3 1 . 8 S
fo r P i p e l i n e I n te gri ty M a n a ge m e n t
PTB-1 2-201 7
GUIDELINES FOR ADDRESSING

DATA GAPS AND
RECORDKEEPING FOR ASME
B31 .4, B31 .8 AND B31 .8S FOR
PIPELINE INTEGRITY
MANAGEMENT
Prepared by:
Andrew R. Lutz, P.E., DNV-GL
Elizabeth King, DNV-GL
Dr. Thomas Bubenik, DNV-GL
Dr. William Harper, DNV-GL
Date of Issuance: June 28, 2017
This guideline was prepared by ASME Standards Technology, LLC (ASME ST-LLC) and sponsored
by The American Society of Mechanical Engineers (ASME) Pressure Technology Codes & Standards.
Neither ASME, the author, nor others involved in the preparation or review of this document, nor any
of their respective employees, members or persons acting on their behalf, makes any warranty, express or
implied, or assumes any legal liability or responsibility for the accuracy, completeness or usefulness of any
information, apparatus, product or process disclosed, or represents that its use would not infringe upon
privately owned rights.
Reference herein to any specific commercial product, process or service by trade name, trademark,
manufacturer or otherwise does not necessarily constitute or imply its endorsement, recommendation or
favoring by ASME or others involved in the preparation or review of this document, or any agency thereof.
The views and opinions of the authors, contributors and reviewers of the document expressed herein do not
necessarily reflect those of ASME or others involved in the preparation or review of this document, or any
agency thereof.
ASME does not “approve,” “rate, ” or “endorse” any item, construction, proprietary device, or activity.
ASME does not take any position with respect to the validity of any patent rights asserted in connection
with any items mentioned in this document, and does not undertake to insure anyone utilizing a standard
against liability for infringement of any applicable letters patent, nor assume any such liability. Users of a
code or standard are expressly advised that determination of the validity of any such patent rights, and the
risk of infringement of such rights, is entirely their own responsibility.
Participation by federal agency representative(s) or person(s) affiliated with industry is not to be
interpreted as government or industry endorsement of this code or standard.
ASME is the registered trademark of The American Society of Mechanical Engineers.
No part of this document may be reproduced in any form,

in an electronic retrieval system or otherwise,
without the prior written permission of the publisher.
The American Society of Mechanical Engineers

Two Park Avenue, New York, NY 10016-5990
ISBN No. 978-0-7918-7171-3
Copyright © 2017 by
THE AMERICAN SOCIETY OF MECHANICAL ENGINEERS
All Rights Reserved
PTB-12-2017: Guidelines for Addressing Data Gaps and Recordkeeping for ASME B31.4, B31.8 and B31.8s for
Pipeline Integrity Management
TABLE OF CONTENTS
Foreword ................................................................................................................................................. v
Abstract .................................................................................................................................................. vi
Abbreviations and Acronyms .............................................................................................................. vii
Definitions .......................................................................................................................................... viii
1 Purpose and Use ............................................................................................................................... 1
1.1 Background of Data and Recordkeeping in the Pipeline Industry ............................................. 1
1.1.1 United States ................................................................................................................. 1
1.1.2 CSA Z662 ..................................................................................................................... 3
1.1.3 AS 2885 ........................................................................................................................ 3
1.1.4 International .................................................................................................................. 3
1.1.5 Recent Improvements ................................................................................................... 4
1.2 Review of Data Management Challenges and Solutions in Other Industries ............................ 5
1.2.1 Medical Industry ........................................................................................................... 5
1.2.2 Nuclear Industry ........................................................................................................... 6
1.3 Recommendations ...................................................................................................................... 9
1.4 Recommendations for Revisions to ASME B31.4, B31.8, B31.8S, and Future Work .............. 9
2 Program Overview and Typical Roles and Responsibilities .......................................................... 11
3 Company Pipeline Information Database ....................................................................................... 12
3.1 Locations and Formats of Pertinent Data ................................................................................. 12
3.2 Modernizing and Digitizing Data and Information .................................................................. 14
3.2.1 Document Indexing with Metadata ............................................................................. 15
3.2.2 Digitization Quality Assurance ................................................................................... 17
3.2.3 Disposition – Disposal of Data ................................................................................... 17
3.3 Structuring Data and Data Sources to Maximize Access ........................................................ 18
3.3.1 Data Mining and Gap Filling to Establish “Direct Links” to Source Records ............ 18
3.4 Work Flow Process Diagram ................................................................................................... 20
4 Establishing Likely Boundaries for Unknown Values ................................................................... 21
4.1 Database Retrieval ................................................................................................................... 21
4.2 Research Gathering .................................................................................................................. 22
4.3 Determine Boundaries ............................................................................................................. 23
5 Establishing Data Quality and Reliability ...................................................................................... 26
5.1 Document Type Review .......................................................................................................... 26
5.2 Research Gathering .................................................................................................................. 27
5.3 Determine Quality and Reliability Metrics .............................................................................. 29
6 On-going Data Collection............................................................................................................... 35
6.1 Sources of On-Going Data Collection ..................................................................................... 35
6.2 Controlled Process for Management of Change ...................................................................... 35
6.3 Continuous Cross-Check and Quality Control......................................................................... 36
6.4 Isolating and Resolving Inconsistencies .................................................................................. 37
7 Closure............................................................................................................................................ 40
iii
Appendix A: Guidance for Using a Bayesian Network to Bound Unknown Values into Likely Values41
Appendix B: Guidance for Using a Document Hierarchy to Systematically Determine Document
Reliability ....................................................................................................................................... 49
Appendix C: Statistical Method for Tracking Pipeline Data Confidence and Reliability ................... 52
Appendix D: Itemized Lists of Potential Revision Locations for ASME Standards B31.4, B31.8, and
B31.8S ............................................................................................................................................ 60
References ............................................................................................................................................. 69
LIST OF FIGURES
Figure 1-1: Gas Transmission Lines by Decade of Installment ............................................................. 1
Figure 1-2: Prescriptive Quality Assurance Records Requirements from BPVC-III-D3 ..................... 7
Figure 1-3: Partial Standard Repair/Replacement Form from BPVC-XI .............................................. 7
Figure 3-1: An Example of a Pressure Test Dead Weight Chart. ......................................................... 15
List of Tables
Table 2-1: Roles and Responsibilities .................................................................................................. 11
Table 5-1: Method 1 Document Hierarchy Example. Relative Ranking by Source Record Type ...... 30
Table 5-2: Method 2 Document Hierarchy Example. Source Record Bins According to Document
Attributes ............................................................................................................................. 32
iv
FOREWORD
The San Bruno event on September 9, 2010 was paradigm shifting for the pipeline industry. This guideline
represents a continuing effort to learn from our mistakes, to learn from outside sources, to share knowledge
and to improve the pipeline industry in the interest of public safety.
The following individuals are acknowledged for their technical and editorial peer review of this guideline:
David Anderson, Michael Rosenfeld, Marvin Hovis, Joel Brandt, Keith Leewis, David Johnson, Michael
Zerella, and Rick Kivela. In addition, the efforts of Richard Lucas of ASME and Carlton Ramcharran of
ASME ST-LLC are acknowledged for their management of the peer review group, review of the manuscript
prior to publication, editing and document preparation resulting in the publication of this document. Finally,
a special thanks is offered to the pipeline operators who invested in developing processes and technologies
over the past five years that supported the development of this document.
Established in 1880, ASME is a professional not-for-profit organization with more than 135,000 members
and volunteers promoting the art, science and practice of mechanical and multidisciplinary engineering and
allied sciences. ASME develops codes and standards that enhance public safety, and provides lifelong
learning and technical exchange opportunities benefiting the engineering and technology community. Visit
(https://www.asme.org/) for more information.
ASME ST-LLC is a not-for-profit Limited Liability Company, with ASME as the sole member, formed in
2004 to carry out work related to new and developing technology. The ASME ST-LLC mission includes
meeting the needs of industry and government by providing new standards-related products and services,
which advance the application of emerging and newly commercialized science and technology, and
providing the research and technology development needed to establish and maintain the technical
relevance of codes and standards. Visit (http://asmestllc.org/) for more information.
v
ABSTRACT
This guideline provides a recommendation and potential guidance to address observed gaps in data and
recordkeeping practices that are currently prescribed in the ASME pipeline standards B31.8, Gas
Transmission and Distribution Piping Systems, B31.8S, Managing System Integrity of Gas Pipelines, and
B31.4, Pipeline Transportation Systems for Liquid Hydrocarbons and Other Liquids. The authors reviewed
pipeline industry standards and standards from other industries to identify potential best practices and
lessons learned.
In practice, once operators have established upgraded data and recordkeeping systems, the systems will
need to be managed and maintained, or else there would be risk of repeating old mistakes, and the need to
re-create the system from the ground up. A records maintenance team and controlled processes are required
to manage and maintain the systems effectively. This guideline endeavors to summarize guidance that a
pipeline operator could use to enhance a data and recordkeeping structure in accordance with modern
standards.
It is recommended that each operator maintain a chief source of pipeline system information, so he can
make auditable, repeatable, and trustworthy decisions such as those needed for fitness-for-service
calculations and risk assessments. The authors of this guideline recommend a practice of maintaining digital
source records directly linked to specific pipeline components within an operator’s geographic information
system (GIS) (i.e., a pipeline information database). This recommendation is based on a philosophy of
providing the operator’s decision makers easier access to the source records.
Guidance is provided on bounding likely values to address data gaps through research. The purpose of the
research is to gather information that allows the team to assign conservative, realistic ranges of values for
missing parameters.
Guidance is provided to develop a quality and reliability process for pipeline system data. If the existing
quality and reliability determination process is found unacceptable or it is found that one does not exist,
then a new process must be created. A team should be assembled to research information to accurately
develop a quality and reliability determination process. If the existing quality and reliability determination
process is found to be acceptable, then it should be communicated to relevant personnel. This process is to
be adopted and will be used to verify the pipeline information database.
Data collection, as it pertains to information about a pipeline system, has always occurred continuously in
the pipeline industry. Operators and their contractors are constantly recording observations and
documenting measurements as part of field, survey, and testing activities. These new observations and
measurements in effect create new source records with respect to the pipeline information database. An
ongoing data collection process is described to maximize the utility of this data towards filling gaps,
increasing confidence, and maintaining the database into the future.
vi
ABBREVIATIONS AND ACRONYMS
Acronym / Abbreviation Meaning
ADB Advisory Bulletin
AGA American Gas Association
ANSI American National Standards Institute
API American Petroleum Institute
ASME American Society of Mechanical Engineers
APDM ArcGIS Pipeline Data Model
CFR U.S. Code of Federal Regulations
DOT U. S. Department of Transportation
DPI Dots per inch
FAQ Frequently Asked Questions
FDA U.S. Food and Drug Administration
GIS Geographic Information System
GPS Global Positioning System
HCA High Consequence Area
IMP Integrity Management Program
ISO International Organization for Standardization
MAOP Maximum Allowable Operating Pressure
MOC Management of Change
MOP Maximum Operating Pressure
NACE NACE International
(formerly) National Association of Corrosion Engineers
NPMS National Pipelines Mapping System
NTSB U.S. National Transportation Safety Board
O&M Operations and Maintenance
PDF Portable Document Format
PHMSA U.S. Pipeline and Hazardous Materials Safety Administration
PODS Pipeline Open Data Standard
PPI Pixels per Inch
PRCI Pipeline Research Council International
QA/QC Quality Assurance/ Quality Control
SCADA Supervisory Control and Data Acquisition
SME Subject Matter Expert
SMYS Specified Minimum Yield Strength
UPDM Utility and Pipeline Data Model
vii
DEFINITIONS
1 Access: right, opportunity, means of finding, using, or retrieving information
2 Bayesian Network: graphic statistical model that represents probabilistic relationships between
variables. This type of analytic process can model the probabilities of cause and effect relationships in
order to make predictions based on known distributions and values (see Appendix A).
3 Company segment: A natural division within a company that may drive a separate document
structure, such as a legacy acquisition, or geographic region (e.g., “southwest district”) where a
unique document organization may exist.
4 Complete record: a record that is “finalized by a signature, data, or other appropriate marking . ”
5 Conversion: process of changing records from one format to another.
6 Data Gap: missing information.
7 Data mining: the practice of searching through large amounts of information, to locate specific data or
identify useful trends.
8 Database: a large collection of data organized for quick access and retrievability.
9 Database Owner: a person responsible for managing and maintaining the data within the Pipeline
Database for the relevant section of pipe.
10 Decision Maker: The staff and stakeholders whose decisions require the information from the
Pipeline Database, which may include critical operations decisions (e.g., pressure reductions and
fitness-for-service analysis). Examples of Decision Makers include company management, integrity
engineers, and risk engineers.
11 Destruction: process of eliminating or deleting a record, beyond any possible reconstruction.
12 Disposition: range of processes associated with implementing records retention, destruction, or
transfer decisions which are documented in disposition authorities or other instruments.
13 Flowchart: a graphical representation of a process.
14 Media: the physical form of the documentation such as paper, or electronic.
15 Metadata for records: structured information which enables the indexing, sorting, retrieval, and use of
records.
16 Metric: a standard of measurement by which quality, performance, or progress may be measured.
17 Microfiche: a flat piece of film containing microphotographs of the pages of a newspaper, catalog, or
other document. [1]
18 Microfilm: a length of film containing microphotographs of a newspaper, catalog, or other document.
[1]
19 Other Communications (pertaining to regulatory requirements): U.S. Pipeline and Hazardous
Materials Safety Administration (PHMSA) Emails, Letters, Advisory Bulletins, and FAQs.
20 Pipeline Information Database: an organized collection of data specific to the needs and decisions of
the pipeline company.
21 Record(s): Recorded information or data on a particular subject, collected and preserved to
demonstrate compliance with a rule or process requirement. [2]
22 Records Librarian: a person who is familiar with the hard copy records pertaining to the data that are
or will be used to populate the Pipeline Database.
23 Records management: field of management responsible for the efficient and systematic control of the
creation, receipt, maintenance, use, and disposition of records, including processes for capturing and
maintaining evidence of, and information about, activities in the form of records.
viii
24 Records system: information system which captures, manages, and provides access to records over
time.
25 Researcher: a person who is available to conduct the necessary research into the background
information related to the desired information.
26 Schema: logical plan showing the relationships between metadata elements, normally through
establishing rules for the use and management of metadata specifically as regards the semantics, the
syntax, and the optionality (obligation level) of values. [3]
27 SCADA: Supervisory control and data acquisition system, used for monitoring and control.
28 Source Records: a record that is original for the data included within.
29 Subject Matter Expert (SME): An individual recognized as having a special skill or specialized
knowledge of a process in a particular field, or of a piece of equipment. [4] Types of SMEs include
welding and materials experts, pipeline construction project experts, legacy acquired company
experts, etc.
30 Traceable record: a record that can clearly be linked to original information about a pipeline segment
or facility.
31 Useable: (pertaining to records) a record/data that is accessible to the Decision Maker within a
reasonable time period. [5]
32 Verifiable record: a record that contains information that can be “confirmed by other complementary,
but separate, documentation.”
ix
1 PURPOSE AND USE

This guideline provides a recommendation and potential guidance to address observed gaps in data and
recordkeeping practices that are currently prescribed in the ASME pipeline standards B31.8, B31.8S, and
B31.4. The authors reviewed pipeline industry standards and standards from other industries to identify
potential best practices and lessons learned. The industry needs a go-forward approach that will manage
new records and data in a way that will maximize the utility of available records wherever practical, and
improve upon record keeping practices for the future to take advantage of modern technology and lessons
learned.
In practice, once operators have established upgraded data and recordkeeping systems, the systems will
need to be managed and maintained, or else there would be risk of repeating old mistakes and the need to
re-create the system from the ground up. A records maintenance team and controlled processes are required
to manage and maintain the systems effectively. This guideline summarizes guidance that a pipeline
operator could use to enhance a data and recordkeeping structure in accordance with modern standards.
1 .1 Background of Data and Recordkeeping in the Pipeline Industry

The pipeline industry’s data and recordkeeping systems are a product of the industry’s historical practices.
In order to identify ways in which ASME codes and standards can provide guidance to industry, it is
important to first understand historic pipeline record keeping practices and to understand how the industry’s
infrastructure and regulatory history steered the records to their current state.
1 .1 .1 United States
In the United States (U.S.), over one half of the transmission pipeline network in the country was installed
prior to 1970 (and prior to any federal pipeline safety regulations with records requirements). Figure 1-1-1
shows gas transmission pipeline installation activity by decade, which peaked in the 1950s (22.31%) and
the 1960s (23.44%). [6]
Figure 1 -1 : Gas Transmission Lines by Decade of Installment
25% 23.44%
22.31%
20%
Miles Installed
15%
10.53% 10.15% 9.96%
10% 8.47%
7.10%
4.04% 4.01%
5%
0%
Pre 1940 1940 - 1950 - 1960 - 1970 - 1980 - 1990 - 2000 - 2010 -
or 1949 1959 1969 1979 1989 1999 2009 2019
Unknown
Decade Installed
Prior to the first U.S. pipeline regulations in late 1 960s, the industry’s only record keeping requirements
were provided by industry standards. Rosenfeld and Gailing summarized the situation well, “It would be
1
reasonable to expect that a variety of documents related to the design and construction of a pipeline facility
be retained long-term. However, retention of technical documents was not addressed by the engineering
standards of the day. It was generally thought that a copy of the specifications under which the pipeline was
built (and supplemented by commercial documents, e.g. contracts and purchase orders) would generally be
adequate to provide evidence of the work that was done.” [7]
The authors have made similar observations during their professional experiences with records review and
management. There are occasions where operators have stored what are now recognized to be highly
valuable records, but it is highly unusual that they stored them to meet any specific regulatory requirement.
When highly valuable documents are discovered (particularly for pipe that was constructed prior to
regulation and/or pipe that was acquired), they are frequently met with pleasant surprise.
In 1938, American Standards Association ASA B31.1 first required that records be kept on welder
qualifications and their identifying marks. Subsequent revisions expanded welder related record keeping.
In 1955, B31.1.8 (which eventually became B31.8) first recommended basic risk based design concepts
with 4 location class factors. It required the pipeline operator, or contractor, to maintain records related to
welders and pressure testing. The standard was the first to recommend operations and maintenance records
mentioning external and internal corrosion related to leaks and repairs, and inspection reports. A later
revision in 1968 required recordkeeping related to corrosion inspection and leak investigation.
The federal pipeline regulations were passed in the 1960s with the first federal laws effecting liquid
pipelines and the Natural Gas Pipeline Safety Act of 1968. Concurrent to these shifts in pipeline records
requirements and expansion of the U.S. pipeline infrastructure indicated above was large amounts of
population growth over the past 50-60 years. Pipelines had to be re-routed to accommodate the additional
infrastructure (e.g., highways, waterlines) in congested areas, which created more records and/or additional
pipeline system materials to track.
The Natural Gas Integrity Management Rule 49 CFR Part 193 Subpart O was introduced in 2003, three
years after a similar rule for liquid pipelines (49 CFR Part 195). Following the integrity management rules,
operators were more frequently audited, which required them to focus more on material properties as part
of risk modeling and fitness-for-service analysis. The gas (and, similarly, the liquid) integrity management
(IM) rule specified how pipeline operators must identify, prioritize, assess, evaluate, repair, and validate
the integrity of gas (or liquid) transmission pipelines that could, in the event of a leak or failure, affect High
Consequence Areas (HCAs) within the U.S. The IM rules required large improvement in HCA-related
record keeping for most operators in the U.S. [8]
The industry heightened its focus on data and recordkeeping following the San Bruno failure in 2010, when
the U.S. National Transportation Safety Board (NTSB) included in its findings that the pipe had been
incorrectly listed as “seamless . ” The first records quality criteria in the U.S. were provided in 2011, with
PHMSA’s issuance of Advisory Bulletin (ADB) ADB-11-01 [15], which made operators aware that
operational decisions should be based on documents that are “traceable, verifiable, and complete” (PHMSA
provided definitions for the terms approximately 16 months later in ADB-12-06) [16]. Many operators had
to satisfy this requirement1 by locating, sorting, deciphering, and prioritizing decades of legacy and historic
records. It was found that many records, particularly pre-regulation, were not retained as there were no
requirements to keep them on file. Economic shifts have caused mergers and acquisitions (creating
situations where the records may not have been transferred from one operator to the next). There are many
other reasons for loss of records, including lack of retention of hardcopy after poor quality archiving to
1The requirement to review system records and verify MAOP was made into law by the U.S. Congress when the Pipeline Safety, Regulatory
Certainty, and Job Creation Act of 2011 was passed on January 3, 2012. [17]
2
microfilm or digitization. Disasters such as a fire or flood, or clerical mishandling 2 have also destroyed
invaluable records that are costly to reproduce. Operators of these pipelines have been placed in a difficult
situation.
1 .1 .2 CSA Z662
Canadian Standard Association (CSA) standard Z662-15 provides a more flexible view of historical records
with its requirements: “The format and level of detail associated with the material records specified in
Clause 5.7 is not defined and is therefore at the discretion of the company. Although detailed
documentation, such as mill test reports, often provide useful historical data for future reference (e.g., for
engineering assessments pertaining to failure investigations or development of in-service welding
procedures), it is not mandatory that such detailed documentation be retained as part of the permanent
record. It is the intent that basic material data, such as material standards, specifications, grades, and
dimensions, at a minimum, be included in the permanent records.”
While the requirements of CSA Z662-15 may be a bit more flexible with respect to management of the pre-
regulation era records, it allows operators to discard source records and doesn’t require operators to manage
their data with any type of reliability or quality criteria.
1 .1 .3 AS 2885
The Australian Standard 2885 series of standards (AS 2885.0 through AS 2885.5) [9][10][11][12][13][14]
include some extensive lists of specific records requirements. For example, AS 2885.5 – 2012 (Field
Pressure Testing) [14] contains approximately fifty specific and itemized reporting requirements for each
test. It also provides an example of an acceptable format for a pressure test “certificate” that serves to record
the required information.
The AS 2885 standards also require a Records Management Plan, and AS 2885.1 – 2012 (Design and
Construction) [10] states “all pressure-containing materials installed on a pipeline system shall be traceable
to the purchase documentation, the manufacturing Standard, the testing standard, and to inspection and
acceptance documents. The pipeline Licensee shall maintain the records until the pipeline is abandoned or
removed.”
The Australian Design and Construction standard (AS 2885.1 – 2012) makes recommendations about
digitization of records, as follows: “Electronic records that can be accessed by common text, database or
spreadsheet programs are preferred. Where documents are only available on paper, they should be scanned
into an appropriate format.” And the standard requires the following link between materials and source
records: “The identity of all materials shall be recorded and this identity shall include reference to the test
certificates and/or inspection reports. ”
1 .1 .4 International
The International Organization for Standardization (ISO) standard 13623 [18] prescribes generally that
“records shall be kept and maintained throughout (the pipeline’s) lifetime to demonstrate compliance with
the requirements of this international standard” and includes brief statements that require:
 Leak detection surveys.
 Records to demonstrate the system is “operated and maintained in accordance” to operations and
maintenance and integrity plans and they are “effective . ”
 Pre-commissioning and commissioning records.
2“An anecdote reported to [Rosenfeld and Gailing] was an occasion where a clerical worker, instructed to photocopy hydrostatic test records, first
separated the pressure charts from the test report forms which had been stapled together into separate piles, irreversibly breaking the link between
pressure records and test segments.” [7]
3
 Pressure test records, with specific itemized requirements including testing procedure, instrument
calibrations, test charts, explanation and disposition of pressure discontinuities, etc.
 For construction records, it specifies some general document types. It prescribes a specific
requirement that these documents should be made “permanent in reproducible and retrievable
form, ” which seems to imply they should be available upon demand if someone needs them for an
audit or analysis.
While the ISO standards are a bit more specific in some areas (e.g., pre-commissioning/commissioning),
and less specific in others (e.g., operations and maintenance), the standard doesn’t provide much records
management guidance except for the last bullet related to construction records. The ISO standard
recognized a specific need to continuously have access to the construction records. This improvement in
practice of providing more access to records is also recognized by the nuclear industry in the next section.
NACE International (NACE) standard SP0113- 2013 “Pipeline Integrity Method Selection” [19] provides
an overview of available assessment methods and selection guidance to choose an appropriate method to
assess the integrity of liquid or gas pipelines for external corrosion, internal corrosion, and SCC threats.
This standard is primarily focused on records collection and prescribes very little (if any) records
management. It does not prescribe or require any records from the selection process to be stored in any
specific manner or for any specific amount of time. The NACE standard describes selecting a second
assessment based on information from the first but doesn’t describe how to store the information in between.
The reason the NACE document doesn’t prescribe record keeping practices is because it is intended to
complement a robust integrity management and operations and maintenance program, which should contain
recordkeeping requirements. The NACE document provides technical guidance, but it may not be able to
serve its purpose without adequate recordkeeping guidance provided by another standard.
1 .1 .5 R ecen t I m provem en ts
The industry will need a go-forward approach that will manage new records and data in a way that will
maximize the utility of available records as much as practical, and improve upon record keeping practices
for the future, to take advantage of modern technology and lessons learned.
Modern records management programs have been presented at various U.S. pipeline operator conferences
and forums frequently since around 2012. Operators in the U.S. have taken action in response to the
NTSB’s, PHMSA’s, and U.S. Congressional recommendations and rules. Presenters across the industry
have valued peer review of their data and records management review programs. Common themes of the
review programs include organizing and searching source records, creating a document ranking system,
and creating a cross-reference between the source records and the verified pipeline system materials data.
Some operators have found that digitization has allowed them to create a system that is more sustainable
and allows more of the operators’ employees to access data and recordkeeping system, which is perceived
as an increase in value on the investment. Specific examples include operators that have digitally captured
or transferred source records into a records management system, which allows the operator to initially
conduct research efficiently, but ultimately links source records to GIS. The GIS system provides an
interface with the pipeline records system that allows access for a much wider set of employees. These GIS-
based systems can provide more accessibility than most traditional analogue systems (e.g.., like maps, or
alignment sheets) because they can often be navigated more quickly and intuitively and on a system-wide
scale. They can also provide more accessibility than a spreadsheet based cross-reference between source
records and verified data because of the barriers to understanding the spreadsheet cross-reference.
4
Once established, operators have understood these upgraded data and recordkeeping systems will need to
be managed and maintained, or else there would be need to re-create the system from the ground up. A
records maintenance team and controlled processes are required to manage and maintain the systems
effectively.
This guideline endeavors to summarize guidance that a pipeline operator could use to enhance a data and
recordkeeping structure in accordance with modern standards.
1 .2 Review of Data Management Challenges and Solutions in Other Industries
Comparisons of the general data and recordkeeping practices of the pipeline industry to other industries
have highlighted some key observations. The pipeline industry has the following unique combination of
characteristics that create challenges for data and recordkeeping. In the U.S., the pipeline infrastructure is:
 By majority more than 50 years old.
 Materials dependent (i.e. subject to degradation mechanisms like corrosion and cracking).
 Typically buried underground (and can’t be readily seen or inspected) .
 Vastly spread out geographically, and
 Highly critical to society’s safety, environmental, and infrastructure ( e.g., energy, economy)
requirements.
The other industries reviewed for this guideline exhibit some of the same characteristics, but the authors
couldn’t identify an industry which exhibited all of these same characteristics. The authors reviewed
practices within the following industries to evaluate how the recordkeeping practices might be transferred
to the pipeline industry.
1 .2.1 Medical Industry
Recordkeeping practices were reviewed for the implanted medical device sector of the medical industry.
This part of the medical industry shares the following characteristics with the pipeline industry:
 Critical to safety;
 Can’t be readily seen or inspected (once installed) ; and are
 Materials dependent.
Tracking requirements and practices are quite rigid within the U.S. Code of Federal Regulations (CFR) for
implanted medical devices. Specifically, the medical industry requires a unique device number assigned to
all medical devices implanted within patients, so the manufacturer can retrieve manuals, install dates, etc.
for any of its devices for the U.S. Food and Drug Administration (FDA) within ten days of any request.
[20]
This level of tracking is beyond what is required for most of the pipeline industry. For example, the marking
and naming requirement is similar to what is required for valves [21] (where a unique id, nameplate, and
body markings are required), but the requirements for how the information will be tracked after the valve
is installed is currently very general.
The pipeline industry could benefit from a practice of uniquely tracking each component that is installed
(although it would be tedious) and storing the information in a retrievable manner. This practice is more
realistic as a future goal, since it may not be practical to retroactively obtain this level of detail from project
records (i.e., from past or current projects).
5
1 .2.2 Nuclear Industry
Recordkeeping practices were also reviewed for the nuclear industry. The nuclear industry shares the
following characteristics with the pipeline industry:
 Can’t be readily seen or inspected (once installed) ;
 Materials dependent; and
 Highly critical to society’s safety, environmental, and infrastructure (e.g., energy, economy)
requirements.
The nuclear industry in general was reviewed with respect to the recordkeeping requirements for
comparison with the pipeline industry. There were a number of key findings during this review. The nuclear
industry has a substantial number of robust practices related to data and recordkeeping requirements.
Quality Assurance Requirements for Nuclear Facility Applications [22] prescribes requirements for:
 Identification of traceability of items;
 Authentication of records;
 Receipt control of records; and
 Maintenance of records.
Some of the requirements, like “Identification of traceability of items, ” specify that a heat number may be
used to trace a material grade, but they don’t elaborate with specific guidance on how this should be
performed (e.g., within documentation).
“Authentication of records” serve s to set the expectations of a source record (i.e., an “authentic data
source”). For example, “Documents shall be considered valid records only if stamped, initialed, or signed
and dated by authorized personnel or otherwise authenticated. Corrections to documents shall be reviewed
and approved by the responsible individual from the originating or authorized organization.”
These criteria are similar to the types of markings that often appear in pipeline industry construction
documents. The authors of this guideline have reviewed construction and maintenance documentation from
the pipeline industry and subjected it to similar criteria on a project-to-project basis. Although many
operators have procedures, and forms that require signatures, dates, etc., the first requirement for all records
to have such markings was PHMSA’s advisory bulletin ADB -11-01 [15], which required pipeline operators
to have “complete” records to verify their operating pressures.
The Nuclear Standard section on “receipt control of records” designates a role for receiving records, with
the responsibility and organizing them in both temporary and permanent storage. A requirement for
dedicated records personnel is one that many pipeline operators have begun to realize is needed. Operators
have developed teams to “review” records initially, and they have maintained the teams as permanent
records management fixtures once the initial review is complete.
The “maintenance of records” has requirements about accessibility, management of change, and version
control of records, but it doesn’t go into detail about how this will be completed.
In the ASME Boiler and Pressure Vessel Code Section III Division 3 (BPVC-III-D3) “Containments for
Transportation & Storage of Spent Nuclear Fuel & High Level Radioactive Material & Waste, ” [23] there
are also requirements that the pipeline industry could potentially learn from.
This standard also provides itemized tables of lifetime quality assurance records and nonpermanent quality
assurance records, which are as follows in Figure 1-2.
6
Figure 1 -2: Prescriptive Quality Assurance Records Requirements
from BPVC-III-D3
These tables show the exact document set that is required to be available for each nuclear facility. The
nuclear industry’s requirements for records appear far more normalized (i.e., have an effective minimum
uniform dataset that everyone should meet) than the requirements of the pipeline industry. There are other
examples from BPVC-III-D3 that show a higher level of normalization from the nuclear industry, such as
specifications for nameplates, which are required to match a specified form- type for “each containment or
part to which a Certification Mark is applied. ”
There are also examples of a higher standard of normalized documentation in the ASME BPVC-XI “Rules
for In-service Inspection of Nuclear Power Plant Components. ” This standard provides standard forms for
repair/replacement activities that include bill of material tables, which are required to be completed by the
individuals that carry out the repair activities. The following figure, Figure 1-3, shows a direct capture of a
portion of Report of Contracted Repair/Replacement Activity.
Figure 1 -3: Partial Standard Repair/Replacement Form from BPVC-XI
7
It may not be practical to expect a single list of document types to cover all required information about all
types of systems, materials, and components that could populate a pipeline system during its lifetime. It
also may not be practical to require a specific “form- type” of nameplate be attached to all pipe components,
or a standard form to fit all pipeline repair/replacement activities, provided how much flexibility is required
to handle the unique challenges within the pipeline industry. It does make sense, however, for the pipeline
industry to move toward better normalization with improved and specific minimum data requirements and
formats.
The nuclear standard BPVC-XI specifically requires a “records index” in the Quality Assurance Records
section. The requirement states that “The records shall be indexed. The records and the indices thereto shall
be accessible to the Owner, Owner’s designee, and Authorized Nuclear Inspector.” This type of
accessibility allows for more effective decision making because the Decision Maker has access to the record
itself and not just a transcribed version, like an alignment sheet. When a decision (i.e., fitness-for-service
analysis or maximum allowable operating pressure (MAOP) determination) is made based on original
construction documents, the Decision Maker can observe first-hand any items within the documents that
could impact the specific decision, like the type of test that was performed or instrument used. In contrast,
when the information is transcribed onto an alignment sheet, the ability to scrutinize the source is taken
away from the Decision Maker.
Computer networking and software developments of the past one to two decades have provided the pipeline
industry with an opportunity to change the philosophy of what is feasible from an accessibility standpoint
for records. Twenty years ago, it wouldn’t have been possible to provide source records digitally and
instantaneously throughout an organization. In the past few years, pipeline operators have used the records-
8
review initiative, following the San Bruno failure, to index their records thoroughly with document
management systems and are using GIS to interface with records for specific pipeline component(s). This
document management index provides them with an easier and more efficient document review, and the
GIS interface provides the operator and the operator’s designees (i.e., the Decision Makers) more efficient
on-going access to the pipeline system data and records.
1 .3 Recommendations
Practices from industries, such as the automotive, airline/aerospace, and food processing industries, were
also reviewed, but the pipeline industry’s unique set of characteristics made direct comparisons challenging.
For example, the airline industry is highly critical to safety and is materials dependent, but all of the
components of an airplane can be removed and/or inspected from a single hanger and without the need to
excavate. This makes the data and recordkeeping needs fundamentally different from pipelines, which are
hidden from view across a widespread geographic area and must be excavated to be inspected.
Lessons learned from studying international pipeline standards in Section 1.1, Background of Data and
Recordkeeping in the Pipeline Industry, and from other industries reviewed in Section 1.2 have pointed to
the following improvements that can potentially be made in the near term and longer term, including:
1. Normalize the data and record types that should be maintained.
a. The authors propose a legacy normalization process that utilizes a common reliability
ranking system (or document hierarchy; See Section 5).
b. The authors propose a go-forward normalization approach of common minimum data
forms and report templates like those found in the ASME BPVC standards for the nuclear
industry.
2. Formally dedicate resources specifically to recordkeeping roles and responsibilities.
3. Provide more accessibility of the data and records to company deputies.
These lessons are likely to be a continuous improvement effort, as opposed to a one-time fix, but the text
in the remainder of this guideline is provided as guidance to help the industry take the next steps to address
these lessons learned.
The ISO standard 15489-1 would suggest the actions taken to make records more “useable” ensures “a user
has the ability to access the necessary data within a reasonable time period. ” [24]
1 .4 Recommendations for Revisions to ASME B31 .4, B31 .8, B31 .8S, and Future
Work
“Prior to pipeline regulations and modern evolutions of the ASME Code, there were no mandated levels of
data quality, which suggests that the prevailing levels of quality in present data as a result of historical
practices are no longer viewed by the public or regulators as adequate. Data quality requirements set forth
by the Code today are overly general and are inadequate for the future. The issue then is what can ASME
do to enable an operator to evaluate and assure data quality?” [25] ASME B31.4, B31.8, and B31.8S were
reviewed for potential locations for revision based on the recommendations and lessons learned described
in previous sections.
Current Revision Recommendations
Two locations in ASME-B31.8S show direct relevance, above others, to the lessons learned of the previous
section. Paragraphs 4.3 Data Sources, and 4.4 Data Collection Review and Analysis each describe system-
wide research efforts to collect and evaluate pipeline system data. Section 4.4 recommends “A plan for
collecting, reviewing, and analyzing the data shall be created and in place from the conception of the data
collection effort.” ASME-B31.8S has provided specific guidance based on proven industry experience
collecting, reviewing, and analyzing data (and rating it for reliability). It also provides specific guidance on
9
how the data should be stored and made accessible to company Decision Makers, how they should be cross-
checked and updated, and how the team in charge of maintaining this pipeline system information database
should be structured.
The contents in Sections 2 through 6 and Appendices A, B, and C of this guideline have more current data
collection and recordkeeping guidance than what is currently in ASME B31.8S Sections 4.3 and 4.4. The
plan consists of the following (by guideline section):
 Section 2 - An overview of the program with roles and responsibilities.
 Section 3 - A description to structure the pipeline information database to optimize access to source
records.
 Section 4 - A process to determine likely boundaries for unknown values.
 Section 5 - A process to determine quality and reliability of data based on attributes of the source
records.
 Section 6 - A process for on-going data collection and management of the records database.
 Appendices A, B, and C to support the main text.
The most relevant location for this guidance would be as a supplementary appendix to these sections in
ASME B31.8S. Sections 4.3 and 4.4, which would remain in-full and a reference to the supplementary
appendix could be added as the “recommended guidance to develop a plan for collecting, reviewing, and
analyzing data. ”
Future Revision Recommendations
The guidance within this document is a product of the authors’ and industry’s experienc e over the last half-
decade and was developed for MAOP verification. The focus of these efforts was typically on construction
and maintenance records, which included pressure tests and material installation records. While the
guidance in this document has been tested thoroughly for pipeline system materials and pressure test
records, it has not been tested for some other types of data that ASME B31.8S Sections 4.3 and 4.4 require
(e.g., SCADA, soil resistivity/moisture levels, one-calls). It is the authors’ hope that the guidance and
philosophies regarding accessibility and dedicated recordkeeping resources contained in this guideline
could extend beyond materials and be adjusted to apply more effectively to other data types as future work.
During the review, there were many locations where records requirements were “overly general” as
described above. A prime example follows, where the requirement is clearly adamant about the importance
of detail, but provides no guidance on what details are needed or how they should be stored or tracked:
“Records shall be made covering all leaks discovered and repairs made. All pipeline breaks shall be reported
in detail.” [26] Providing a form with minimum data requirements could provide operators with a valuable
information gathering tool (particularly smaller operators who may not have the resources to call upon
SMEs for each occurrence).
Meeting the spirit of the normalization standard set by the ASME-BPVC standards for the nuclear industry
would require many specialized forms and report templates with minimum data requirements. Providing a
recommendation on how to normalize each type of data needed for operations and maintenance and
integrity and risk management practices in ASME B31.4, B31.8, and B31.8S is not possible within the
scope of this guideline. Multiple subject matter experts will likely be needed to meet the normalization
requirements consistent with the level of the ASME BPVC standards for the nuclear industry.
The remaining notable paragraphs from ASME B31.4, B31.8, and B31.8S that are candidates for
normalization were itemized into tables shown in Appendix D.
10
2 PROGRAM OVERVIEW AND TYPICAL ROLES AND RESPONSIBILITIES
An operator who is interested in upgrading or maintaining a modern data and records management system
as described in this guideline should include personnel and resources to fill the roles and responsibilities
indicated below. These roles and responsibilities are referenced throughout the processes described within
this guideline. The following roles3 will need to be delegated to the appropriate personnel upon starting this
process:
Table 2-1 : Roles and Responsibilities
Role Responsibility Examples Title
Database Must be responsible for managing and maintaining the data within the  Pipeline Integrity
Owner Pipeline Database for the relevant section of pipe. Manager
 Pipeline Integrity
Engineer
 Risk Engineer
Records Collection, storage, organization, and manipulation of data and/or  Pipeline Integrity
Librarian documents (a data mining expert). Must be familiar with the hard copy Engineer
records and “soft” data that is (or will be) used to populate the Pipeline  Risk Engineer
Database. Must provide technical support to the Research teams about  Administrator
source records, their locations and formats.  GIS specialist
Subject Matter Provides extensive knowledge about the relevant integrity practices,  Pipeline Integrity
Expert (SME) pipeline construction and procurement, materials, company knowledge, etc. Engineer
Typically provides technical support about this knowledge to others in the  Procurement
company. personnel
Records Oversees teams of Researchers and Data Miners and performs quality  Pipeline Integrity
Engineer assurance (including process controls and design) and quality control on Engineer
their work.  Risk Engineer
Researcher/ Conducts the necessary research into the background information related to  Pipeline Integrity
Data Miner records gaps or document reliability. Must be familiar with industry Engineer
standards, compliance and regulator communications, and/or company  Documents and
information and documentation. Records
Reviews new records/information and compares them to the database. Management Team
Carries out the MOC process listed within this guideline and is responsible  GIS specialist
for resolving data discrepancies.
Decision Maker Makes decisions that require the information from the Pipeline Database,  Company
which may include critical operations decisions (e.g., pressure reductions management
and fitness-for-service analysis).  Pipeline Integrity
Engineer
 Risk engineers
Company Ensures compliance of the pipeline system, which includes the records and  VP of Engineering
Management data management. Responsible for ensuring others on this list have the  Compliance
resources necessary to carry out their roles and responsibilities. Manager
 Pipeline Integrity
Manager
3Depending on the size of the operator and the number of records to maintain, it may be appropriate for
some individuals to fill multiple roles.
11
3 COMPANY PIPELINE INFORMATION DATABASE
It is recommended that each operator maintain a chief source of pipeline system information, so they can
make auditable, repeatable, and trustworthy decisions such as those needed for fitness-for-service
calculations and risk assessments. The authors of this guideline recommend a practice of maintaining digital
source records directly linked to specific pipeline components within an operator’s GIS. This
recommendation is based on a philosophy of providing the operator’s Decision Makers easier access to the
source records.
This system might not fit every operator’s system. There may be some operators that can provide their
Decision Makers ready access to all source records in a way that can be as efficient as directly linking the
source records within GIS. This could be the case for a very small operator where records are always
immediately retrievable for the entire system (and where it may not be practical to invest in the required
network infrastructure and software considering the benefit).
This section describes guidance on how to create a company pipeline information database with digital
source records linked within GIS. The guidance below is drawn from the authors’ and the industry’s
experience working with pipeline system materials records in the past half-decade to calculate design
pressures as part of verifying MAOP. In many cases, the philosophies provided in this guidance could be
extended beyond materials and applied to other data types as well (e.g., SCADA, soil resistivity/moisture
levels). The Work Flow Process Diagram in Section 3.3 summarizes the process described.
If an operator elects not to maintain this type of database, there may be many parts of this section that will
not apply. However, as long as the operator maintains a chief source of pipeline system information
(e.g., master alignment sheets) most of the guidance in subsequent sections will apply. This operator must
know that when the authors reference the “company pipeline information database . ” the guidance is applied
to the “chief source of pipeline system information. ”
3.1 Locations and Formats of Pertinent Data

The first steps in preparing a database of digital source records are to identify the types of records that are
expected to be included and imported in the records database and to identify the locations of these records
during a “pertinent data search, ” so they can be organized for import systematically. A key step in this
process is recognizing which records are, or are not, pertinent or relevant. This, and other key steps, should
maximize the value (amount of information identified as pertinent) relative to the effort required to retrieve
and review records.
When identifying records that are expected to be imported and included, an operator should target locations
of records that are essential to material properties verification. The locations may be buildings that serve as
maintenance hubs or offices that belonged to a primary project manager (or project management team) for
construction projects. The locations may also include record storage locations (either vendor or company
owned).
The search team should research how the records have been stored and transferred through the history of
the organization. Some key events that impact record locations are mergers and acquisitions (both of
companies and of assets), closing offices down, or moving branch offices or headquarters.
This search usually begins at a primary, or representative, document storage location. Consideration should
be given to communication and coordination with local personnel prior to arriving onsite. In many cases,
the search team will be working with and removing documents that are critical to onsite personnel and/or
local operations. Gaining cooperation from the onsite personnel could be critical to having a successful
project.
12
Once the team arrives at a records storage location, the first priority is to become familiar with the onsite
file organization. The onsite file organization can help discern between large groups of pertinent and non-
pertinent information. For example, areas of the onsite file organization designated for “office maintenance
files” are non-pertinent, and the team can save large amounts of time by avoiding further review. The
effectiveness of this strategy is largely dependent upon the efficiency and organization of the onsite file
structure.
This initial search should be a broad search and thought of as including or precluding data from being
imported to the records database on a box-by-box, shelf-by-shelf, or bin-by-bin level or larger. It is
important to note that this philosophy will also apply to softer media locations, such as compact disk files
or microfilm and microfiche. If encountered, the team should endeavor to group records from these other
media types and include or exclude them, when practical, in the same manner as the box, shelf, or bin level.
A search team should consist of personnel familiar with documentation for pipeline construction and
maintenance. The team should be led by a Records Engineer with data mining experience, who can identify
pertinent data types for the search. An ideal candidate for the team leader is an engineer that has led records
verification projects, or an engineer with material experience as a project lead in pipeline construction and
maintenance projects (or specifically the types of data that is being targeted by the search).
Project numbers can be the primary tool for linking (i.e., establishing traceability with) source record
documents, such as purchasing records, hydrostatic test records, mill test records, and construction files to
specific components or locations within a pipeline system. If the company has tracked information related
to project numbers, this information could be useful during the pertinent data search. The utility of a list for
this purpose will depend on the type of data that was tracked along with each project number. For example,
if the operator tracked project type, or perhaps a short description with each project number, this information
could be used to target projects that are more pertinent to the data needed or rule out non-pertinent data
(i.e., a right-of-way land ownership dispute survey is not likely to produce any pertinent information related
to pipeline system materials data).
Regardless of how helpful the project list is during the pertinent data search, it is good practice to begin a
project number list during this stage in the project. It will also be used during later stages. A project number
list should attempt to track at a minimum:
 Project number
 Project date (or year)
 District/area
 Pipeline name(s)/number(s), and
 Location/stationing.
Some operators have had success creating such a list from accounting records in addition to project
management records. The project list can also be expanded as the document search proceeds. A robust
project list will support locating pertinent data, document indexing (as metadata), and data mining of the
records. The latter two of these items are discussed in subsequent sections.
The last type of check that can be used to discern pertinent from non-pertinent data is the quick peek check.
This is the last filtering check if the onsite file organization and company project list haven’t removed non -
pertinent data at the box/shelf/bin level, then the team members will do a manual quick scan, or peek at the
contents of each box, shelf, or bin to determine if the data is pertinent or non-pertinent. Each box, shelf, or
bin that can’t be concluded as non-pertinent will be imported to the records database.
13
3. 2 M o d ern i zi n g an d Di g i ti zi n g Data an d I n form ati o n
Once the proper records have been located, they should be digitally captured (i.e., scanned). Care must be
taken to ensure that there is sufficient resolution to make quality copies. In the authors’ experiences,
300 PPI4 is an acceptable resolution for image capture. It provides adequate resolution, while not
overburdening storage and network infrastructures with its file size requirements. It is important to have
color scanning capabilities, but it is preferred to have the option to scan in color only when needed. This
will allow for faster load times for black and white documents, whenever possible.
The authors consider the following guidelines [27] good practice for pipeline records digitization efforts:
 Digitize to the original size of the records.
 No cropping allowed – records should be complete.
 Back reflection of originals – Back all originals with bright white opaque paper.
 Scanning oversize items – oversize items may produce very large file sizes; legibility of small
characters may need to be evaluated for resolution versus file size needs. 5
Figure 3-1 is an example of a pressure test dead weight chart, which is scanned at 300 PPI and is color-coded
to indicate temperature in one color and pressure in another. This image shows that the image quality is
certainly sufficient for retrieval and review by Decision Makers. The color scan was critical to this image
(to identify the red temperature trace with respect to the blue pressure trace). There is a portion of the chart
that is truncated and although it appears there is no critical information missing, this should be avoided
because future reviewers may wonder about the missing portion. Some digital document management
systems will allow quality control comments to be stored as metadata with each record. These comments
can be populated to assure future reviewers that there was no critical information missing, or no different
information in the original. Populating this field can be time consuming, but it can provide useful quality
control information if an operator would like a permanent digital file structure (as discussed below).
4 PPI (pixels per inch) is often used interchangeably with DPI (dots per inch). PPI refers to the resolution
of an electronic image, whereas DPI refers to the resolution of a printed image.
5 Note that for pipeline records the size of the oversize file is often justified because the small print is often
critical.
14
Figure 3-1 : An Example of a Pressure Test Dead Weight Chart.
3.2.1 Document Indexing with Metadata

Digital document management systems, such as SAP®, OnBase™, and SharePoint™, are commercial
options that have been successfully used to organize documents by document indexing with metadata in
large-scale industry projects. Each has its own strengths and limitations. It is recommended that operators
select a digital document management solution that is most compatible with their organization (or that they
find a digital document management solution within their organization that is suitable to the requirements
of this section).
The following parameters have been used by operators to select digital document management solutions:
 Security features
 Data backup features
15
 Customization capabilities for user-access (e.g., can it restrict certain documents, or groups of
documents from certain users? Can it allow some users read-only access?)
 Ability to integrate with other software packages
 Ability to integrate across the operator’s organization (i.e. , is this a product that multiple
departments will use?)
 Platform sustainability (is it likely to be supported by the publisher ten years from now?)
 Ease-of-use
 Cost (including on-going subscription fees)
 Data limitations
 Quality and capabilities of service providers associated with the platform
 Options for remote access (e.g., local network, intranet, or internet)
The records system described in this guidance report “requires a set of metadata and unique naming
convention for each scanned document. The metadata and naming convention system should be designed
by a (Records Engineer) .” An effective indexing system should be consistent, understandable, and intuitive
for Researchers and Decision Makers to target and retrieve documents. The metadata should allow the
digital file structure to be searched and grouped dynamically. The metadata should include:
1. Unique record identifier (Document Identification)
2. Pipeline name or number (whichever is more likely to sustain over time)
3. Project number
4. Document type
5. Date created
6. Document author
7. Milepost or station number
8. Operator name
9. Hard copy location
10. Equipment or Joint Identification, if applicable
11. Version Number / current.
The first five indexing fields are the most important to an effort to systematically review pipeline system
material records for design pressure data to contribute to MAOP verification. It is highly important to
understand the chronology of the projects that occurred on each pipeline in order to understand the current
configuration of pipeline materials (and replaced pipeline materials). The first five indexing fields allow a
Researcher to understand the order of the projects and research where each project occurred within each
pipeline.
Once a unique record identifier is assigned to a document, some operators have preferred to place the unique
record identifier directly onto the hard copy document (i.e., with a barcode). This has provided them
assurance that the hard copy will always be linked to the digital scan.
Other operators have felt that the bar code covers a portion of the document impacting its integrity. These
operators have preferred to maintain the link between the hard copy and digital scan through metadata, by
diligently tracking the hard copy location field, or through controlled version numbers.
As metadata and document naming conventions become more commonplace in the pipeline industry, there
may be off-the-shelf options that are sufficient for some operator’s needs. It would be advisable to conduct
a short industry search to see if there are off-the-shelf metadata and naming convention options that an
operator could incorporate. Most operators will have a sub-set of company specific naming conventions
that will need to be taken into consideration (for example some operators may use “Valve Section” as a
16
critical part of their stationing, while others may require a date to properly apply a “Version Number”).
While off-the-shelf solutions can provide efficiencies, consideration should be given to implementation for
a specific operator.
3.2.2 Digitization Quality Assurance
Large scale digitization efforts can be completed with the assistance of a scanning vendor. The scanning
vendor will have procedures to carefully deconstruct the project boxes, folders, and binders to digitize the
documents within. The indexing of metadata will be completed primarily by the scanning vendor (with
some training on identifying the metadata within the pipeline records from the operator and/or Records
Engineer).
The scanning vendor typically performs pre-digitization quality control checks. There can also be additional
special handling requirements when analyzing old pipeline records. Some paperwork may not have been
archived properly and may have been exposed to mold or degradation. Some paperwork may have faded
over time, which will make a poor scan.
A scanning Quality Control (QC) process could check the following items to the original hard copy to
ensure a quality scan process:
 Is the image accurate?
 Has the entire image been captured?
 Has it been correctly oriented?
 Is it centered?
 Is the order of pages correct?
 Are there contrasts or uneven tones?
 Is all writing legible?
 Is the color image true to the original (and how close does it need to be)?
 Has it been indexed correctly?
 What is the acceptable error rate for both accuracy and quality?
Some scanning vendors offer double-blind metadata indexing to double check their own work to ensure a
very high level of quality. If the QC check is not required to be 100 percent, each operator must select its
own acceptability levels for the QC check.
3.2.3 Disposition – Disposal of Data
Pipeline operators may be tempted to destroy original documentation once it has been digitized. Holding
on to the original document could hamper document control, or it may be costly to manage. However, the
original documentation could prove useful if future potential purchasers of assets have conflicting digital
document policies, potential digital format conflicts, or if they wish to verify the digital document quality.
It is conceivable that a potential sale would not occur if the asset information could not be independently
verified.
There should be a plan in place for disposal of records, including who should make the decision, and
justification standards. At the minimum, the plan should include a QC check of 100 percent of the digital
copies with respect to the hard copy for the items mentioned in the section above, a plan to ensure the
integrity of the digital copies with back-up storage location, and a plan to ensure the format of the digital
records is stable from obsolescence.
17
3.3 Structuring Data and Data Sources to Maximize Access
Structuring data and data sources to maximize access is one of the most important steps an operator can
take with its source records. This step can unlock the true potential and utility of a robust records library by
putting the information in the Decision Makers’ hands at the time when it is needed the most.
A Decision Maker can use GIS software to navigate a map interface to a particular section of pipe or a
particular fitting and instantly retrieve attribute data from the pipeline system database (e.g., PODS, APDM,
UPDM), which is a core function of GIS.
A primary recommendation of this guideline is to increase access to the source records that verify the
pipeline system information (e.g., original construction records). Some operators are currently endeavoring
to meet this recommendation by including “direct links” within the GIS interface to each verifying source
record for the information in the pipeline system database. That is, once a Decision Maker navigates to a
section of pipe and retrieves the material properties, he or she will also be able to instantly click and view
the source records that verify each material property. The authors of this guideline have helped develop
successful applications of this structure for pipeline system materials and pressure test information,
including:
 Outside diameter
 Wall thickness
 Pipe Grade
 Seam type
 American National Standards Institute (ANSI) rating (if applicable)
 Test pressure
 Test date
 Test duration
 Test company
 Test medium
 Test employee/company.
This structure has also been applied for other information that appears in pipeline system data models and
can be obtained from construction and maintenance files, such as coating type, girth weld type, and year of
manufacture. It may be possible to use this type of structure (i.e., GIS interface with a link to other source
data sets) for most, if not all, data sets that appear in the pipeline database model.
3.3.1 Data Mining and Gap Filling to Establish “Direct Links” to Source Records
The source records must be linked to the pipeline system attribute data by performing systematic records
research (a.k.a. “data mining”) . This can be very labor intensive, so it is recommended that a team of Data
Miners conduct this step with oversight from a Records Engineer.
The data mining efforts can benefit from the digital document database in several different ways once it has
been created. The digital document database can serve to centralize documents, whose hard copies were
previously spread across several document locations, into a single data mining location.
The document index “metadata” that was recommended in Section 3.2.1 (and assigned by a scanning
vendor) will also assist during data mining. The Data Miners should use the metadata fields to filter through
the records and target the most relevant documents to research.
For example, the Data Miners could sort to a particular pipeline number to review all of the project numbers
that have been indexed with that pipeline. The Data Miner could also review the dates indexed for each of
18
the project numbers, which gives an idea for the chronology of the projects. Organizing the chronology of
the projects within each pipeline is an important step in the data mining process. Many Data Miners have
found that researching the projects in their chronological order provides a logical reconstruction of the
pipeline materials verifying source records.
The project list that was mentioned in Section 3.1 could also be used during the data mining process. The
project list can be used to help organize and perform QC checks on the chronology of the projects within
each pipeline.
Section 5 is a discussion and methodology to create a data reliability index based on source record quality.
This methodology is based around creating a document hierarchy or document ranking system. The
document hierarchy or document ranking system plays a critical role in the data mining process, because it
guides the Data Miners to the document types that could provide the most reliable data (i.e., the “high value
targets”). It also provides a reliability metric for the information retrieved from source records, which is
based upon the quality and characteristics of the source record itself. In general, if an operator is dedicating
resources to perform a large-scale data mining effort, a document hierarchy will also be used to create a
more efficient data mining process and to assign a reliability ranking to the data retrieved. 6
Combining a document hierarchy with a document management system that indexes “document type”
metadata can be a substantial time saving measure if the metadata “document types” have been indexed
reliably to show the same types of document types indicated by an effectively developed document
hierarchy. More details on data mining with a document hierarchy are presented in Appendix B.
The final goal of the data mining effort is to store the direct links between the attribute data and the verifying
source records within the GIS system itself. During the data mining process, it may be easier to track results
in a Microsoft Excel™ spreadsheet. This will allow the results to be adjusted and QC checked easily by the
overseeing Records Engineer, when necessary. Once the results are complete and QC checked, they can be
loaded into GIS.
If the GIS system is populated with data, it may need to be reconciled with the data mining results.
Depending on the level of confidence in the legacy GIS data, the data mining team may elect to incorporate
the GIS data earlier in the data mining process and reconcile it along with the records review. If there is
very little, or no, confidence in the GIS data, the team may elect to replace it entirely with the results of the
data mining efforts.
Some GIS systems are directly compatible with digital document management systems (for example,
OnBase™ publishes a specific module for ESRI), while other GIS systems (e.g., Uptime™) will accept
URL address links to documents. Either method will allow documents to be directly linked. This type of
records management (like a GIS system itself) must be maintained by the Database Owner or the Records
Librarian. Section 6.1 discusses the resourcing requirements for on-going records maintenance in more
detail.
If the data mining efforts fail to fill all gaps, it is likely that default values (see Section 4.3) will be used to
supplement the missing information. Use of default values must be specifically tracked. If new
information becomes available, the default values may need to be systematically changed, or in some
cases replaced with data from source records.
6Pipeline operators in the U.S. have been using this approach to satisfy PHMSA’s requirement that pipeline
system design data that contributes to verifying MAOP be “traceable, verifiable, and complete.”
19
PTB-12-2017: Guidelines for Addressing Data Gaps and Recordkeeping for ASME B31.4, B31.8 and B31.8s for Pipeline Integrity Management
3.4 Work Flow Process Diagram

Structure to Data and Da ta Sources to
Loca ti ons of Pertin en t Da ta Di giti za tion a nd /or I mage Ca pture
Ma ximize Access
Ch ie f so u rce of pi pe l i n e
s ys te m i n form a tio n a n d Da ta mi n i n g to e sta bl i s h “d i re ct li n ks ” to
d i g iti z a tio n i n i tia ti ve s ou r ce re co rd s u s in g:
-I n d e xe d me ta d a ta
-A proj e ct ch ron o lo gy fo r e a ch p ip e li n e
H a rd c o py o r
so ft m edi a -pro j e ct l i s t
( e. g. m icro fi ch e, -Do cu m e n t h i e ra rch y ( S ec tio n 5 )

I d e n tify l oca ti on s of pe rti n e n t
c om pa c t di sc s, et c. )?
d a ta an d be g in a l is t of p roj e cts
H a rd C o py So ft M ed ia
R em ov e n on -pe rti n e n t
d a ta at th e “bo x/s h e l f/
bi n ” l e ve l E n g i n e e r QC ch e ck o f d a ta
D igit a l im a ge
m in in g re s u l ts
Al tern a t e p roc ess by
c a pt ure ( i . e.
m edi a t ype
sc a nn in g)
Ca n a “box /s h e l f/bi n ”
be ru l e d o u t ba se d o n on s i te
fil e org a n i z ati on , pro j e ct
i n form a tio n , or a Loa d /re co n ci le d a ta mi n i n g re s u l ts w ith
“pe e k ” c h e ck? “d i re ct l i n ks ” to so u rce re co rd s ( a n d
po te n ti al l y w ith d a ta re l i ab il i ty ra n k i n g s
ba s e d on th e pr oce s s from Se cti on 5 )
I nd ex in g w it h i n to G I S.
m eta da t a a n d
NO
rec ord QA/QC
Pre p ar e o r m a rk th e “b ox/s h e l f/bi n ”
for i mp ort a n d d i g iti z a tio n

YES
Sto re in d oc um en t Co mp an y Pi pe l i n e I n form ati on
m a na ge men t syst em Da ta ba s e E s ta bl is h e d
R em ov e th e “box /s h e l f/bi n ” from
pro ce s s
En gin eer wi th d a t a m in ers Sca nn in g ven d or En gin eer wi th D a t a m in ers

Q
20
4 ESTABLISHING LIKELY BOUNDARIES FOR UNKNOWN VALUES

Establishing likely boundaries for unknown values will generally follow an operator’s decision to initiate
efforts to address gaps within the company pipeline information database. This decision is often based on
a substantial change within the company or industry such as:
 Merger or acquisition
 Large expansion of assets
 Change in regulations, or industry standard.
This decision may also be the result of an identified deficiency, for example:
 Regulatory audit finding
 Self-audit finding, or self-identified gap.
In the worst case, this decision may be the result of a failure. The process to establish likely boundaries is
described in Sections 4.1 through 4.3 and shown schematically by the flowchart in Section 4.4.
4.1 Database Retrieval
The Database Owner should be responsible for data gathering with potential support or input from company
SMEs. The data targeted for gathering and the specific SME support will depend on the nature of the gap
identified. Some gaps may relate to a short replacement of one specific area on a pipeline, which may only
require input from one engineer or project manager. Other gaps may be related to a system wide deficiency
in verified pipeline materials information, which might require input from the pipeline integrity manager
or a pipeline materials expert.
The Database Owner will capture a broad view of the data available by exporting all relevant known
information from the pipeline database (i.e., if gaps have been identified with pipe material properties, then
the initial step is to export and review the material properties that are present in the database). If the
Database Owner suspects he/she is missing relevant information within the database, an SME may be able
to help identify the location(s) of the relevant information. If information is still missing after the SMEs
have been informed, it should be assumed as a gap.
The Database Owner, SMEs, or both will examine the exported data for missing and unknown values that
would indicate the information needs of the company haven’t been met. The following questions may help
determine if the exported data meets the information needs:
 Does the data contain gaps that substantiate the concerns that initiated the review?
 Does the data contain gaps that prevent the operator from determining required operating
parameters, such as maximum allowable operating pressure?
 Does the data contain gaps that prevent the operator from performing accurate risk assessments on
the pipeline system?
 Does the data contain gaps that prevent the operator from performing fitness-for-service
assessments (e.g., ASME B31G burst pressure evaluation)?
 Does the data contain gaps that prevent the operator from effectively implementing their integrity
management program or other necessary programs and procedures?
If the Database Owner and SME agree that the exported data does not meet the operator’s information
needs, then conservative boundaries of likely values must be identified to allow critical calculations and
evaluations to be performed when data gaps exist. The Database Owner will proceed to research the
appropriate range(s) for values of the unknown parameter(s) in the next section of this guideline.
21
If the Database Owner and SMEs agree that the exported data meets the information needs of the company,
then there are no gaps in the exported data and the Database Owner will proceed to the quality and reliability
review covered in Section 5.
4.2 Research Gathering
This section is the research phase and is expected to take the majority of the time when bounding likely
values. The personnel required for this portion are the Database Owner and Researcher(s) with input from
an SME.
The purpose of the research is to gather information that allows the team to assign conservative, realistic
ranges of values for missing parameters. For example, if an operator identifies a gap in wall-thicknesses for
parts of the pipeline system, this could trigger a review of historical manufacturing practices (American
Petroleum Institute (API) API 5L and API 5LX) to identify the range of wall thicknesses provided by
industry standards. It may also trigger a review of historical purchasing records to evaluate the range of
wall thicknesses that have been purchased (and documented). The company may also survey field engineers
and technicians about the range of wall thicknesses within the system according to their knowledge.
Note that the SME provides input and support in multiple ways in this section. It may be appropriate for a
company’s SME to provide the Researchers with some background knowledge on the research topics and
provide some direction prior to conducting research (i.e., which sources should be reviewed first, help
estimating research efforts, etc.). In some cases, the SME may be able to use the exported data discussed in
the previous section of this guideline to help narrow the scope of research to make the efforts more efficient.
The SME may provide guidance for the Researchers during their review, including insight and confirming
the research results (Was the research material gathered from the right locations? Was the information from
a credible source?).
The research should include reviews of source material from the following four categories: 7
1) Industry standards should be reviewed to establish standard practices and known physical
limitations related to the gap data. Typical industry standards may be published by the following
sources: ASME, API, Pipeline Research Council International (PRCI), and NACE.
2) A compliance review should be performed, which should include regulatory requirements and
communications . This review will ensure that any relevant regulatory concerns are addressed
and that the boundaries placed on likely values do not create a compliance issue. Sources for
regulatory requirements include CSA Z-662, the CFR, and applicable state regulations. Sources
that should be reviewed for regulatory compliance guidance include PHMSA Advisory Bulletins,
frequently asked questions, and PHMSA’s integrity management inspection protocols.
3) Company information should be reviewed to understand the range of documented values within
company records. Pertinent records may include: As-built drawings, Purchasing Catalogs,
Process and Procedure Documents, External and Internal Studies, and Operating Records (e.g., of
current or past MAOP/MOP settings).
7 This step is highly critical for each gap type. For example, stop-gap values determined from a paper study
described in this section (including Bayesian Network approaches) may not meet minimum industry safety
requirements or U.S. regulatory requirements for “traceable, verifiable, and complete” until fully verified
with field measurements.
22
4) Subject Matter Experts with extensive knowledge about the pipeline system and/or subject
matter should be consulted to confirm the ranges of values are realistic and applicable to the
pipeline system. In some cases, the SMEs can also provide ranges of values based on their
knowledge and experience. SMEs that may help include, but are not limited to, pipeline and risk
engineers, purchasing, GIS, consultants, pipeliner/technician, etc.
Examples are provided above for each of these categories, however there may be other relevant
organizations and sources that have relevant information. The utility of the research performed on each
category will be depend on the nature of data gap, reinforcing the importance of consulting an SME prior
to starting the research to provide the Researchers with some direction and background knowledge.
Once ranges of values for missing parameters are identified through the research described above, the
Database Owner will use the research results to determine the boundaries to use in the company database
in the next step.
4.3 Determine Boundaries
The personnel required to determine and approve the boundary limits are the Database Owner, SMEs, and
potential Decision Makers including Company Management.
A list of all possible values (or ranges of values) will be established for the unknown data points. Unrealistic
values will be ruled-out, whenever possible, and justified based on the research that was gathered and input
from SMEs. The remaining values will be the possible “default values” that will be used with conservatism
in the case of a null or gap in the company database.
It is advised to determine at least two individual values for each potential gap or null in the company
database. For example, if the operator has identified a gap in wall-thickness values in parts of their pipeline
system, he may elect to establish default wall thickness values for each pipe size: One that represents the
“most likely case” value and the other that represents the “worst probable case” value.
The “most likely case” may be the wall-thickness that is most common for a particular pipe size, while the
“worst probable case” may be the thinnest wall observed in API 5L and API 5LX for each pipe size. By
selecting the “most likely case” and “worst probable case” values , the team can evaluate the impact of the
uncertainty against conservatism for the default value(s). In some cases, conservatism may dictate that the
operator should only use the “worst probable case” and not justify consideration for a less conservative
“most likely case. ” In other cases, for example in some risk models, it may be more appropriate to utilize
“most likely cases” to ensure the “worst probable cases” don’t skew the results away from true issues of
concern.
The “most likely case” and “worst probable case” values can be determined by one of the following
methods:
 Review the range of possible “default values” with an SME who is knowledgeable about the subject
matter to select what the “most likely case” and the “worst probable case” value should be.
 Using information from other similar pipelines or segments of the same line to determine the most
common (“most likely case”) value and maximum/minimum (“worst probable case”) value across
the pipeline system. The values that determine what a “similar” pipeline/segment is will depend on
the nature of the data gap.
 Create a Bayesian Network Model to determine the uncertainty between the different values and
then use those results to select the average ( “most likely case”) value and maximum/minimum
(“worst probable case”) value. Bayesian networks are powerful tools for filling gaps in pipeline
databases, particularly when used for risk model applications. Bayesian networks calculate the
23
probability of different states (values) of unknown parameters. This method is most effective when
there are multiple known variables that have an implied or direct influence on the unknown (gap)
variable. Bayesian analyses have been used for applications in the pipeline industry during risk
determination, root cause analysis, or maintenance prediction. More details are given in
Appendix A.
Regardless of the method used, it is imperative to document the decision making process. The reasoning
for choosing and ruling out values should be tracked, so that if additional information becomes available,
the data can be revised.
As discussed in Section 3.3.1, use of default values must be specifically tracked. If new information
becomes available, the default values may need to be systematically changed, or some cases replaced with
data from source records.
The options presented above increase in detail and comprehensiveness from top to bottom, so the team
should select the appropriate method for each data type that is reviewed. Once the “most likely case” and
the “worst probable case” values are selected, they should be reviewed for approval.
 If Approved: The previously unknown values have been assigned a “most likely case” and a “worst
possible case. ” The team will create a report to document the research and decision making criteria
used and any tables or lists that show the most likely and worst case values used. It will then proceed
to the quality and reliability review covered in Section 5 8.
 If Rejected: One/Both of the values selected do not meet one of the decision criteria. The Decision
Makers and SME will provide feedback to the Database Owner and Research team and they will
need to return to the research stage in order to obtain additional information to address the feedback
received and complete the re-attempt to determine boundaries for the data gap.
8 Note that in the authors’ experience operators have used this approach to fill data gaps until verification
measurements (i.e. direct examinations and material tests) can be completed. No operators have interpreted
this method to satisfy regulatory “traceable, verifiable, and complete” requirements (to the authors’
knowledge), so it is highly recommended these entries are tracked as “assumed worst probable case,” or
“assumed most likely case.”
24

DATABASE RETRI EVAL RESEARCH GATHERIN G Determ ine Boundaries
A n e e d t o re t ri e ve i n form a t i on
Research So urces...
R u l e o u t u n l i ke l y opt i o n s u si n g re se a rch a n d
from t h e P ip e li n e D at a ba s e is
kn ow n in fo rm at i on .
i d e n t i fi ed . P o ssi b l e S o u r c es t o C o n s u l t
 A SM E
I n d u s tr y S ta n d a r d s  API
 P RCI
B as e d on re s e a rc h , i d e n t i fy th e “m os t l i ke l y

E xpo rt re l ev a n t kn o wn i n for ma t i on
N AC E
ca s e” v al u e a n d t h e “w ors t pos s i bl e ca s e ”
from P ip e li n e D at a ba s e .
va l u e .
P o ssi b l e S o u r c es t o C o n s u l t
S el e c t t h e s e va l u e s b y:
Co mp li a n c e & R eg u l a t or
 P H M SA
Co mm u n ica t i on s
- A d vi s o r y B u l l e ti n
 U si n g SM E i n p u t t o pi ck from t h e
Do e s t h e g at h e re d
e s t ab li s h e d ra n g e
- F A Qs
i n form a t io n
 U si n g a ve ra g e a n d m a x/m in v a lu e s
m e e t t h e i d e n t ifi e d  CFR
from s i m il a r pi pe l i n e s /s e gm e n t s
n e e d s?
 Cr e at e a B a ye s i an N e tw ork M od e l to
P o ssi b l e S o u r c es t o C o n s u l t c a l cu l at e t h e s e va l u e s
Co m p a n y I n fo rm a t i o n
 P u r ch a s i n g C a ta l o gs
NO
YES
 P r o ce s s a n d P ro c e d u re D o c u m e n ts
 S t u d i es
De t e r mi n e ra n g e for v al u e s
P o ssi b l e S o u r c es t o C o n s u l t
Approval
S u b j e c t M a tt er E xp e r t
 E n gi n ee r i n g
 A cc u r a te J u s ti fi c a t i o n
 P u r ch a s i n g
 C o r re c t I n fo rm a t i o n
 GI S
 L eve l o f Co n fi d e n c e
 Co n su l t a n t
 P i p e l i n er /Te c h n i ci a n
NO
YES
Pro ce e d to Qu a l it y /R el i a bi l it y
De t e r mi n a t i on
D a ta b a se o wn er w it h in p ut fr o m S M E D a ta b a se Ow ne r wi th r es ea r ch er su p po r t a nd i np u t fro m S M E
Rel eva nt compan y stakeholders:


Ma na gement, Database Owners
SME
Q
 Decisi on Ma kers (Engi ne er, Proj ect Man ag er, etc.)
25
5 ESTABLISHING DATA QUALITY AND RELIABILITY

Establishing data quality and reliability will generally follow a company decision or requirement to
determine the quality of a source record(s). The decision to determine the quality and reliability of source
record(s) is often based on a substantial change within the company or industry such as:
 Merger or acquisition
 Large expansion of assets
 Discovery of New or Different Industry/Recommended Practices.
This decision may also be the result of an identified deficiency, for example:
 Regulatory audit finding
 Self-audit finding
 Discovery of Conflicting or Errant Records.
In the worst case, this decision may be the result of a failure. The process to establish data quality and
reliability is described in Sections 5.1 through 5.3 and shown schematically by the flowchart in Section
5.44.4.
5.1 Document Type Review
The Database Owner along with the Records Librarian should compile a list of all types of source records
that contain information or data that populates the pipeline information database (e.g., construction project
records that show pipe grades, wall thickness, test pressures, etc.). The Database Owner and/or Records
Librarian should also consult SMEs (e.g., engineers who prepared construction project files for audit or
permanent storage) to provide insight about the types of source records the company has used to document
pipeline system information. Examples of source records include construction reports and notes, final
drawings, design pressure form, mill certificates, materials testing reports, maintenance reports, and
materials invoices. It is also recommended to search through construction project files from different
geographic locations, legacy companies (if acquisitions took place), and time periods as a way to capture a
variety of source record types. Once a complete list of source record types is thought to be compiled, the
Database Owner and Records Librarian should meet with an SME to confirm the list.
There should be a pre-existing process to determine source records ’ quality and reliability. The Database
Owner should examine company documentation to determine if the company already has an acceptable
method to determine the quality and reliability of a source records.
The following questions may help determine if a quality and reliability determination process is acceptable.
(1) Does the existing process comply with the most recent practices and communications from
regulatory entities?
(2) Does the existing process incorporate the source record types identified from the search described
above?
(3) Does the SME agree with quality and reliability ranking based on his/her experience with the
company’s source records?
(4) Does the process establish traceable, verifiable, and complete documents?
(5) Is the process adequate to resolve the development mentioned at the beginning of this section,
which triggered the need for metrics for data quality and reliability?
If the existing quality and reliability determination process is found unacceptable or does not exist, then a
new process must be created (see next section). If the existing quality and reliability determination process
is found to be acceptable, then it may be used.
26
5.2 Research Gathering
The research gathering phase of the quality and reliability metric determination process is a crucial stage in
developing an acceptable process. The personnel required for this section are the Database Owner and
Researchers with likely input from an SMEs.
The purpose of the research is to gather information that allows the team to create a compliant process that
is suitable for the company. The team must achieve an understanding of which source record types
(i.e., document types) should be categorized with high quality and reliability. The quality and reliability of
source records should relate to the attributes of the records including the intent of the documents ’ creation.
To achieve an understanding of which source record types should be categorized with high quality and
reliability, these attributes must be well understood for all record types.
In many cases the types and formats of source records have significant differences for different periods in
time or for different geographic areas. Depending on the company size and structure, it is possible that a
certain source record type may require different level(s) of quality and reliability in different districts of the
company. For example, a company geographic region or districts may have historically emphasized a
‘Project Completion List’ as the most important source record for recording pipe and fitting materials data
for company records. But another district may have typically emphasized a “Completion Drawing” as the
most important source record for permanently recording pipe and fitting materials data.
The quality and reliability of a source record type will be influenced by the person who was responsible for
creating it and how long after the installation the document was created. A “Project Completion List”
created by a project inspector immediately upon completion of a construction project is generally
considered more reliable than a materials list that was transcribed years after a project and by someone with
no direct knowledge of the construction.
Other causes of internal record quality variances include:
 Records organization/storage practices
 Manufacturer/Supplier documentation practices
 Completion markings (i.e., stamps, dates, and/or signatures).
These types of variance should be researched and taken into account when creating acceptable metrics to
determine quality and reliability of source records.
The research should include a review of recommendations on records quality from regulatory bodies and
other industry organizations like those that followed the San Bruno pipeline failure. PHMSA released two
advisory bulletins and a letter to the American Gas Association (AGA) [15][16][31] describing
clarifications of regulation to ensure a line is being operated at a verified MAOP/MOP.
The practice to verify MAOP/MOP is to confirm an operator possesses the information necessary to satisfy
applicable regulations (e.g., CFR 192.619 for gas pipe in the U.S. and CFR 195.406 for liquid pipe in the
U.S.) with data that can be verified by source records that meet regulatory requirements. Source records
that verify MAOP/MOP may include hydrotest records and/or construction materials records that provide
the parameters used to calculate the design pressure (CFR 192.105 for gas pipe in the U.S. and CFR 195.106
for liquid pipe in the U.S.). PHMSA stated that if an operator is using records to determine the MAOP/MOP
of a line segment, then the referenced records must be “traceable, verifiable, and complete.” PHMSA
provided the following descriptions of the criteria for these traits:
 Traceable meaning the record can be “clearly linked to original information about a pipeline
segment of facility.” Examples given included: Pipe mill records, purchase requisition, as-built
documentation.
27
 Verifiable meaning the record contains information that is “confirmed by other complementary,
but separate, documentation.” Examples given included: Contract specifications for a pressure test
complemented by pressure charts or field logs.
 Complete meaning the record is “finalized as evidenced by a signature, date, or other appropriate
marking.” Examples given included: pressure testing record should identify a specific segment of
pipe, who conducted the test, the duration of the test, the test medium, temperatures, accurate
pressure readings, and elevation information as applicable.
PHMSA further clarified “verifiable” in a follow- up letter to AGA. PHMSA confirmed that “verifiable”
source records can be a single quality, traceable and complete record or a record confirmed by other
complementary, but separate, documents. The letter does not define criteria about PHMSA’s interpretation
of “quality, ” other than “a single quality document that is traceable and complete, as evidenced by
appropriate markings, would be acceptable.”
Criteria that operators have used to determine a quality record include, but are not limited to, who created
it, when it was created, the intent of the document (i.e., document type), and the amount of information
filled in (with respect to what was expected/required to be filled in), presence of a title block, presence of
a document date, completion stamp, or a signature. The extent to which each of the above criteria determines
a quality record generally varies by operator.
Before starting the research process the SMEs and Database Owner should delegate areas and provide the
research team with a plan of action to minimize overlapping topics. The research should include, but are
not limited to, reviews of the following:
(1) Industry standards should be reviewed to determine if there are standard practices or document
quality trait requirements. It is recommended to look in some of the typical industry organizations
such as ASME, API, and PRCI, but there may be other organization related to other industries that
have some general guidance on data management practices and record quality verification.
(2) Regulatory requirements and communications should be reviewed to ensure the quality and
reliability determination method to be created will not create a compliance issue. In addition to the
sources mentioned above, PHMSA may have released additional Advisory Bulletins, frequently
asked questions, or other communications that contain interpretations of record quality and
verification. It is recommended to review the code of federal regulations and applicable state
regulations as they may also provide compliance requirements for quality of source records.
(3) Company information should be reviewed to understand previous company decisions and trust in
the different records source types. There may be other quality processes and practices utilized in
other areas that could be applied to this process to ensure that accurate information is being used.
As demonstrated in previous examples, different segments of the company may have different
experiences with record type quality.
(4) Subject Matter Experts familiar with company quality processes as well as those with a
substantial amount of experience reviewing and working with the different source record types
should be consulted to provide their insights specific to the company and the other research
gathered. The type of people could include pipeline and risk engineers, purchasing, GIS
consultants, pipeliner/technicians, and/or documentation managers.
Once the quality and reliability criteria requirements and recommendations are identified through the
research described above, the Database Owner and SMEs will work together to develop specific ranking
criteria for source records in the next section.
28
5.3 Determine Quality and Reliability Metrics
The personnel required to determine the Quality and Reliability Metrics are the Database Owner, SME(s)
(i.e., those familiar with the company’s regions and/or history, and familiar with construction procedure
development), and Decision Makers including company management.
The team should review the quality and reliability criteria requirements and recommendations from the
research step (Section 5.2) to create a methodology to determine data quality and reliability. The team will
decide on appropriate criteria to rank, rate, and grade the source records, which will determine the quality
and reliability of information extracted from them and placed into the pipeline information database. The
examples below present different types of quality and reliability ranking systems.
One potential quality ranking method (“Method 1”) is a relative ranking system by source record type. This
method requires the team to rank all source record types from most reliable (“1”) to least reliable (“15”).
This type of listing can be very easy to interpret, but it places nearly all emphasis on source record type (as
opposed to other potential attributes). This is appropriate when the source record types are very similar
within a pipeline system and have very little variation (i.e., if construction project files within a geographic
region were created with a high degree of consistency). If this method is used, it is important to have an
excellent understanding of which source record types have been the most trustworthy in the construction
project files.
The ranking of source record types may need to be adjusted to account for the specific data being extracted.
For example, a visual inspection document may provide reliable information about coating type and rank
very high (“1”), but it provides less reliable information about yield strength and would rank low (“15”) .
Separate rankings may be needed for each data type to resolve this discrepancy. The ranking of source
record types may also need to be adjusted to account for variability between time periods or geographic
region (i.e., “project completion list” may have a “1” ranking for the North region, but the “as -built
drawing” may carry a “1” ranking for the South region). This method may require a lot of deliberation
between the Database Owner, SMEs, and Decision Makers to create a final list(s). Some advantages and
disadvantages of “Method 1: relative ranking by source record type” are provided here:
Advantages:
 Easy to use during data mining because it requires very little interpretation.
 Results are highly repeatable.
Disadvantages:
 Has the potential to overemphasize source record type above other criteria.
 May require a lot of deliberation to adequately frame up list(s) to consider all relevant data types
and legacy influences (i.e., time period, geographic region).
An abbreviated example of a document hierarchy using Method 1 is shown in Table 5-1 Error! Reference
source not found. .
29
Table 5-1 : Method 1 Document Hierarchy Example. Relative Ranking by Source Record Type
Number of Required
Ranking Source Record Type
Documents
1 Materials Certification 1 required
2 Completion Drawing 1 required
… … …
15 Notes from Field 2 required
Method 1 is best suited when the source record types are all consistent with respect to the type and detail
of information contained in each individual record (i.e., if the Materials Certification and the Completion
Drawing consistently meet company records requirements, like “traceable, verifiable, and complete”) . If
this is true, then the benefit of creating a Method 1 process is that it is very straightforward to use once it is
set up.
It is recommended that Data Miners review each individual document being evaluated to ensure documents
are consistent with expectations and/or any standards set by the operator based on research findings (i.e. no
missing signatures, etc.). This type of document hierarchy tends to be specialized to the segment of the
company that created it. It is recommended to review source record type quality assumptions before use in
a different company segment. If the intent is to create one method (as opposed to multiple ranking sheets),
or if the company’s source records are found to lack consistency, then the relative ranking by source type
method may not be appropriate.
Another quality sorting method is to develop bins based on potential source record traits (“Method 2”) . This
method is more flexible than the discrete rankings shown above and less dependent on the source record
type. Based on the research conducted, there are specific traits that have been assigned to evaluate the
quality and reliability of any given source record. These traits should include, but may not be limited to the
following:
 Time that the record was produced
o Was it prior to project completion?
o Was it long after project completion? or
o Was it in the timeframe right after completion when information would be most reliable?
 Creator of the document

o Is the creator of the document noted by name and title?
o What was their relationship to the project?
 Was it created by a clerical administrator without any design or materials
experience?
 Was it created by a project inspector who would have direct knowledge of the
materials installed?
 Was it created by a project engineer who would have direct knowledge about the
project design process?
 Intent of the document
o Was the document intended to be a rough preliminary costing tool for the project?
o Was the document intended to be the permanent record of materials installed by the
project?
 Formality of the document (related to intent)
30
o Was the document created on a personal journal or notepad (i.e., intended for personal
reuse)?
o Was the document created on a company letterhead suitable for distribution?
 Completeness of the document
o Is the document signed, dated, or stamped in any way?
o Does the document have any fields where a signature, date, or stamp are clearly expected
or required but are missing?
Table 5-2 shows an example hierarchy utilizing some of the traits above. Some advantages and
disadvantages of “ Method 2: Source record bins according to specific document attributes ” are provided
here:
Advantages:
 Can be applied to documents that have a large amount of variability (i.e., different source record
types or variability in a single source record type).
 This is typically easier to develop (compared to the Method 1) document hierarchy.
Disadvantages:
 Requires the ranking of each document to be interpreted.
 May be less repeatable if two individuals identify attributes inconsistently.
31
Table 5-2: Method 2 Document Hierarchy Example. Source Record Bins According to Document
Attributes
Bin 1 A Additional comment/example:
Common Examples: Pressure Test Assessment

Commissioning Pressure Test Chart
Timeframe: Created during the process or within 1 year
Crew foreman
Creator: Someone on site Inspector
Project Manager
Clear location (Lat / Long)
Completeness: Must have all expected information Project Manager
Date
Signature/Stamp
Bin 1 B Additional comment/example:
Common Examples: Work Order

As-built/Completion drawing
Timeframe: Created upon project completion
Creator: Someone with direct association to the project Engineer

Project Manager
Clear Description
Completeness: Must have all expected information Date
Signature/Stamp
Bin 2 Additional comment/example:
Common Examples: Alignment Sheet data
Timeframe: Created within years after completion
Creator: Someone with direct association to the project Engineer

Project Manager
Clear location (Lat / Long)
Completeness: Needs only the critical expected information Project Manager
Date
For the sake of the example there are three bins shown. It may be necessary to create additional bins to
adequately cover all ranking criteria. The bins above are shown as Bin 1A, Bin 1B, and Bin 2. Bin 1A
represents high quality pressure test documents that meet the operator’s “traceable, verifiable, and
complete” document quality requirements for pressure test data without the need for any separate verifying
records. Bin 1B represents high quality documents that meet the operator’s “traceable, verifiable, and
complete” document quality requirements for pipe material property data without any separate verifying
records. Bin 2 represents documents that will only meet the operator’s “traceable, verifiable, and complete”
requirements for pipe material property data when accompanied by a separate complementary verifying
source record (which may also be another Bin 2 record).
The benefit to this type of model is the quality checks intrinsically built into the task of matching a bin to a
specific source record, however, due to this flexibility it can become more time consuming when attempting
to verify a source record’s quality and reliability if there is no perfect match to a specific bin. Operators
have used both of the methods above to denote how many verifying source documents are required to meet
company needs.
Typically, in either Method 1 or Method 2, operators denote a “high quality” document category, where
one document is required to meet company reliability requirements (e.g., with “traceable, verifiable, and
32
complete” source records). Operators will also denote requirements for a records category that must be
verified by other records to meet minimum company reliability requirements. These two document
categories are denoted “Bin 1 (A or B)” and “Bin 2” in the Method 2 example above.
After the process is created, it is highly recommended the team that developed the process should proceed
to perform a test pilot to sort through source records and apply the methodology. During this task, the team
should pay close attention to the ease, efficiency, and intuitiveness of the metric determination process
developed. If the team identifies issues that hinder or prevent source records from being evaluated as
intended, then process adjustments should be made. Once all necessary adjustments are made, the quality
and reliability determination process is finalized and approved.
 If Approved: The quality and reliability determination process that has been developed will be
put in use to verify the quality and reliability of source records for data in the pipeline
information database.
 If Rejected: The quality and reliability determination process remains unacceptable. The
Decision Makers, Database Owner, and SMEs will need to conduct more research and/or
adjust the process for improvement.
33

DOCUM ENT TYPE REVI EW RESEARCH GATHERING DETERMINE QUALITY CRI TERI A
A n e e d i s i d e n ti fi e d to ve ri fy th e
Research Sources... B as e d on g a th e re d re s e a rch a n d , e s t ab li s h a
q u al i t y of a s ou rce r e cord u se d to m e t h od for cl a ss i fi ca ti on o r g ra d i n g fo r e ach
po pu l a te t h e p i pe l in e d ata ba s e. P o ssi b l e S o u r c es t o Co n su l t i d e n ti fi ed d a ta s ou r ce.
 A SM E
 R an k e ac h re cord s ou rce type a n d pl a ce
I n d u s tr y Sta n d a r d s  API i n a n org an i ze d l i st
Cr e ate a l i s t o f a l l ty pe s o f s ou rce
 P RC I  De v e lo p bi n s wi th ce rt a i n
re co rd s th a t co n t a in pi pe l in e  N ACE ch a ra cte ri s ti cs
d a t a ba s e i n form a ti on
 De v e lo p a s cor in g s ys te m ba s e d on
va ri ou s a ttri bu te s of a s ou r ce re c ord
P o ssi b l e So u r c es t o C o n su l t

Co mp li a n ce & R eg u l a tor
P H M SA
Co mm u n ica ti on s
- A d vi so r y Bu l l e ti n
- F A Qs
Do e s th e com pa n y
- O th e r Off i ci a l Co m m u n i c a t i o n s S ort th e i d e n t i fie d d ata s ou rce s a ccord i n g

h a ve a n e xi s tin g re co rd s r an ki n g
s ys t e m to d e te r mi n e s ou rc e  CF R to m e t h od d e ve l ope d t o cr e ate a d ocu me n t
q u al i t y?  CS A Z6 6 2
h i e ra rch y or re l i ab il i ty cri te ri a
Co m p a n y I n fo rm a t i o n
P o ssi b l e So u r c es t o Co n su l t
 P r o ce ss a n d P ro c e d u re D o c u m e n ts
 St u d i es
 Co n st ru c t i o n P ro j e c t Re c o rd s
Approval
De v e lo p a n e w ra n k in g s ys te m to
Su b j e c t M a tt er E xp e r t
( D i g i ta l a n d H a rd Co p y )
YES
d e t e rm i n e re cor d so u rce q u a l i ty a n d
 A cc u r a te J u s ti fi c a t i o n

a cce pta n ce cri te ri a
Co r re c t I n fo rm a t i o n
P o ssi b l e S o u r c es t o Co n su l t
 Co m p l i a n t
 E n gi n ee r i n g
NO  P u r ch a si n g

Do e s th e
GI S
NO
e xi s t in g ra n k i n g s ys te m YES
 Co n su l t a n t
pro vi d e a cce pta bl e gu id an ce
 P i p e l i n er /Te c h n i ci a n
for t h e i d e n ti fie d
s ou r ce re co rd s ?
Ye s Pro ce e d to Pi pe Da t a G a th e ri n g
D a ta b a se Owne r a nd R e co rd s Lib ra ria n wit h in p ut fro m
SM E
D a ta b a se Ow ne r wi th res ea rch er su p po rt a nd i np u t fro m SM E
Releva nt company stakeholders:


Ma na gement, Database Owner
SME
Q
 Decision Ma kers (Engineer, Project Manag er, etc.)
34
6 ON-GOING DATA COLLECTION

Data collection, as it pertains to information about a pipeline system, has always occurred continually in
the pipeline industry. Operators and their contractors are constantly recording observations and
documenting measurements as part of field, survey, and testing activities. These new observations and
measurements in effect create new source records with respect to the pipeline system database. An on-going
data collection process is described in this section to maximize the utility of this data towards filling gaps,
increasing confidence, and maintaining the database into the future. The process is shown by a flowchart
in Section 6.5.
This process will require support from Data Miners to collect the incoming data under the oversight and
coordination of the Database Owner and SMEs.
6.1 Sources of On-Going Data Collection
An operator must identify all practical sources of ongoing data collection (that can be integrated back to
the pipeline information database). Since the number of potential sources of data can be large, this can be
a challenge and will likely require work from Data Miners with input from SMEs (about the best data
sources to fill specific database needs). It is recommended that the sources of ongoing data collection should
include (but are likely not limited to):
 Direct examination reports that record observations made or measurements taken about attributes
in the pipeline information database (e.g., observations on coating type, or wall thickness
measurements).
 Maintenance reports that record new materials installed onto the pipe (e.g., a valve replacement
report).
 ILI reports that document information along the pipe. The exact information depends on the ILI
technology (e.g., wall thickness changes).
 Cut-out mechanical test results (e.g., Charpy toughness).
 As-built reports for new build and replacement projects.
 GIS centerline survey reports (could be part of ILI, or CIS surveys).
The sources of incoming information can be used by the Data Miners to fill data gaps or increase an
operator’s confidence in areas of the pipeline information database where the legacy source records
provided low reliability.
In some cases, the information may have documented new materials and infrastructure that were installed,
which will call for effective management of change for the new source records to replace the old source
records. In other cases, measurements and observations are made about the existing materials and
infrastructure, which can be used to confirm and increase confidence levels in the existing materials
database. These scenarios are discussed in the following paragraphs.
6.2 Controlled Process for Management of Change
If an operator installs new materials or replaces materials, the operator must ensure that the new source
records adhere to the established system for records management. The stakeholders for the project that
generated new source records must be made aware of the established system for records management. There
must be a solid understanding of how the records quality is rated.
35
Ideally, new projects should always produce records of the highest reliability. This will hold true if there is
a solid understanding of the expectations for the new source records. For newly replaced or installed pipe,
it is recommended that each operator prepare as-built reports that include the following specific parameters:
 Weld map(s) that reference a heat number for each joint of pipe installed.
 The weld map(s) must have sub-meter GPS coordinates for each end of a replacement segment,
every angle point, or every half-mile (whichever is more frequent).
 An accompanying MTR(s) that includes each heat number that appears within the as-built.
 A dated signature from an onsite inspector specifically to confirm each joint was installed as drawn.
The signature date should follow (or be coincident with) the date the joint was welded in place.
These parameters should help meet the spirit of normalizing (standardizing) as-built documentation of
materials for installed pipe and set expectations between the construction team and the data mining team.
Similar parameters are sufficient for elbow fittings that meet pipe grade steel requirements and for
maintenance forms that document materials of a type-B full encirclement sleeve repair.
Note that a change in materials in an infrastructure requires a controlled process to manage the change in
records if it is a large change in infrastructure (e.g., a re-route) or a small change (e.g., a single joint
replacement). In either case, Data Miners will identify the locations in the existing pipeline information
database that needs replacement. These segment(s) of pipe and fitting(s) will need to be retired into a
controlled location according to the database schema (e.g., the protocol required by PODS, APDM,
UPDM).
New “direct links” and new reliability rankings will be determined for the new records by the Data Miners
according to the processes from Sections 4 and Sections 5. Ideally, the “direct links” to the old source
records would persist with the retired segment(s) of pipe and fitting(s) to the retired location within the
database (but this may not be possible with current database schema). The “direct links” to the old source
record may not be allowed to remain with any new attribute data or system materials, because this will
create a version control issue and the potential for misinterpretation by the company Decision Makers is
too great.
As discussed earlier in the recommendations, normalization of records is an important step for enhanced
data and recordkeeping for integrity management. On-going data collection represents the stage at which
the industry can potentially create new projects with new records that repeat past mistakes by not meeting
minimum quality and reliability criteria. Normalizing the records that should be produced for each reporting
method can eliminate these mistakes but will require understanding and collaboration from those who create
each record type and the Data Miners who review and assimilate them into the database.
6.3 Continuous Cross-Check and Quality Control
Operators can use measurements and observations from on-going field activities to gain confidence and fill
gaps in the pipeline information database. This is accomplished by cross-checking new observations and
measurements against the existing data within the database. This will require Data Miners to systematically
review the new information as it comes in and take action based on its consistency to the database. These
may be the same Data Miners who were conducting the document research in earlier sections.
The operator must track and document verifying measurements within a pipe population. A statistical
reliability model and approach is presented in Appendix C to determine and track statistical confidence by
verifying a data set. This model can be used as a metric to compare and prioritize confidence levels in
different pipe populations based on the results of verifying measurements or observations that have been
completed.
36
This metric could be used as part of an operator’s plan, along with overall risk and the reliability of the
source records, to prioritize verification activities (e.g., future field measurements). Pipe populations of
lower confidence may be prioritized higher for verification efforts depending on the overall pipeline risk
and the quality and reliability of the original source records from the pipeline information database at the
specific location.
When comparing verification measurements to the pipeline information database, occasionally an operator
will identify an inconsistency between the new information and the existing data within the database. The
next section will describe managing and resolving these inconsistencies.
6. 4 I sol ati n g an d R eso l vi n g I n con si sten ci es
It’s inevitable when cross-checking that operators will obtain new information (e.g., through verification
measurements, or surveys) that is inconsistent with a record in the pipeline information database. Making
this type of discovery is the intent of incorporating cross-checks.
When an inconsistency is identified, it can present a challenge for operators, because it may not be possible
to find a resolution without large-scale verification activities (e.g., running In-line inspection or performing
many direct examinations). The following process is an effective way to systematically revisit the source
records to see if large-scale verification activities are avoidable.
 Identify and isolate the data point(s) or source documents that are the cause of the records failure.
 Establish a boundary around the potential extents of the records failure.
 Systematically research and/or investigate within the established boundary and verify the data
discrepancy until it is resolved.
These steps will allow an operator to identify the set of documents that may have caused the inconsistency
issue. A detailed review of the documents related to the inconsistency will allow the operator to attempt to
obtain a full understanding about the nature of the inconsistency.
The project numbers and geographic locations of the inconsistent data point are commonly used to identify
the source records that are the cause of a records failure. During the review, the operator will search for
trends that identify the initial set of source records and determine how the set should be pared down. There
may be evidence that the records failure is specific to a certain type of pipe, a certain location, or project.
The operator can establish a boundary (and reduce the boundary) around the extents of the records failure
by reviewing the records for evidence that can reduce the set of records that require consideration. This is
an important step because it creates efficiencies in the final step, which may require costly verification.
The final step is reviewing the applicable set of source records and attempting to find any new information
possible to find what went wrong. The new information that led to the inconsistency may suggest more
field work needs to be done, or it may lead to more specific document research that wasn’t performed
previously (e.g., the realization that a project file had been hidden away in an engineer’s office). It is
important to note that depending on each specific scenario, inconsistency, and set of source records, it may
or may not be possible to obtain a full understanding from the source records alone.
If document review can’t resolve the inconsistency, or if it becomes impractical to continue, the data that
were the cause of the records failure should be tracked with the statistical reliability metric (See
Appendix C). At the operator’s discretion, the source records may need to be removed from the pipeline
system information database (for appropriate locations and data points), which will create a gap. Either case
will result in a downgrade in the system’s records reliability, which will likely require verification by
materials testing to return the system to its previous reliability levels.
37
This resolution can potentially be costly (depending on the final extent of the records failure). The industry
is developing rapidly in materials testing technology. There are solutions currently evolving within in-line
inspection, in-the-ditch methods that use proprietary tools and technologies, and in-the-ditch methods that
can be applied more universally. An operator should consider a technology review to evaluate the most
appropriate verification testing solution.
An example of an inconsistency that might be identified during cross-checking would be a Data Miner who,
while reviewing an ultrasonic wall-thickness ILI report, observed a reported wall thickness of 0.308 inch
for one joint in an area where the pipeline information database shows a nominal wall thickness of
0.365 inch. The Data Miner determined the ILI reported value of 0.308 inch was too low to be consistent
with a nominal wall thickness of 0.365 inch.
The Data Miner used the following steps to help resolve this inconsistency.
 Identify and isolate the data point(s) or source documents that are the cause of the records failure.
 Establish a boundary around the potential extents of the records failure.
 Systematically research and/or investigate within the established boundary and verify the data
discrepancy until it is resolved.
The Data Miner identified and isolated the source records that were the cause of the records failure by using
the “direct link” to retrieve them from the GIS system (i.e. , the source records linked to wall thickness at
the location of the discrepancy). The source record was the as-built drawing from the original construction
of the pipeline and it did not show the thinner-walled joint of pipe. The Data Miner established a boundary
around the potential extents of the records failure by reviewing the remainder of the ILI report to ensure the
wall thickness discrepancy did not occur in other locations.
The Data Miner observed other locations with similar wall thickness, but they were all replacements
performed following the previous in-line inspection. The Data Miner suspected this location was likely a
replacement as well. The Data Miner continued to research but could not find the primary maintenance
form documenting the repair. After reviewing the project files from the company network drive, he
identified the project inspector and contacted him.
The project inspector remembered the exact excavation site and confirmed the replacement took place. The
Data Miner received an email summary from the inspector and collected the project planning files from the
network drive (i.e., material acquisition forms, one call, project log). The Data Miner provided his findings
to the Database Owner, who wasn’t satisfied with the level of reliability that had been achieved, so an
excavation was scheduled for verification purposes.
In-the-ditch measurements were taken for wall-thickness, hardness (which was used to estimate lower
bound yield strength), and pipe chemistry. The direct examination report included these measurements
along with observations about seam-type and coating. The Database Owner enlisted the help of a company
SME in materials to compare the direct exam findings with the mill test results from the prior replacement
project. The SME concluded the pipe chemistry and yield strength showed the pipe were likely the same
type.
The Database Owner was satisfied with the level of reliability achieved with this new information. He
instructed the Data Miner to initiate the management of change process for this location to replace the
legacy source records with the new source records obtained and the inconsistency was resolved.
38

Continuous Cross-Check and Resolving Inconsistencies with the
On-Going Data Collection Management of Change
NO
Pipeline Information Database

Cr os s-ch e ck n e w re co rd s
A d e ci si o n is ma d e to i n te g ra te a ga i n s t th e pi pe l in e
i n form a tio n fr om on -g oi n g d ata i n form a tio n d a ta ba se .
col l e cti on i n to th e p ip e li n e
i n form a tio n d a ta ba se
I nc lu de t he n ew so urc e rec o rd s
in t o ma n a gem en t of c ha n ge
Is it
p ro ce ss.
I s th e
I s th e n e w re a s on a bl e to
d a ta fro m th e n e w
i n form a tio n mo re d i s re g ard th e n e w
re co rd s c on s i ste n t No No
re l i ab le th a n th e o ld i n form a tio n in fa vo r
w ith e x is ti n g
d a ta ? of th e ol d
re co rd s ?
R evie w so urce rec ord s a n d a ssi gn d a ta
reli a b il it y ra ti ng a cc ord in g t o th e
p ro ce ss th a t wa s es ta b li sh ed fo r
I d e n tify s ou rce s of on -g oi n g d a ta qu a l it y a n d re lia bi li ty.

Ye s
YES
d a ta col le ct io n
De te r mi n e th e e xte n t of th e
i n con si s te n cy ( g e og ra ph i c an d No
d o cu m e n t)
I nt egra t e n ew d a t a a n d a sso ci a t ed
so urc e rec o rd s i nt o t he est a b li she d

Ye s
p ip eli ne i nfo rma t io n d a ta ba se . Co n d u ct r e se a rch , s u rv e yi n g , a n d
Ye s
te s tin g u n ti l u n re li a bl e d a ta p oi n ts
Ye s
h a ve be e n r e so lv e d .
Do e s n e w d a ta re q u i re
Do e s th e
m an ag e m e n t of ch an ge ( i . e . d o e s
n e w in fo rma ti on
i t d ocu m e n t n e w ma te ri a l be i n g D isrega rd n ew
re q u i re a
in fo rma t io n a n d a ll ow
i n s ta ll e d ) ? No
M o C?
o ri gin a l da ta to rem a in
E n te r n e w d a ta i n to th e p ip e li n e
No
i n form a tio n d a ta ba se
Q Tra ck ve r ifi ca tio n wi th S tati s ti cal con fi d e n ce m e tri c ma y
s tati s ti ca l con fi d e n ce con tri bu te to pr io ri tiz e fu tu re

C ha n ge in te gra ti on c om pl et e
m e tri c ve r ifi ca tio n or d a ta c ol le cti o n efforts .
D eci sio n M a kers , D a ta b a se Owne r, a n d
rese a rc he r( s)
D a ta b a se Owne r wi th res ea rch er( s) Database Owner, researcher(s), SME(s) Q
39
7 CLOSURE
In summary, the processes provided in Sections 3 through 6 of this guideline show an enhanced data and
recordkeeping plan to manage records for pipeline integrity management.
Lessons learned from other industries reviewed in Section 1.2 have pointed to the following improvements
that can potentially be made in the near term and longer term, including:
1. Normalize the data and record types that should be maintained.
2. Formally dedicate resources specifically to recordkeeping roles and responsibilities.
3. Provide more accessibility of the data and records to company deputies.
 The plan addresses the first lesson learned with a system to rank legacy pipeline system materials
records with the document hierarchy in Section 5. The plan provides some normalization
guidance for construction records for new pipe in Section 6.2. Normalization for the entire set of
integrity management data is recommended for future work.
 The plan addresses the second lesson learned by indicating roles and responsibilities for each
process, including Section 6 On-going Data Collection, which will require an on-going records
maintenance team.
 The plan addresses the third lesson learned by defining a pipeline information database structure
with direct links between the GIS interface and the verifying source records. This interface allows
decision makers access to source records when making important fitness-for-service and risk
decisions.
The guideline presented a detailed “plan for collecting, reviewing, and analyzing” and maintaining data
that could be referenced in Sections 4.3 and 4.4 of ASME B31.8S. The plan implements a philosophy that
was created from the records review initiative following San Bruno to create a structured database and
make source records (as the basis for the information in the database) accessible to company Decision
Makers.
40
APPENDIX A: GUIDANCE FOR USING A BAYESIAN NETWORK TO BOUND

UNKNOWN VALUES INTO LIKELY VALUES
The following example of a Bayesian network was used, with the authors’ permission, from “Quantitative
Assessment of Corrosion Probability – A Bayesian Network Approach,” NACE International, Corrosion
Vol. 70, No. 11, 2014. [35]
Bayesian networks have proven useful to pipeline operators for filling unacceptable gaps in pipeline
datasets (or databases). One of the Bayesian methods to make this determination could be to build a
Bayesian Network in order to calculate the probability of different states. This method is most effective if
there are multiple known variables that have an implied influence on the unknown (gap) variable. This type
of analysis has been used for applications in the pipeline industry during risk determination, root cause
analysis, or maintenance prediction.
A Bayesian Network is a type of graphic statistical model that represents probabilistic relationships between
variables. This analytic process is used to model the probabilities of cause and effect relationships to make
predictions based on known distributions and values. If parameter “X” is true, then the network shows the
probability of the values for related parameters “W, ” “Y, ” and “Z. ” Below is an example of the graphical
model for a simple Bayesian network.
Figure A-1 : Example Graphical Bayesian Network Model for Uninhibited Internal Corrosion Growth
Rate
CO2 Fe2+ O2 H2S

Temperature pH
Uninhibited
Corrosion
Growth Rate
In this example, the goal is to determine the uninhibited internal corrosion growth rate (i.e., the unknown
“gap” variable) for an area taking into account environmental info rmation.
1.1 Creating a Bayesian Network
A functional Bayesian Network can be created more quickly and easily with a software package specifically
designed to streamline the process. There are various commercial and free options with different levels of
complexity and functions. The commercial options will be the most comprehensive but might not be
practical if there are no plans for repeat use. Software options can be found from various websites, and it is
recommended to look at the available resources online and select the best option for the desired application.
If there are no suitable free options, it is suggested to consult with an industry expert with commercial
Bayesian Network software experience that preferably also has pipeline experience.
41
1.1.1 Developing the Graphical Model
The following instructions are representative of typical software options, but if the instructions below are
not helpful, then it is recommended to seek out instructional material for the specific software selected.
Two components are needed to create a basic Bayesian Network: the “Chance” bubble depicted as an oval
and the “Arc” connectors depicted as an arrow. The “Chance” ovals are used to represent variables and the
“Arc” arrows are used to assign the relationships between the different “Chance” ovals.
Begin by placing “Chance” ovals and assigning variable names to each one. This step requires a detailed
understanding of the result and the factors that are directly influencing it. The model is more likely to be an
accurate representation of reality if most (or all) of the influencing variables are present and are considered.
It is important the arrows that link the variables to one another in the model reflect direct cause and effect
relationships and not simply a correlation. For example, when looking at pipeline failure data, there may be
a correlation between the year the pipe was manufactured and the number of failures from that decade. The
age of the pipe does not directly cause the pipe to fail, but it does directly influence some of the pipe
properties, such as the coating and seam type. To correctly link the manufacturing year to the failure in a
Bayesian Network, there are intermediate variables needed, such as coating and seam type.
1.1.2 Developing the Conditional Probability Tables
Once all variables and relationships are created and assigned, the user must input the probabilities for each
case the variable could represent. The case could represent a decision (e.g., yes or no), a numerical value
or range, or a current state (e.g., on or off). The relationships in the Bayesian network become more complex
with each additional case that the variables could potentially represent.
The probability for each case can be assigned manually or, for networks containing variables with multiple
cases, using a program to calculate the probabilities for each case. After all the cases are assigned
probabilities, all values in one column (i.e., possible cases for a single variable) must add up to one. In most
software there is “normalize” option when entering data to scale all the values proportionally so that they
automatically add to one. Although this will create a probability set that is functional, this does not fix
incorrect data entries. A Bayesian Network will only be as accurate as the probabilities assigned to different
variable cases and the relationships assigned, so it is pivotal that these be verified once completed.
There are three main ways to determine appropriate probabilities and relationships: SME input,
research/observation data, tested and recognized models. For variables with many potential cases, SME
input may not be the best information source. However, SME input is highly useful when assigning
relationships and for identifying scenarios where the condition probability will be either zero or one or
identifying scenarios when using a uniform distribution is appropriate.
Research or observed data can be applied to determine conditional probabilities. The quality of this
information is only as good as the source and sample it was taken from. If an operator wanted to determine
the probability distribution for pipe grade, the operator’s total pipe mileage could be broken down into
percentages based on the grade and then applied to the Bayesian Network. Depending on the variable, there
may also be industry sources or studies that will provide the necessary data to determine percentages.
Using a model to determine conditional probabilities is typically the most preferred method, when
applicable. It is also the ideal method when a variable has multiple relationships, which will result in a large
table of conditional probabilities. This method determines probabilities from the results of running the
model based on different inputs each with a given range, similar to a Monte Carlo calculation. By
calculating possible combinations, a probability distribution can be developed by creating bins and
determining the percentage of results that fell inside the bin. Some pipeline related models could include
42
something as complex as multiphase flow or corrosion rate or something well established, such as the design
pressure equation.
Take the corrosion growth example depicted above. Once the graphical model is created, the probabilities
will need to be assigned. The following tables show potential probability assignments for this Bayesian
Network assuming none of the input parameters are known.
Table A-1 : Probability Assignments for Example Bayesian Network
For the top row of variables, each one was split into appropriate ranges as determined by SMEs and assigned
uniform probability distributions. It is possible to go in later and update these probabilities based on known
information, typically called “Evidence” by various software packages. There could also be additional
levels of detail added to make these distributions different, such as temperature is affected by barometric
pressure and the time of year and the pH of the soil is impacted by the location. Adding these extra levels
of detail will change these variables from uniform distributions to a custom distribution that will need to be
determined by a model or quality data and potentially improve the accuracy of the model. However, it is
possible that these extra variables will have minimal impact on the desired result or can be compensated
for by known information, so it is best to consult with an SME to determine the level of detail required.
To determine the probability distribution for the corrosion growth rate variable, a corrosion rate model that
considers these inputs will need to be used due to the number of potential combinations. There are six
variables, five with four possible cases and one with three possible cases, that impact corrosion growth.
This will result in 3072 different probabilities per corrosion growth bin because each combination of
variables will need to be considered. Below is a table containing the first and last five columns of the
conditional probability table for the corrosion growth variable. In this case, the corrosion rate probabilities
are based on a corrosion model that uses the environmental parameters as input.
43
Table A-2: Example Results from the Conditional Probability Table
1.2 Using a Bayesian Network
Once the network has been created and all the inputs have been verified, it can be used within the Bayesian
Network software package, by referencing the model using an external program routine, such as Excel,
Visual Basic (VBA), or MATLAB, or by using a pre-existing software.
1.2.1 Running in the Bayesian Network Software
To run the network in the software that was used to create it, first flip the “Chance” ovals to view as a Bar
Chart. Some packages may have this as a default, but others will have it as a secondary view which can
typically be found under a “View” or “Node Format” related menu. Some packages may use other methods
or not have this functionality, so it is recommended to review instructional materials for the specific
software used if the suggestions above are not helpful.
Once the different variables are shown as a bar chart, make sure to update the view as some packages do
not update automatically. This can usually be found as an icon along the top tool bar. After the page is
refreshed, the bar charts should display the probabilities for each case of each variable.
After updating the network in the software, the table below shows the results with no known data.
Table A-3: Bayesian Example Results for Scenario with No Known Data
This step is also a good chance to verify that the conditional probabilities yield accurate results. Sometimes
the initial distributions may not be intuitive to conceptualize (which is the advantage of using a Bayesian
Network), so it is important to evaluate the network using some known data points. To enter known
information, double click the case in the bar chart view and the probability will change to 100%. It is also
possible to set “uncertain” evidence, where the updated probabilities eliminate one or more of the states for
a variable. Doing this does not erase the data previously entered and can be undone at any time. It is also
important to update the view if the network does not auto update. In the example tables below, the red cells
are known values, the blue cells are uncertain values, and the green cells are unknowns and are recalculated
by the Bayesian Network.
44
Table A-4: Bayesian Example Results for Scenario with a Mixture of Known, Unknown, and
Uncertain Data
Table A-5: Bayesian Example Results for Scenario A with a Mixture of Known and Unknown Data
As more precise information is added to the network, unknowns will become more refined. Also, as
demonstrated in the second table, although some variables remain as unknowns, a corrosion growth rate
might still be able to be determined with a high degree of certainty.
In the example, the corrosion growth rate is most likely the desired information, but it is important to still
think of a Bayesian Network as a loop rather than as a routine with a start and end. For example, if the
corrosion growth rate was already known, the Bayesian Network can calculate the probabilities for the other
unknown variables. To do this in the creation software, double click the bin case on the bar chart
representation of the corrosion growth variable and update the view if necessary. Below is an example using
a known corrosion growth rate, the cell color meanings are the same as the above examples.
Table A-6: Bayesian Example Results for Scenario B with a Mixture of Known and Unknown Data
This functionality of entering in some known/uncertain information to determine the unknown values is the
intended goal of using a Bayesian Network to make decisions for the “most likely case” and “worst probable
case” values in the bounding likely values process.
1.2.2 Running in Other Software
The authors of this guideline don’t necessarily expect software will be required for all Bayesian Network
applications resulting from the practices in this guideline. However, once a Bayesian Network is built it can
also be used by custom programs/macros written in programs such as MATLAB or Excel. An API or library
of commands compatible with Bayesian Networks can be found for free or from a paid provider online
(e.g., SMILE from BayesFusion, LLC [32]) and will facilitate use of the Bayesian Network. Also available
are premade software to handle and visualize the Bayesian Network results.
45
1.3 Hypothetical Case Study
During an internal audit process an operator identified missing wall thickness and SMYS values along one
of its pipeline segments. At some locations, just one variable was unknown, and at other locations both
were unknown. Without accurate information, the pipeline is at risk for inaccurate fitness-for-service
evaluations and incomplete design data for MAOP calculations. With these considerations, the operator
assembled a small team to compile and research relevant records to compile materials data of the other
representative and potentially comparable pipe types within the system.
The team was able to compile records about the materials information for other lines constructed during the
same time period. There were also some design pressure calculations found that confirm the operating
pressure. The team realized that due to multiple unknown values, it needed to use a process that could
handle a large amount of combinations that can handle multiple factors or variables. The team decided that
the best course of action forward is to build a Bayesian Network to determine the missing wall thickness
and SMYS values.
First, the Bayesian Network team identified that the missing variables were most commonly associated
through calculating the design pressure. The team referred to the internal design pressure formula listed in
CFR 195.1069 to start compiling a list of nodes.
2푆푡
푃=( )∗퐹∗퐺
퐸
Where,
P = Internal design pressure
S = Yield strength
t = Nominal wall thickness
D = Nominal outside diameter
E = Seam joint factor
F = Design factor
The team began to then list all possible values for SMYS, wall thickness, diameter, seam factor, and design
factor. During this process, the team also discovered that certain properties were also impacted by the year
manufactured and the seam type so they were added as extra nodes to network. The team identified 11
possible SMYS values, 51 possible wall thickness values, 19 possible outside diameters, three possible
seam factors, three possible design factors, ten possible decades, and eight possible seam types. Then to list
out possible design pressures, the team elected to use pressure bins for every 10 psig starting at 0 psig and
going to 2000 psig.
After establishing the nodes and entering in all the possible values, the team began assigning the
relationships. The figure below shows the resulting network diagram.
9 This equation is similar to other regulatory design formulae, such as CFR 192.105 and Z662 4.3.3.1.1.
46
Figure A-2: Example of a Network Diagram that shows the relationships between nodes
These relationships were determined based on the design pressure model as well as some of the research
documents used. Combinations of SMYS, seam, outside diameter, and wall thicknesses were identified
from manufacturing standard API 5L. After studying the API 5L combinations in conjunction with the
legacy procurement practices, pipe grade A and a number of seam weld types were eliminated from
possibility. The only relationship remaining from this source was between the outside diameter and wall
thickness. Also, CFR 195.106 specified a simple relationship between the seam type and seam factor. Using
these extra relationships will allow the team to refine the network.
The next step was for the team to start creating the conditional probability tables. Based on this network,
tables were needed for Seam Type, Seam Factor, Outside Diameter, Wall Thickness, and MOP nodes. Most
of these tables were simple to create as there was only one relationship; and most of these relationships
were in a “yes”/”no” format (e.g. , “yes,” 4” OD pipe was made with a 0.250” WT, but “no” it was not made
with 1” WT). This resulted in a small number of table cells to fill in, a uniform distribution for all “yes”
pairs, and zero for all “no” pairs.
Next, the team addressed the conditional probability table for the “MOP” node. This table was the most
complex and most time-consuming part of the network development because this cell had five different
relationships (one for each path leading into the “MOP” node). Each combination of possibilities was
considered to determine the distribution of possible design pressures. In total the team needed to consider
95,931 possible combinations of the five variables in the design pressure model to complete the Bayesian
network. Completing this Bayesian network was possible with the right software, but by using the
observations made during research of API 5L the team was able to refine the network instead. Two of the
nodes (seam type and design factor) were replaced with constant values, which effectively removed them
from the model and reduced the size and complexity significantly. The total number of property
combinations was reduced to 10,659 and the amount of time and resources to set-up the software and write
the problem-solving script was also reduced. The figure below shows the simplified network.
47
Figure A-3: Example of a Simplified Network Diagram that shows the relationships between
nodes
When preparing the problem-solving script (e.g., in VBA) it was important to ensure that the order of the
pipe properties used to calculate the “MOP” in the script appeared in the same as the order as they appeared
in the Bayesian Network software. The tables could then be copied directly from Excel into the Bayesian
Network software.
Once all the probability tables were created, the model was ready. The line was operated at a pressure of
985 psi and had a nominal outside diameter of 1 2”. These values had been verified by multiple sources
collected by the research team and were consistent with the current operating parameters. The team set
these parameters as known within the Bayesian Network. After the evidence was set, the network revealed
that grade B SMYS and 0.1 72” WT were the worst-case values for SMYS and wall thickness, respectively
(but not as combination) , and X42 SMYS and 0.250” WT was the most likely combination. The team used
these values going forward in the missing value verification process and continued with the process detailed
in Section 4.
It is important to note that in this case, the operator had a high level of confidence in its design practices
for the pipe in question. There can be inherent flaws using the current operating pressure to make inferences
about material properties and this step may not be recommended in all circumstances. Each operator must
decide when this method (as with other Bayesian input variables) is appropriate for purpose.
48
APPENDIX B: GUIDANCE FOR USING A DOCUMENT HIERARCHY TO
SYSTEMATICALLY DETERMINE DOCUMENT RELIABILITY
The discussion in Section 7 focused on establishing a document hierarchy, or document ranking scheme,
which can be used to determine the quality and reliability of the data extracted from within each source
record. Once created, the document hierarchy not only establishes data quality and reliability, it also
provides an effective tool to assist in efficient data mining. This can become essential if a large data mining
effort is required.
A team consisting of Data Miners with oversight from engineers is typically assembled to complete large
data mining efforts, like those that occur when construction records are reviewed to verify maximum
allowable operating pressure (MAOP). This team structure can provide cost advantages because the data
mining team members are typically more cost-effective (less expensive) than engineers.
The Data Miners in this team structure require oversight from engineers and process controls to assure the
quality of their results. The document hierarchy is a valuable process control for this type of effort. It can
create efficiencies by helping the Data Miners to understand which document types are more likely to
contain the targeted information. It can also create repeatability by providing guidance about interpreting
document precedence and how to resolve conflicting information.
This appendix discusses how to use a document hierarchy during a data mining project, and it provides a
brief hypothetical case study to convey a potential real life context.
Data Mining with a Document Hierarchy
When functioning properly, a document hierarchy will allow Data Miners to target higher value documents
and help them to discern information between conflicting source records to arrive at the most reliable set
of pipeline data. The authors have typically used document hierarchies when data mining construction
records, pressure test, and maintenance records, but have also found they apply to in-line inspection results
and expect that they apply to other data types as well (e.g., potentially control room pressure records) once
properly configured and found fit for purpose by company SMEs who are familiar with the targeted data
and the sources under review.
The figure below shows a fully developed “Method 1” document hierarchy based on document type. This
document hierarchy is fit for purpose when the source records under review are very consistent. This has
been observed to occur when common procedures were followed closely and common forms were used
consistently over long periods of time. This practice produces document types that are very consistent
between projects and can be relied upon to have the same types of data.
A Data Miner would use the document hierarchy in conjunction with a digital document database and
project list to target and retrieve high value documents on a project by project basis to reconstruct each
pipeline. Here are a few steps the Data Miner might follow:
1. Once the project numbers are organized chronologically for a particular pipeline, the Data
Miner will use the document index metadata to target “mapping documents ” (like drawings)
from the oldest project to define the project extents (with GIS stationing, if available).
2. Once the Data Miner understands where the project took place he will review the available
highest ranking mapping documents to identify where various pipe and fitting materials were
installed along the line.
3. If the highest-ranking mapping document provides adequate attribute information for the Data
Miner’s research requirements, the Data Miner moves to the next pipe, fitting, etc., but if the
mapping document has gaps in attribute data or requires a second verifying document, the Data
49
Miner continues to review documents from the same project number. The Data Miner will use
the metadata to target the highest value documents first. If there is a “Design Materials
Engineering Document Report” (from Table B-1, below) available in this project file, that will
be targeted first to fill the materials data gaps. This search will continue until all gaps are filled
or project records are exhausted.
4. The Data Miner will then move to the next oldest project on the project list and repeat this
process. If projects are found to replace one another, the materials information will be treated
in a similar manner to the MOC process from Section 6.2. When all projects have been
reviewed, the pipeline is current and the on-going data collection begins.
Table B-1 : Fully Developed Method 1 Document Hierarchy Based on Document Type
Number of Required
Ranking Source Record Type
Documents
1 Design Materials Engineering Report 1 required
2 Completion Drawing 1 required
3 Material Certifications 1 required
4 Pressure Test Job Report 1 required
5 Welding Inspection Report 1 required
6 Materials Invoice 2 required
7 Material Purchase Order 2 required
8 Material Transfer Order 2 required
9 Re-Claimed Material 2 required
10 Atlas Sheets 2 required
11 Strip Maps 2 required
12 Material Requisition 2 required
13 Construction Bid Specifications 2 required
14 Construction Inspector Notes 2 required

15 Field Notes 2 required
16 Notes from Field 2 required
Resolving Conflicting Information with a Document Hierarchy

Data Miners will also use the document hierarchy to resolve conflicting information. The steps below
provide an example of how a Data Miner might use a document hierarchy to resolve conflicting information.
1. A Data Miner has defined project extents for the current project under review and researched pipe
and components drawn on a strip map that was used to document the materials installed. The strip
map shows grade X-42 pipe with 0.365-inch wall thickness, but the document hierarchy requires a
second verifying document for a strip map so the Data Miner continues researching.
2. The Data Miner opens a “construction bid specification” that shows the pipe with 0.365 -inch wall
thickness with grade B (which is in conflict with the “X-42, ” mentioned above). In order to resolve
this conflict the Data Miner must continue to research.
50
3. The Data Miner reviews a Material Invoice that shows the 0.365-inch wall thickness pipe for the
project was X-42, in agreement with the strip map. The Data Miner consults the document
hierarchy, which specifies the purchase order and strip map both rank higher than the construction
bid specification.
4. The Data Miner uses the document ranking order to override the construction bid specification and
verify the grade X42 with the strip map and material invoice.
The examples above were both simplified to show the principle of the document hierarchy. Method 1 is
applicable for document sets that have a high degree of consistency, but they will still need to be reviewed
for reliability flags (e.g., missing signatures, date/quantity inconsistencies, etc.) and the review may not be
as simple as described. In practice, Data Miners will need to be trained to scrutinize the reliability of
documents in addition to using the document hierarchy as a guidance tool, which is why engineer oversight
and review is critical.
The examples described above could also apply to a Method 2 document hierarchy, but the Data Miner
would have to perform more scrutiny of each document and interpretation of the documents attributes’
(since a Method 2 document hierarchy is typically used for more inconsistent document sets).
When implementing a document hierarchy into a data mining process for the first time, a low production
pilot is highly recommended. Unexpected document types and procedural gaps are common at the
beginning of data mining projects. Taking the time to conduct a low production pilot (with 1-3 Data Miners
for 3-6 weeks) will allow a team to address these challenges prior to wasting resources during a full-
production effort.
51
APPENDIX C: STATISTICAL METHOD FOR TRACKING PIPELINE DATA
CONFIDENCE AND RELIABILITY
This appendix outlines an approach to calculate the number of digs needed to verify a material property
such as pipe grade, or wall thickness, is consistent with verification measurements to a given statistical
confidencel level on a given pipeline. This calculation can also be used as a reliability metric when
comparing verifiability for prioritization between different pipe populations.
The approach is based on establishing a lower bound likelihood that any member of a population is
consistent with results obtained by (prior) sampling of the population. A sample is consistent if it matches
prior results, and it is inconsistent if it does not. The approach can be used, for example, to estimate a lower-
bound likelihood any random pipe joint in a pipeline is the same grade or wall thickness as that found by
sampling one or more locations.
In this appendix, a population is a finite set of something (e.g., pipe joints of unknown type or wall thickness
on a pipeline). Sampling is the process of selecting a random member of the population to determine if it
meets predefined criteria (e.g., if the pipe joint is X52). Consistency is the proportion of the population that
matches the predefined criteria.
The approach is based on a hybrid algorithm that combines an exact solution for an infinite population with
a newly developed finite-population normal approximation based on the Agresti-Coull method in API 1163.
[36] Any normal approximation requires a reasonable sample size to be accurate. The exact method
(detailed in this appendix) is conservative for finite populations. By using a hybrid methodology, the finite
population approach will be used once sufficient number of joints have been assessed. For smaller numbers
of joints the exact conservative solution may be used.
Binomial proportions take on one of two states (e.g., sampled yield strengths are or are not consistent with
a given grade). The confidence interval surrounds the proportion of pipe that matches a target property in a
finite population of pipe joints. The method can be used in situations where the sampling results are not
consistent.
As an example of the levels of confidence that are achievable by completing a very large number of
verification measurements, Table C-1 summarizes the number of digs needed to achieve a given confidence
level and consistency for an infinite population using the exact solution, assuming no inconsistencies. Here,
consistency is defined as the minimum (lower bound) proportion of pipe on the pipeline that matches a
target grade. If an operator wants to have 95% confidence that 95% of the pipe joints on a pipeline are the
same grade, he must perform 59 digs, all of which must be consistent with the target grade. If sufficient
data is available, the finite population approach covered later may require fewer samples.
52
Table C-1 : Number of Digs Required as a Function of Confidence Level and Consistency with the
Exact Conservative Solution
Consistency
80% 90% 95% 99%
80% 8 16 32 160
Confidence
Level 90% 11 22 45 228
95% 14 29 59 297
99% 21 44 90 456
In the event inconsistencies cannot be explained by typical variability in various pipe characteristics (e.g.,
one pipe joint is X52 and another is X65), separate populations must be developed. This scenario typically
coincides with multiple inconsistent measurements.
Statistical Models for Calculating Confidence Levels
This section discusses a procedure to calculate a lower bound “consistency” (i.e., proportion of pipe meeting
a given criteria). The lower consistency bound is for the full population (relevant pipe segment with
assumed similar diameter, wall thickness, grade, seam, install date, manufacturer, and manufacture date)
based on the available sample information.
Exact Binomial Confidence Interval Formulas
This section is primarily taken from Harper (2005) [37], including the Excel VBA function that was used
for the exact results shown in this appendix. Hollander and Wolfe [38] explain exact Clopper-Pearson [39]
binomial confidence limits. The lower and upper confidence bounds are given below.
pL (n, B)  B
B  (n  B 1) f /2,2(n B1),2 B
pU (n, B) 1  pL (n, n  B)
where B is the number of successes in the n Bernoulli trials and 푓훾,푛1 ,푛2 is the upper 훾 푡ℎ percentile of the F
distribution with n1 and n2 degrees of freedom.
Table B-2, which is based on a 95% confidence level, provides an example to aid in the interpretation of
lower consistency bounds. Table B-2 shows that the full population has at least (similar to a worst case) a
given consistency level. Consistency may be viewed as either a proportion in the range [0, 1] or a percentage
in the range [0%, 100%]. The computed consistency is a function of:
 Desired confidence level (80%, 90% or 95% are suggested)
 # observations (labeled “# joints inspected”): these are the row labels in the first column
 # inconsistencies: these are the column labels in row 10.
53
Table B-2: Lower-Bound Consistency Table at a 95% Confidence Level for an Infinite Population
# Inconsistencies
# Joints
inspected 0 1 2 3 4 5 6 7 8 9
1 0. 050 0. 000
2 0. 224 0. 025 0. 000
3 0. 368 0. 1 35 0. 01 7 0. 000
4 0. 473 0. 249 0. 098 0. 01 3 0. 000
5 0. 549 0. 343 0. 1 89 0. 076 0. 01 0 0. 000
6 0. 607 0. 41 8 0. 271 0. 1 53 0. 063 0. 009 0. 000
7 0. 652 0. 479 0. 341 0. 225 0. 1 29 0. 053 0. 007 0. 000
8 0. 688 0. 529 0. 400 0. 289 0. 1 93 0. 1 1 1 0. 046 0. 006 0. 000
9 0. 71 7 0. 571 0. 450 0. 345 0. 251 0. 1 69 0. 098 0. 041 0. 006 0. 000
10 0. 741 0. 606 0. 493 0. 393 0. 304 0. 222 0. 1 50 0. 087 0. 037 0. 005
11 0. 762 0. 636 0. 530 0. 436 0. 350 0. 271 0. 200 0. 1 35 0. 079 0. 033
12 0. 779 0. 661 0. 562 0. 473 0. 391 0. 31 5 0. 245 0. 1 81 0. 1 23 0. 072
13 0. 794 0. 684 0. 590 0. 505 0. 427 0. 355 0. 287 0. 224 0. 1 66 0. 1 1 3
14 0. 807 0. 703 0. 61 5 0. 534 0. 460 0. 390 0. 325 0. 264 0. 206 0. 1 53
15 0. 81 9 0. 721 0. 637 0. 560 0. 489 0. 423 0. 360 0. 300 0. 244 0. 1 91
16 0. 829 0. 736 0. 656 0. 583 0. 51 6 0. 452 0. 391 0. 333 0. 279 0. 227
17 0. 838 0. 750 0. 674 0. 604 0. 539 0. 478 0. 420 0. 364 0. 31 1 0. 260
18 0. 847 0. 762 0. 690 0. 623 0. 561 0. 502 0. 446 0. 392 0. 341 0. 291
19 0. 854 0. 774 0. 704 0. 641 0. 581 0. 524 0. 470 0. 41 8 0. 368 0. 320
20 0. 861 0. 784 0. 71 7 0. 656 0. 599 0. 544 0. 492 0. 442 0. 394 0. 347
21 0. 867 0. 793 0. 729 0. 671 0. 61 6 0. 563 0. 51 3 0. 464 0. 41 7 0. 372
22 0. 873 0. 802 0. 741 0. 684 0. 631 0. 580 0. 532 0. 485 0. 439 0. 395
23 0. 878 0. 81 0 0. 751 0. 696 0. 645 0. 596 0. 549 0. 504 0. 460 0. 41 7
24 0. 883 0. 81 7 0. 760 0. 708 0. 658 0. 61 1 0. 565 0. 521 0. 479 0. 437
25 0. 887 0. 824 0. 769 0. 71 8 0. 670 0. 625 0. 580 0. 538 0. 496 0. 456
26 0. 891 0. 830 0. 777 0. 728 0. 682 0. 637 0. 595 0. 553 0. 51 3 0. 474
27 0. 895 0. 836 0. 785 0. 737 0. 692 0. 649 0. 608 0. 568 0. 529 0. 491
28 0. 899 0. 841 0. 792 0. 746 0. 702 0. 661 0. 620 0. 581 0. 543 0. 506
29 0. 902 0. 847 0. 798 0. 754 0. 71 2 0. 671 0. 632 0. 594 0. 557 0. 521
30 0. 905 0. 851 0. 805 0. 761 0. 720 0. 681 0. 643 0. 606 0. 570 0. 535
For example, an operator conducts 14 excavations, all of which are consistent with a given grade. As seen
in Table B-2, a lower bound consistency of 0.807 (80.7%) is obtained at the 95% confidence level when
zero inconsistencies are found in 14 observations. A consistency of 0.807 implies that the worst case (or
95% lower bound) consistency for the full population is at least 80.7%. That is, at least 80.7% of the
population is the same grade. Of course, for this example it could be as high as 100% since no
inconsistencies have been found.
If the operator wants a higher statistical consistency, more excavations are required. For example, if the
operator wants 90% of the population to be the same grade, 29 consistent excavations are needed. What
happens if the operator finds an inconsistent measurement? The number of required digs to achieve the
same statistical consistency increases. If there is one inconsistency, Table B-2 shows that 22 excavations
are needed to conclude that 80% of the population is the same grade. As the number of inconsistencies
increases, the consistency drops for a given number of excavations.
Table B-3, Table B-4, and Table B-5 show similar lower-bound consistency values at the 99%, 90%, and
80% confidence level.
54
# Inconsistencies
# Joints
inspected 0 1 2 3 4 5 6 7 8 9
1 0. 01 0 0. 000
2 0. 1 00 0. 005 0. 000
3 0. 21 5 0. 059 0. 003 0. 000
4 0. 31 6 0. 1 41 0. 042 0. 003 0. 000
5 0. 398 0. 222 0. 1 06 0. 033 0. 002 0. 000
6 0. 464 0. 294 0. 1 73 0. 085 0. 027 0. 002 0. 000
7 0. 51 8 0. 357 0. 236 0. 1 42 0. 071 0. 023 0. 001 0. 000
8 0. 562 0. 41 0 0. 293 0. 1 98 0. 1 21 0. 061 0. 020 0. 001 0. 000
9 0. 599 0. 456 0. 344 0. 250 0. 1 71 0. 1 05 0. 053 0. 01 7 0. 001 0. 000
10 0. 631 0. 496 0. 388 0. 297 0. 21 8 0. 1 50 0. 093 0. 048 0. 01 6 0. 001
11 0. 658 0. 530 0. 428 0. 340 0. 262 0. 1 94 0. 1 34 0. 084 0. 043 0. 01 4
12 0. 681 0. 560 0. 463 0. 378 0. 302 0. 235 0. 1 75 0. 1 21 0. 076 0. 039
13 0. 702 0. 587 0. 494 0. 41 2 0. 339 0. 273 0. 21 3 0. 1 59 0. 1 1 1 0. 069
14 0. 720 0. 61 1 0. 522 0. 443 0. 373 0. 308 0. 249 0. 1 95 0. 1 46 0. 1 02
15 0. 736 0. 632 0. 547 0. 471 0. 403 0. 340 0. 282 0. 229 0. 1 79 0. 1 35
16 0. 750 0. 651 0. 570 0. 497 0. 431 0. 370 0. 31 3 0. 261 0. 21 2 0. 1 66
17 0. 763 0. 668 0. 590 0. 520 0. 457 0. 397 0. 342 0. 291 0. 242 0. 1 97
18 0. 774 0. 684 0. 609 0. 542 0. 480 0. 423 0. 369 0. 31 9 0. 271 0. 226
19 0. 785 0. 698 0. 626 0. 561 0. 502 0. 446 0. 394 0. 345 0. 298 0. 254
20 0. 794 0. 71 1 0. 642 0. 579 0. 522 0. 468 0. 41 7 0. 369 0. 323 0. 280
21 0. 803 0. 723 0. 656 0. 596 0. 540 0. 488 0. 439 0. 392 0. 347 0. 305
22 0. 81 1 0. 734 0. 670 0. 61 1 0. 557 0. 507 0. 459 0. 41 3 0. 370 0. 328
23 0. 81 9 0. 744 0. 682 0. 626 0. 573 0. 524 0. 478 0. 433 0. 391 0. 350
24 0. 825 0. 754 0. 693 0. 639 0. 588 0. 540 0. 495 0. 452 0. 41 0 0. 370
25 0. 832 0. 763 0. 704 0. 651 0. 602 0. 556 0. 51 2 0. 469 0. 429 0. 390
26 0. 838 0. 771 0. 71 4 0. 663 0. 61 5 0. 570 0. 527 0. 486 0. 446 0. 408
27 0. 843 0. 778 0. 723 0. 674 0. 627 0. 583 0. 542 0. 502 0. 463 0. 426
28 0. 848 0. 785 0. 732 0. 684 0. 639 0. 596 0. 555 0. 51 6 0. 479 0. 442
29 0. 853 0. 792 0. 740 0. 693 0. 650 0. 608 0. 568 0. 530 0. 493 0. 458
30 0. 858 0. 798 0. 748 0. 702 0. 660 0. 61 9 0. 580 0. 543 0. 507 0. 473
55
# Inconsistencies
# Joints
inspected 0 1 2 3 4 5 6 7 8 9
1 0.1 00 0.000
2 0.31 6 0.051 0.000
3 0.464 0.1 96 0.035 0.000
4 0.562 0.320 0.1 43 0.026 0.000
5 0.631 0.41 6 0.247 0.1 1 2 0.021 0.000
6 0.681 0.490 0.333 0.201 0.093 0.01 7 0.000
7 0.720 0.547 0.404 0.279 0.1 70 0.079 0.01 5 0.000
8 0.750 0.594 0.462 0.345 0.240 0.1 47 0.069 0.01 3 0.000
9 0.774 0.632 0.51 0 0.401 0.301 0.21 0 0.1 29 0.061 0.01 2 0.000
10 0.794 0.663 0.550 0.448 0.354 0.267 0.1 88 0.1 1 6 0.055 0.01 0
11 0.81 1 0.690 0.585 0.489 0.401 0.31 8 0.241 0.1 69 0.1 05 0.049
12 0.825 0.71 3 0.61 4 0.525 0.441 0.362 0.288 0.21 9 0.1 54 0.096
13 0.838 0.732 0.640 0.556 0.477 0.402 0.331 0.264 0.201 0.1 42
14 0.848 0.749 0.663 0.583 0.508 0.437 0.369 0.305 0.243 0.1 85
15 0.858 0.764 0.683 0.607 0.536 0.468 0.404 0.342 0.282 0.226
16 0.866 0.778 0.700 0.629 0.561 0.496 0.435 0.375 0.31 8 0.263
17 0.873 0.790 0.71 6 0.648 0.584 0.522 0.463 0.406 0.350 0.297
18 0.880 0.801 0.731 0.666 0.604 0.545 0.488 0.433 0.380 0.329
19 0.886 0.81 0 0.743 0.681 0.622 0.566 0.51 1 0.459 0.408 0.358
20 0.891 0.81 9 0.755 0.696 0.639 0.585 0.533 0.482 0.433 0.385
21 0.896 0.827 0.766 0.709 0.655 0.603 0.552 0.503 0.456 0.41 0
22 0.901 0.834 0.776 0.721 0.669 0.61 9 0.570 0.523 0.477 0.432
23 0.905 0.841 0.785 0.732 0.682 0.634 0.587 0.541 0.497 0.454
24 0.909 0.847 0.793 0.742 0.694 0.648 0.602 0.558 0.51 6 0.474
25 0.91 2 0.853 0.801 0.752 0.705 0.660 0.61 7 0.574 0.533 0.492
26 0.91 5 0.858 0.808 0.761 0.71 6 0.672 0.630 0.589 0.549 0.509
27 0.91 8 0.863 0.81 5 0.769 0.725 0.683 0.642 0.603 0.564 0.525
28 0.921 0.868 0.821 0.777 0.735 0.694 0.654 0.61 5 0.578 0.541
29 0.924 0.872 0.827 0.784 0.743 0.703 0.665 0.628 0.591 0.555
30 0.926 0.876 0.832 0.791 0.751 0.71 3 0.675 0.639 0.603 0.568
56
Table B-5: Lower-Bound Consistency Table at an 80% Confidence Level for an Infinite Population
# Inconsistencies
# Joints
inspected 0 1 2 3 4 5 6 7 8 9
1 0.200 0.000
2 0.447 0.1 06 0.000
3 0.585 0.287 0.072 0.000
4 0.669 0.41 8 0.21 2 0.054 0.000
5 0.725 0.51 0 0.327 0.1 69 0.044 0.000
6 0.765 0.578 0.41 5 0.269 0.1 40 0.037 0.000
7 0.795 0.629 0.483 0.350 0.228 0.1 20 0.031 0.000
8 0.81 8 0.670 0.538 0.41 6 0.303 0.1 99 0.1 04 0.028 0.000
9 0.836 0.702 0.582 0.471 0.366 0.268 0.1 76 0.093 0.024 0.000
10 0.851 0.729 0.61 9 0.51 6 0.41 9 0.327 0.239 0.1 58 0.083 0.022
11 0.864 0.751 0.650 0.555 0.464 0.378 0.295 0.21 7 0.1 43 0.076
12 0.874 0.770 0.676 0.588 0.503 0.422 0.344 0.269 0.1 98 0.1 31
13 0.884 0.787 0.699 0.61 6 0.537 0.461 0.387 0.31 6 0.248 0.1 82
14 0.891 0.801 0.71 9 0.641 0.566 0.494 0.425 0.357 0.292 0.229
15 0.898 0.81 3 0.736 0.663 0.592 0.524 0.458 0.394 0.332 0.272
16 0.904 0.824 0.751 0.682 0.61 5 0.551 0.488 0.427 0.368 0.31 0
17 0.91 0 0.834 0.765 0.699 0.636 0.575 0.51 5 0.457 0.400 0.345
18 0.91 4 0.843 0.777 0.71 5 0.655 0.596 0.540 0.484 0.430 0.377
19 0.91 9 0.850 0.788 0.729 0.671 0.61 6 0.562 0.509 0.457 0.406
20 0.923 0.858 0.798 0.741 0.687 0.634 0.582 0.531 0.481 0.432
21 0.926 0.864 0.807 0.753 0.701 0.650 0.600 0.551 0.503 0.456
22 0.929 0.870 0.81 5 0.763 0.71 3 0.664 0.61 7 0.570 0.524 0.479
23 0.932 0.875 0.823 0.773 0.725 0.678 0.632 0.587 0.543 0.499
24 0.935 0.880 0.830 0.782 0.736 0.691 0.646 0.603 0.560 0.51 8
25 0.938 0.885 0.837 0.790 0.746 0.702 0.660 0.61 8 0.576 0.536
26 0.940 0.889 0.843 0.798 0.755 0.71 3 0.672 0.631 0.592 0.552
27 0.942 0.893 0.848 0.805 0.764 0.723 0.683 0.644 0.606 0.568
28 0.944 0.897 0.853 0.81 2 0.772 0.732 0.694 0.656 0.61 9 0.582
29 0.946 0.900 0.858 0.81 8 0.779 0.741 0.704 0.667 0.631 0.595
30 0.948 0.903 0.863 0.824 0.786 0.749 0.71 3 0.678 0.643 0.608
Finite Population Adaption of the Agresti-Coull Normal Approximation to the Binomial

The finite population adaption of the Agresti-Coull normal approximation to the binomial may be found in
Harper (2017). [40] The infinite population Agresti-Coull methodology is explained in the 2nd edition of
API-1163 [36] as well as the original publication.[41] Samples sizes of at least 10 (preferably more) are
needed for a reasonable normal approximation to the binomial distribution. One example follows in Table
B-6.
57
Table B-6: Lower-Bound Consistency Table at a 95% Confidence Level for a Finite Population of
50 using the Agresti-Coull finite population formulation
Simple Verification Prioritization Case

The recommended application for the statistical model above is for prioritizing field verification activities
as in the following example. An integrity engineer has enough budget for one more verification excavation
on pipeline A. The integrity engineer’s verification procedure requires verification excavation locations to
be prioritized based on the following parameters.
 Pipeline risk model
 Source record reliability
 Statistical reliability from previous verification measurements (95% CL).
Pipeline A has two primary pipe populations primarily consisting of 0.250-inch wall pipe (1000 joints),
with 0.375-inch wall pipe at some locations (50 joints). For simplicity, it is assumed that pipeline A has a
relatively uniform risk profile and the reliability of legacy source records for the 0.250-inch wall thickness
pipe is similar to the reliability of the records for the 0.375-inch pipe.
58
Previously, there were 15 joints of 0.250-inch pipe excavated and the field measurements were consistent
(within the operator’ s acceptable tolerances for wall thickness measurements). The integrity engineer used
the statistical model to calculate a reliability level of 0.821 (95% CL) shown in Table B-7. The integrity
engineer also used the statistical model to calculate the reliability level for 14 prior excavated joints from
the 0.375-inch pipe, which was 0.826 (95% CL) shown in Table B-6.
The integrity engineer scheduled the next verification excavation on the 0.250-inch wall thickness pipe,
even though it had more verification digs performed previously, because it had the lower statistical
reliability (and the pipeline risk and source record reliability parameters did not factor into the decision
because they were equivalent in both pipe populations).
Table B-7: Lower-Bound Consistency Table at a 95% Confidence Level for a Finite Population of
1 000 using the Agresti-Coull finite population formulation
59
APPENDIX D: ITEMIZED LISTS OF POTENTIAL REVISION LOCATIONS FOR ASME STANDARDS B31 .4, B31 .8,
AND B31 .8S
Table D-1 : ASME B31 .4 Potential Locations for Future Revision
Sub-
Chapter Section Section Page ASME B31.4-2016 Text
VI- 436.5.1 "On systems where pipe is telescoped by grade, wall thickness, or both, particular care shall be taken
Inspection Visual a-3 58 to ensure proper placement of pipe. Permanent records shall be kept showing the location as installed
and Testing of each grade, wall thickness, type, specification, and manufacturer of the pipe."
"A record shall be maintained in the files of the operating company relative to design, construction,
VI- 437.7 and testing of each main line within the scope of this code. These records shall include material
Inspection Records 61 specifications; route maps and assignments sheets for "as-built" condition; location of each pipe size,
and Testing grade, wall thickness, type of seam (if any), and manufacturer; coatings; and test data. These records
shall be kept for the life of the facility. See para 436.5.1(a)(3)."
450.2
VII- Operation "Each operating company having a transportation piping system within the scope of this Code shall… .
Operation and and g 62 (g) maintain necessary maps and records to properly administer the plans and procedures, including
Maintenance Maintenanc records listed in section 455."
Procedures e Plans and
Procedures
VII- "For operation and maintenance purposes, the following records shall be properly maintained: (a)
Operation and 455 Records 74 necessary operational data (b) pipeline patrol records (c) corrosion records as required under section
Maintenance 468 (d) leak and break records (e) records pertaining to routine or unusual inspections, such as external
Procedures or internal line conditions (f) pipeline repair records"
456
VII- Qualifying a
Piping
Operation and System for a "Records of such investigations, work performed, and pressure tests conducted shall be preserved as
Maintenance d 75 long as the facilities involved remain in service"
Procedures Higher
Operating
Pressure
60
Sub-
VIII 462.3 "A record of the internal condition of the pipe, of leaks and repairs from corrosion, and of liquids or
Corrosion Existing (b)(4) 80 solids quantities and corrosivity shall be kept and used as a basis for changes in the cleaning pig
Control Installations schedules, chemical treatment program, or liquid treatment facility"
VIII "Records indicating cathodically protected piping, cathodic protection facilities, and other structures
Corrosion 468 Records (a) 83 affected by or affecting the cathodic protection system shall be maintained by the operating company".
Control
VIII "Records of tests, surveys, visual and instrumented inspection results, leaks, etc., necessary for
Corrosion 468 Records (b) 83 evaluating the effectiveness of corrosion control measures shall be maintained and retained for as long
Control as the piping remains in service."
XI Slurry "For slurry pipelines, results of all tests, surveys, and inspections required by this Chapter to indicate
Pipeline C468 (b) 104 the adequacy of corrosion and erosion control measures, as well as records relating to routine or
Systems Records unusual inspections, such as internal or external line conditions when cutting or hot tapping the line,
shall be maintained for the service life of the piping system."
61
Table D-2: ASME B31 .8-201 6 Potential Locations for Future Revisions
Sub-
"Records of the tests that establish the qualification of a welding procedure shall
II-Welding 823.4 Qualification Records 20 be maintained as long as that procedure is in use. The operating company or
contractor shall, during the construction involved, maintain a record of the
welders qualified, showing the dates and results of tests."
841.3.2 Pressure Test

Requirements to Prove "Operating companies shall retain, in their files, for the useful life of each
IV - Design, Strength of pipelines and pipeline and main, records showing the procedures used and the data developed
Installation, and Mains to Operate at Hoop (i) 50 in establishing the maximum allowable operating pressure of that pipeline or
Testing Stresses of 30% or More of main. Refer to section N-7 of Nonmandatory Appendix N for a list of suggested
the Specified Minimum records for retention"
Yield Strength of the Pipe
IV - Design, "The following commissioning records shall be maintained as permanent

Installation, and 841.4.5 Documentation and 51 records: (a) cleaning and drying procedures (b) cleaning and drying results (c)
Testing Records function-testing records of pipeline monitoring (d) control equipment systems
(e) completed prestart checklist."
IV - Design, 845.2.3 Qualification of a " Records of hydrostatic pressure tests and line repairs shall be preserved as
Installation, and Steel Pipeline or Main to 64 long as the facilities involved remain in service"
Testing Establish the MAOP
IV - Design, 845.2.6 Conversion of Low-

Installation, and Pressure Distribution a (2) 65 "past maintenance records including results of any previous leakage surveys"
Testing Systems to High-Pressure
Distribution Systems
"Records shall be made covering all leaks discovered and repairs made. All
V- Operating and 851.6 Pipeline Leak pipeline breaks shall be reported in detail. These records along with leakage
Maintenance Records 81 survey records, line patrol records, and other records relating to routine or
Procedures unusual inspections shall be kept in the file of the operating company, as long as
the section of line remains in service."
62
Sub-
V- Operating and 851.10 Recommissioning of "Before a facility is recommissioned, all maintenance and cathodic protection
Maintenance Transmission Facilities (a) 82 records shall be reviewed to ensure that the condition and integrity of the facility
Procedures has been maintained during the decommissioned period."
V- Operating and "A record shall be maintained for locating valves covered by paras 853.4.1 and
Maintenance 853.4.4 Valve Records 88 8853.4.2. These records may be maintained on operating maps, separate files, or
Procedures summary sheets, and the information on these records shall be readily accessible
to personnel required to respond to emergencies."
“Review the following historical data and make an evaluation of the pipeline's
condition: (a) Study all available information on the original pipeline design,
inspection, and testing. Particular attention should be paid to welding
V- Operating and 855.2 Historical Records procedures used and other joining methods, internal and external coating, pipe,
Maintenance Study 91 and other material descriptions (b) Study available operating and maintenance
Procedures data including leak records, inspections, failures, cathodic protection, and
internal corrosion control practices (c) Consider the age of the pipeline and the
length of time it may have been out of service in preparing a final evaluation to
convert the pipeline to gas service."
V- Operating and 855.5 Records of the "Maintain for life of the pipeline a record of the studies, inspections, tests,
Maintenance Conversion 91 repairs, replacements, and alterations made in connection with conversion of the
Procedures existing steel pipeline to gas service under this Code."
V- Operating and "Records for uprating, including each investigation required by this section,
Maintenance 857.1 General (f) 92 corrective action taken, and pressure test conducted, shall be retained as long as
Procedures the facilities involves remain in service."
857.4 Uprating a Ductile

V- Operating and Iron High-Pressure Main or "The maximum allowable operating pressure of a ductile iron main or system
Maintenance System to a New and (a) 93 shall not be increased to a pressure in excess of that permitted in para.
Procedures Higher Maximum 842.1.1(a). Where records are not complete enough to permit the direct
Allowable Operating application of para. 842.1.1(a), the following procedures shall be used:"
Pressure
63
Sub-
"Records indicating cathodically protected piping, cathodic protection facilities,

VI Corrosion Control 860.1 Scope (e) 95 and other structures affected by or affecting the cathodic protection system shall
be maintained by the operating company."
"Records of tests, surveys, visual and instrumented inspection results, leaks,

VI Corrosion Control 860.1 Scope (f) 95 etc., necessary for evaluating the effectiveness of corrosion control measures
shall be maintained and retained for as long as the piping remains in service."
“The operating company shall maintain in its file, for the useful life of each
VII Offshore Gas A847.5 Records 114 pipeline, records showing the type of test fluid, the test procedure, the test
Transmission pressure, and the duration of the test."
"The operating company shall maintain in its file for the useful life of each
pipeline and main, records showing the following: (a) test medium (b) test
pressure (c) test duration (d) test date (e) pressure recording chart and pressure
Nonmandatory log (f) pressure versus volume plot (if applicable) (g) pressure at high and low
Appendix N elevations (h) elevation at point test pressure measure (i) person(s) conducting
Recommended Practice N-7 Records 176 test, operator, and testing contractor, if utilized (j) environmental factors
for Hydrostatic Testing (ambient temperature, raining, snowing, windy, etc.) (l) pipe specifications
of Pipelines in place (SMYS, diameter, wall thickness, etc.) (m) clear identification of what is
included in each test section (n) description of any leaks or failures and their
disposition. The above records shall be reviewed to ensure that the requirements
of the Code have been met."
64
Table D-3: ASME B31 .8S-201 6 Potential Locations for Future Revisions
Chapter Section Sub-Section Page ASME B31.8S-2016 Text
"Information on the operation, maintenance, patrolling, design, operating

2 Integrity Management 2.3 Integrity 2.3.2 Gathering, history, and specific failures and concerns unique to each segment will be
Program Overview Management Process Reviewing, and 5 needed. Relevant data and information will also include those conditions or
Integrating Data actions that affect defect growth, reduce pipe properties, or relate to the
introduction of new defects.”
2.3.6 Update, After the initial integrity assessments have been performed, the operator has
2 Integrity Management 2.3 Integrity Integrate, and 6 improved and updated information about the condition of the pipeline system
Program Overview Management Process Review Data or segment. This information shall be retained and added to the database of
information used to support future risk assessments and integrity assessments.
"An integral part of the risk assessment process is the incorporation of

5.4 Developing a Risk additional data elements or changes to facility data. To ensure regular updates,
5 Risk Assessment Assessment Approach 12 the operator shall incorporate the risk assessment process into existing field
reporting, engineering, and facility mapping processes and incorporate
additional processes as required."
"The risk assessment process shall be thoroughly and completely documented

5.7 Characteristics of an to provide the background and technical justification for the methods and
5 Risk Assessment Effective Risk (g) Documentation 14 procedures used and their impact on decisions based on the risk estimates.
Assessment Approach Like the risk process itself, such a document should be periodically updated as
modifications or risk process changes are incorporated."
65
"Data collection issues are discussed in section 4. When analyzing the results
5 Risk Assessment 5.9 Data Collection for 15 of the risk assessments, the operator may find that additional data are required.
Risk Assessment Iteration of the risk assessment process may be required to improve the clarity
of the results, as well as confirm the reasonableness of the results."
"Data collected during the inspection and mitigation activities shall be

analyzed and integrated with previously collected data. This is in addition to
other types of integrity management-related data that is constantly being
gathered through normal operations and maintenance activities. The addition
8 Integrity Management 8.2 Updating the Plan 28 of this new data is a continuous process that, over time, will improve the
Plan accuracy of future risk assessments via its integration (see section 4). This
ongoing data integration and periodic risk assessment will result in continual
revision to the integrity assessment and mitigation aspects of the plan. In
addition, changes to the physical and operating aspects of the pipeline system
or segment shall be properly managed (see section 11)"
"This ongoing process will most likely result in a series of additional integrity
8 Integrity Management assessments or review of previous integrity assessments. A series of additional
Plan 8.2 Updating the Plan 28 mitigation activities or follow-up to previous mitigation activities may also be
required. The plan shall be updated periodically as additional information is
acquired and incorporated."
"Operational measures include operational and maintenance trends that
measure how well the system is responding to the integrity management
9.2 Performance 9.2.2 Operational program. An example of such a measure might be the changes in corrosion
9 Performance Plan Measures Measures 30 rates due to the implementation of a more effective CP program. The number
Characteristics of third-party pipeline hits after the implementation of prevention activities,
such as improving the excavation notification process within the system, is
another example."
9.4 Performance … the following overall program measurements shall be determined and
9 Performance Plan Measurement: 31 documented: …
Intrasystem
66
9.4 Performance Table 9.4-1 or Table 9.4-1 Performance Metrics or Table 9.4-2 Overall Performance
9 Performance Plan Measurement: Table 9.4-2 32 Measures
Intrasystem
"If an operator decides to increase pressure in the system from its historical
11 Management of (b)(3) 35 operating pressure to, or closer to, the allowable MAOP, that change shall be
Change Plan reflected in the integrity plan and the threats shall be re-evaluated
accordingly."
"(d) Management of change ensures that the integrity management process

remains viable and effective as changes to the system occur and/or new,
revised, or corrected data becomes available. Any change to equipment or
procedures has the potential to affect pipeline integrity. Most changes,
however small, will have a consequent effect on another aspect of the system.
For example, many equipment changes will require a corresponding technical
(d) and/or (e) 35 or procedural change. All changes shall be identified and reviewed before
implementation. Management of change procedures provides a means of
maintaining order during periods of change in the system and helps to preserve
confidence in the integrity of the pipeline." and "(e) In order to ensure the
integrity of a system, a documented record of changes should be developed
and maintained. This information will provide a better understanding of the
system and possible threats to its integrity. It should include the process and
design information both before and after the changes were put into place."
"The operator shall determine the documentation required and include it in the
12.2 Quality quality program. These documents shall be controlled and maintained at
12 Quality Control Plan Management Control (b)(1) 36 appropriate locations for the duration of the program. Examples of
documented activities include risk assessments, the integrity management plan,
integrity management reports, and data documents."
67

These data are collected in support of performing risk assessment…
(a) Year of installation
(b) Coating type
(c) Coating condition
A-2 (d) Years with adequate CP
Nonmandatory External Corrosion (e) Years with questionable CP
Appendix A Threat Threat A-2.2 Gathering, (f) Years without CP
Process Charts and (similar for other Reviewing, and 45 (g) Soil characteristics
Prescriptive Integrity threats, but with slightly Integration Data (h) Pipe inspection reports (bell hole)
Management Plans different data (i) MIC detected (yes, no, or unknown)
requirements) (j) Leak history
(k) Wall thickness
(l) Diameter
(m) Operating stress level (%SMYS)
(n) Past hydrostatic test information
Nonmandatory A-2
Appendix A Threat External Corrosion During the inspection activities, the operator may discover other data that
Process Charts and Threat ; A-2.6 Other Data 47 should be used when performing risk assessments for other threats.
Prescriptive Integrity (very similar for other
Management Plans threats)
The following performance measures shall be documented… :

Nonmandatory A-2 Number of hydrostatic test failures caused by external corrosion
Appendix A Threat External Corrosion Number of repair actions taken due to in-line inspection results, immediate and
Process Charts and Threat A-2.5 Performance 47 scheduled
Prescriptive Integrity (very similar for other Measures Number of repair actions taken due to direct assessment results, immediate and
Management Plans threats) scheduled
Number of external corrosion leaks (for low-stress pipelines it may be
beneficial to compile leaks by leak classification)
68
REFERENCES
[1] Oxford Dictionaries, https://en.oxforddictionaries.com/
[2] PHMSA Operator Qualifications Glossary https://primis.phmsa.dot.gov/oq/glossary.htm.
[3] ISO 15489-1:2016 which lists the source as ISO 23081-1:2006.
[4] PHMSA Operator Qualifications Glossary https://primis.phmsa.dot.gov/oq/glossary.htm.
[5] International standard ISO 15489-1, section 5.2.2.
[6] Pipeline Replacement Updates, By-Decade Inventory, US DOT, PHMSA, portal date 2/27/2017;
https://opsweb.phmsa.dot.gov/pipeline_replacement/by_decade_installation.asp
[7] Rosenfeld, Michael and Rick Gailing, “Pressure testing and recordkeeping: reconciling historic pipeline
practices with new requirements, ” Pipeline pigging and integrity management conference, Houston, Feb.
2013,p12.;http://kiefner.com/wp-content/uploads/2013/05/Historic-pipeline-pressure-testing-and-
recordkeeping-practices-2013-PPIM-paper.pdf
[8] Gas Transmission Integrity Management Rule (49 CFR Part 192, Subpart O), commonly referred to as
the “Gas IM Rule.”; https://primis.phmsa.dot.gov/gasimp/
[9] Australian Standard - AS 2885.0 - 2008: Pipelines – Gas and Liquid Petroleum: General Requirements.
ME-038 Committee, 2008.
[10] Australian Standard - AS 2885.1 - 2012: Pipelines – Gas and Liquid Petroleum: Design and
Construction. ME-038 Committee, 2012.
[11] Australian Standard - AS 2885.2 - 2016: Pipelines – Gas and Liquid Petroleum: Welding. ME-038
Committee, 2016.
[12] Australian Standard - AS 2885.3 - 2012: Pipelines – Gas and Liquid Petroleum: Operation and
Maintenance. ME-038 Committee, 2012.
[13] Australian Standard - AS 2885.4 - 2010: Pipelines – Gas and Liquid Petroleum: Submarine Pipeline
Systems. ME-038 Committee, 2010.
[14] Australian Standard - AS 2885.5 - 2012: Pipelines – Gas and Liquid Petroleum: Field Pressure Testing.
ME-038 Committee, 2012.
[15] ADB-11- 01 PHMSA Advisory Bulletin. “Pipeline Safety: Establishing Maximum Allowable
Operating Pressure or Maximum Operating Pressure Using Record Evidence, and Integrity Management
Risk Identification, Assessment, Prevention, and Mitigation, ” Docket No. PHMSA-2010-0381.
[16] ADB-12- 06 PHMSA Advisory Bulletin: “Verification of Records Establishing MAOP and MOP , ”
May 7, 2012.
[17] This requirement was confirmed into law by the US Congress when the Pipeline Safety, Regulatory
Certainty, and Job Creation Act of 2011 was passed on January 3, 2012.
[18] ISO 13623:2000 Petroleum and natural gas industries- Pipeline transportation systems First edition
2000-04-15.
[19] NACE International SP0113- 2013 “Pipeline Integrity Method Selection.”
[20] 21 CFR Chapter 1 Subchapter H Part 801.2 Label to bear a unique device identifier, Subpart B
http://www.ecfr.gov/cgi-bin/text-
idx?SID=b38a882ddb80f01a40cc20c0da9d8b68&mc=true&node=pt21.8.801&rgn=div5#se21.8.801_120
[21] Specification for Pipeline and Piping Valves, API Specification 6D Twenty-Fourth Edition, August
2014.
[22] ASME NQA-1-2015 Quality Assurance for Nuclear Facility Applications, American Society of
Mechanical Engineers, February 20, 2015.
[23] ASME BPVC Section III-Rules for Construction of Nuclear Facility Components-Division 3-
Containments for Transportation & Storage of Spent Nuclear Fuel & High level Radioactive Material &
Waste.
[24] ISO 15489- 1:2016 (e), “Information and documentation – Records Management – Part 1 Concepts and
Principles, ” 2016.
[25] RFP Enhanced Data and Recordkeeping Requirements for Effective Pipeline Integrity Management in
ASME B31.8 and B31.8S.
69
[26] ASME B31.8S-2016 Section 851.6 Pipeline Leak Records.
[27] U.S. National Archives and Records Administration (NARA), “Technical Guidelines for Digitizing
Archival Materials for Electronic Access: Creation of Production Master Files – Raster Images, ”
Washington, DC, June 2004: https://www.archives.gov/files/preservation/technical/guidelines.pdf .
[28] Lutz, Andrew, and T. Bubenik, “A Practical Approach to Pipeline System M aterials Verification, ”
2012, ASME, IPC2012-90494, p3.
[29] PHMSA Advisory Bulletin 2011- 01 “Pipeline Safety: Establishing Maximum Allowable Operating
Pressure or Maximum Operating Pressure Using Record Evidence, and Integrity Management Risk
Identification, Assessment, Prevention, and Mitigation, ” DOT and PHMSA, Docket No. PHMSA-2010-
0381, January 4, 2011.
[30] PHMSA Advisory Bulleting 2012- 06 “Pipeline Safety: Verification of Records , ” DOT and PHMSA,
Docket No. PHMSA-2012-0068, May 1, 2012.
[31] Letter from PHMSA to AGA, July 31, 2012. https://www.aga.org/sites/default/files/legacy-assets/our-
issues/safety/pipleinesafety/Agencynotices/2012/Documents/PHMSA%20MAOP%20Record%20Respon
se.pdf
[32] http://www.bayesfusion.com/downloads
[33] CFR 192.105
[34] Z662 4.3.3.1.1.
[35] F. Ayello, S. Jain, N. Shridar, and G.H Koch, “Quantitative Assessment of Corrosion Probability – A
Bayesian Network Approach,” NACE International, Corrosion Vol. 70, No. 11, 2014 .
[36 ] API Standard 1163, “In-line Inspection Systems Qualification, ” 2nd edition, April 2013, American
Petroleum Institute.
[37] Harper, William V., “Excel Functions to Compute Exact Binomial Confidence Intervals , ” proceedings
of the 25th European Meeting of Statisticians, July 2005, Oslo, Norway, Paper P-031.
[38] Hollander, Myles, and Douglas A. Wolfe, Nonparametric Statistical Methods, 2nd edition, John Wiley
& Sons, Inc., New York, 1999, pp. 32, note 20, Interval Endpoints.
[39] Clopper, C. J., and E.S. Pearson, 1934, “The use of confidence or fiducial limits illustrated in the case
of the binomial, ” Biometrika, 26, 404 -413.
[40] Harper, William V., “Finite Population Agresti-Coull based Binomial Confidence Intervals and Excel
VBA Function, ” Proceedings of the Joint Statistical Meetings (JSM2017 - Section on Physical and
Engineering Sciences), August 2017.
[41] Agresti, Alan and Brent A. Coull (1998), “Approximate is Better than ‘ Exact’ for Interval Estimation
of Binomial Proportions ,” The American Statistician, 52:2, 119-126.
70
PTB-1 2-201 7

Asme PTB-12-2017 PDF

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Asme PTB-12-2017 PDF

Загружено:

Авторское право:

Доступные форматы

AS M E PTB-1 2-201 7

GUIDELINES FOR ADDRESSING

No part of this document may be reproduced in any form,

The American Society of Mechanical Engineers

1 PURPOSE AND USE

1 .1 Background of Data and Recordkeeping in the Pipeline Industry

3.1 Locations and Formats of Pertinent Data

Figure 3-1 : An Example of a Pressure Test Dead Weight Chart.

3.2.1 Document Indexing with Metadata

3.4 Work Flow Process Diagram

s ys te m i n form a tio n a n d Da ta mi n i n g to e sta bl i s h “d i re ct li n ks ” to

d i g iti z a tio n i n i tia ti ve s ou r ce re co rd s u s in g:

( e. g. m icro fi ch e, -Do cu m e n t h i e ra rch y ( S ec tio n 5 )

fil e org a n i z ati on , pro j e ct

i n form a tio n , or a Loa d /re co n ci le d a ta mi n i n g re s u l ts w ith

“pe e k ” c h e ck? “d i re ct l i n ks ” to so u rce re co rd s ( a n d

ba s e d on th e pr oce s s from Se cti on 5 )

Pre p ar e o r m a rk th e “b ox/s h e l f/bi n ”

for i mp ort a n d d i g iti z a tio n

Sto re in d oc um en t Co mp an y Pi pe l i n e I n form ati on

R em ov e th e “box /s h e l f/bi n ” from

En gin eer wi th d a t a m in ers Sca nn in g ven d or En gin eer wi th D a t a m in ers

4 ESTABLISHING LIKELY BOUNDARIES FOR UNKNOWN VALUES

4.4 Work Flow Process Diagram

5 ESTABLISHING DATA QUALITY AND RELIABILITY

2 Completion Drawing 1 required

15 Notes from Field 2 required

 Creator of the document

Common Examples: Pressure Test Assessment

Bin 1 B Additional comment/example:

Common Examples: Work Order

Timeframe: Created upon project completion

Creator: Someone with direct association to the project Engineer

Bin 2 Additional comment/example:

Common Examples: Alignment Sheet data

Timeframe: Created within years after completion

Creator: Someone with direct association to the project Engineer

5.4 Work Flow Process Diagram

q u al i t y of a s ou rce r e cord u se d to m e t h od for cl a ss i fi ca ti on o r g ra d i n g fo r e ach

po pu l a te t h e p i pe l in e d ata ba s e. P o ssi b l e S o u r c es t o Co n su l t i d e n ti fi ed d a ta s ou r ce.

I n d u s tr y Sta n d a r d s  API i n a n org an i ze d l i st

- O th e r Off i ci a l Co m m u n i c a t i o n s S ort th e i d e n t i fie d d ata s ou rce s a ccord i n g

s ys t e m to d e te r mi n e s ou rc e  CF R to m e t h od d e ve l ope d t o cr e ate a d ocu me n t

D a ta b a se Owne r a nd R e co rd s Lib ra ria n wit h in p ut fro m

6 ON-GOING DATA COLLECTION

6.5 Work Flow Process Diagram

Pipeline Information Database

i n form a tio n fr om on -g oi n g d ata i n form a tio n d a ta ba se .

I d e n tify s ou rce s of on -g oi n g d a ta qu a l it y a n d re lia bi li ty.

so urc e rec o rd s i nt o t he est a b li she d

Q Tra ck ve r ifi ca tio n wi th S tati s ti cal con fi d e n ce m e tri c ma y

s tati s ti ca l con fi d e n ce con tri bu te to pr io ri tiz e fu tu re

D eci sio n M a kers , D a ta b a se Owne r, a n d

APPENDIX A: GUIDANCE FOR USING A BAYESIAN NETWORK TO BOUND

CO2 Fe2+ O2 H2S

1.2 Using a Bayesian Network

3 Material Certifications 1 required

4 Pressure Test Job Report 1 required

5 Welding Inspection Report 1 required

6 Materials Invoice 2 required

7 Material Purchase Order 2 required

8 Material Transfer Order 2 required

9 Re-Claimed Material 2 required