Practical Implementation of A Data Lake Translating Customer Expectations Into Tangible Technical Goals 1St Edition Nayanjyoti Paul All Chapter

Practical Implementation of a Data
Lake: Translating Customer

Expectations into Tangible Technical
Goals 1st Edition Nayanjyoti Paul
Visit to download the full and correct content document:
https://ebookmass.com/product/practical-implementation-of-a-data-lake-translating-cu
stomer-expectations-into-tangible-technical-goals-1st-edition-nayanjyoti-paul/
Practical
Implementation
of a Data Lake
Translating Customer Expectations
into Tangible Technical Goals
—
Nayanjyoti Paul
Practical
Implementation of a
Data Lake
Translating Customer
Expectations into Tangible
Technical Goals
Nayanjyoti Paul
Practical Implementation of a Data Lake: Translating Customer
Expectations into Tangible Technical Goals
Nayanjyoti Paul
Edison, NJ, USA
ISBN-13 (pbk): 978-1-4842-9734-6 ISBN-13 (electronic): 978-1-4842-9735-3

https://doi.org/10.1007/978-1-4842-9735-3
Copyright © 2023 by Nayanjyoti Paul

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or
part of the material is concerned, specifically the rights of translation, reprinting, reuse of
illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way,
and transmission or information storage and retrieval, electronic adaptation, computer software,
or by similar or dissimilar methodology now known or hereafter developed.
Trademarked names, logos, and images may appear in this book. Rather than use a trademark
symbol with every occurrence of a trademarked name, logo, or image we use the names, logos,
and images only in an editorial fashion and to the benefit of the trademark owner, with no
intention of infringement of the trademark.
The use in this publication of trade names, trademarks, service marks, and similar terms, even if
they are not identified as such, is not to be taken as an expression of opinion as to whether or not
they are subject to proprietary rights.
While the advice and information in this book are believed to be true and accurate at the date of
publication, neither the authors nor the editors nor the publisher can accept any legal
responsibility for any errors or omissions that may be made. The publisher makes no warranty,
express or implied, with respect to the material contained herein.
Managing Director, Apress Media LLC: Welmoed Spahr
Acquisitions Editor: Celestin Suresh John
Development Editor: James Markham
Coordinating Editor: Mark Powers
Cover designed by eStudioCalamar
Cover image by Arek Socha on Pixabay (www.pixabay.com)
Distributed to the book trade worldwide by Apress Media, LLC, 1 New York Plaza, New York, NY
10004, U.S.A. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer-sbm.com,
or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member
(owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance
Inc is a Delaware corporation.
For information on translations, please e-mail booktranslations@springernature.com; for
reprint, paperback, or audio rights, please e-mail bookpermissions@springernature.com.
Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook
versions and licenses are also available for most titles. For more information, reference our Print
and eBook Bulk Sales web page at http://www.apress.com/bulk-sales.
Any source code or other supplementary material referenced by the author in this book is
available to readers on GitHub (https://github.com/Apress). For more detailed information,
please visit https://www.apress.com/gp/services/source-code.
Paper in this product is recyclable.
Table of Contents
About the Author��vii
About the Technical Reviewer��ix
Preface��xi
Introduction��xiii
Chapter 1: Understanding “the Ask”��1

Objective: Asking the Right Questions��1
The Recommendations��2
Decide on the Migration Path, Modernization Techniques,
Enhancements, and the Cloud Vendor��2
Assess the Current Challenges��6
Understand Why Modernizing Data Platforms Is Hard��7
Determine the Top Five Issues to Solve��9
Determine What Is Available On-Premise vs. on the Cloud��10
Create the Meetings Needed Throughout the Project��12
Define Common Terms and Jargon��16
Key Takeaways��17
Chapter 2: Enabling the Security Model��19

Objective: Identifying the Security Considerations��19
PII Columns: RBAC, ABAC Features��21
Central Access Control��27
iii
Table of Contents
Authentication and Authorization (SAML vs. PING, etc.)��31

Strategy for Data Obfuscation��44
GDPR and Other Data Privacy��46
Ownership of the Platform, Interaction with Other
Stakeholders (CISO, Legal Teams, etc.)��48
Legal/Contractual Obligations on Getting/Connecting Data from
a Third Party on the Cloud��50
Chapter 3: Enabling the Organizational Structure��53

Objective: Identifying the Organizational Structure and Role��53
Example Template for the Project��54
Chapter 4: The Data Lake Setup��63

Objective: Detailed Design of the Data Lake��63
Structuring the Different Zones in the Data Lake��64
Defining the Folder Structure of the Zones with a Hierarchy��67
Managing Data Sensitivity as Part of the Folder Structure Design��71
Setting the Encryption/Data Management Keys for Organizing Data��73
Looking at Data Management Principles��77
Understanding Data Flows��81
Setting the Right Access Control for Each Zone��145
Understanding File Formats and Structures in Each Zone��147
iv
Table of Contents
Chapter 5: Production Playground��151

Objective: Production Playground��151
What Is a Production Playground?��153
What Issues Will This Address?��154
What Is a Production Playground Not ?��155
What Does the Production Playground Consist Of?��155
Chapter 6: Production Operationalization��159

Objective: Production Operationalization��159
Chapter 7: Miscellaneous��167
Objective: Advice to Follow��167
Recommendations��167
Managing a Central Framework Along with
Project-Specific Extensions��167
Allowing Project Teams to Build “User-Defined Procedures” and
Contribute to the Central Framework��168
Advantages and Disadvantages of a Single vs. Multi-account Strategy��169
Creating a New Organizational Unit AWS Account vs. Onboard Teams
to a Central IT Managed AWS Account��171
Considerations for Integrating with Schedulers��172
Choosing a Data Warehouse Technology��173
Managing Autoscaling��174
Managing Disaster Recovery��175
v
Table of Contents
AWS Accounts Used for Delivery��176

Data Platform Cost Controls��178
Common Anti-patterns to Avoid��187
Poor Metadata Management��189
Index��195
vi
About the Author
Nayanjyoti Paul is an associate director and
chief Azure architect for GenAI and LLM
CoE for Accenture. He is the product owner
and creator of patented assets. Presently, he
leads multiple projects as a lead architect
around generative AI, large language models,
data analytics, and machine learning. Nayan
is a certified master technology architect,
certified data scientist, and certified Databricks
champion with additional AWS and Azure
certifications. He has been a speaker at conferences like Strata Conference,
Data Works Summit, and AWS Reinvent. He also delivers guest lectures at
universities.
vii
About the Technical Reviewer
Arunkumar is an architect with 20+ years of
experience in the IT industry. He has worked
with a wide variety of technologies in the
data, cloud, and AI spaces. He has experience
working in a variety of industries such as
banking, telecom, healthcare, and avionics.
As a lifelong learner, he enjoys taking on new
fields of study and challenging himself to
master the necessary skills and knowledge.
ix
Preface
This book explains how to implement a data lake strategy, covering the
technical and business challenges architects commonly face. It also
illustrates how and why client requirements should drive architectural
decisions.
Drawing upon a specific case from my own experience, I begin with
the consideration from which all subsequent decisions should flow: what
does your customer need?
I also describe the importance of identifying key stakeholders and the
key points to focus on when starting a project. Next, I take you through
the business and technical requirements-gathering process and how to
translate customer expectations into tangible technical goals.
From there, you’ll gain insight into the security model that will allow
you to establish security and legal guardrails, as well as different aspects of
security from the end user’s perspective. You’ll learn which organizational
roles need to be onboarded into the data lake, their responsibilities,
the services they need access to, and how the hierarchy of escalations
should work.
Subsequent chapters explore how to divide your data lakes into zones,
organize data for security and access, manage data sensitivity, and use
techniques for data obfuscation. Audit and logging capabilities in the
data lake are also covered before a deep dive into designing data lakes to
handle multiple file formats and access patterns. The book concludes by
focusing on production operationalization and solutions to implement a
production setup.
xi
Preface
After completing this book, you will understand how to implement a

data lake and the best practices to employ while doing so, and you will be
armed with practical tips to solve business problems.
What You Will Learn

Specifically, by reading this book, you will
• Understand the challenges associated with

implementing a data lake
• Explore the architectural patterns and processes used

to design a new data lake
• Design and implement data lake capabilities
• Associate business requirements with technical

deliverables to drive success
Who This Book Is For

This book was written for data scientists and architects, machine learning
engineers, and software engineers.
xii
Introduction
I landed at the airport and took an Uber to my customer’s office. I was
supposed to meet with the program manager on the customer side. After
the initial process and getting myself “checked in,” I entered the conference
room that was booked for our team. I knew most of the team from other
projects, but I was meeting a few of them for the first time. After the usual
greetings and a few of my colleagues congratulating me on my new role, I
was ready for the day to unfold.
This customer was a big organization, and there was a clear
“separation of concerns” from multiple teams. The schedule was set up,
and our first tasks were to get acquainted with the different organizational
units, identify the key stakeholders, and understand the stakeholders’
primary “asks.” It was important for my team to understand the key
organizational units and have one-on-one initial discussions. We needed
to connect with the following people and teams:
–– We needed to know the owner of this platform. This

typically includes who will own this data lake as a
platform from the customer’s point of view. Who will
pay the bills and eventually be the key decision-maker
for all technical and business decision-making? We
identified the senior VP of engineering as the key
stakeholder. We set up a one-hour call with him to
understand his expectations and his vision of the
future-state data lake.
xiii
Introduction
–– We wanted to know the team that was handling all the

data and analytics today. As the customer had an
on-premise footprint, we wanted to know the engineer-
ing team who had been managing the entire data and
analytics platform on-premise up to now. Eventually
they would be cross-trained and be the data engineer-
ing team in the cloud after we delivered the data lake.
As all the information of source systems, data onboard-
ing processes, current business reporting needs, etc.,
were managed by them, we needed to understand the
current business process of this team and document
them so that we could draw some parallels for what it
might take to transition those workload and business
requirements into the cloud as part of this journey. We
invited the engineering leads to an initial one-hour call.
–– We needed to connect with the chief information
security officer (CISO) and her team. Venturing into the
cloud was a new entity for my customer. Apart from the
technical questions and recommendations, we needed
to understand the business, contractual, and general
organizational obligations of what was permitted (and
what was not) from a security standpoint. We knew that
every organization has a set of predefined policies that
must be followed. Some of these guidelines come from
geography (like GDPR), some come from industry (like
HIPAA or financial data restrictions), and others may
come from data residency (like data sitting in the
customer’s own on-premise data center versus the
public cloud). Nevertheless, we needed to connect with
this team and understand what these policies meant for
xiv
Introduction
this customer and what considerations we needed to

take when we designing the platform as a whole. We
ended up setting up another one-hour call with
this team.
–– Next we set up a call with the “cloud engineering” team.

This was a new team, and they had started some
groundwork in laying out the “laws of the land,” mostly
in terms of network, services whitelisted, getting access
to a cloud network, access and onboarding of resources
to the cloud system, etc. We wanted to be acquainted
with the current process. Also, from a delivery point of
view, this project was a shared responsibility. Some of
the key aspects that our customer would still be “own-
ing” was the platform management and onboarding
part. Additionally, the strategies around disaster
recovery, high availability, etc., were going to be a
“shared responsibility.” Hence, it was critical for us to
work closely with the cloud engineering team, so we
scheduled a one-hour initial discussion with them.
–– Next was the DBA team. The DBA team currently

owned the databases on-premise but was also respon-
sible for eventually owning any databases, data marts,
and data warehouses that would be set up on the cloud
as part of this program. We set up a one-hour meeting
with them too.
–– Next was the data governance team. One of the key

reasons to move into the cloud (apart from the obvious
reasons of low-lost, easy maintenance, and limitless
storage and compute capacity) was to keep track of and
audit everything that was going on. We believed in a
xv
Introduction
“governance-first” approach, and our customer

believed in that too. They wanted to keep an audit and
lineage trail of everything that would be happening on
the cloud so that the data lake (lake house) did not
become a swamp. An easy and centralized governance
process would make “things” in the data lake very
organized. Additionally, it would introduce data dis-
covery and search capability that would become a
crucial feature for building and establishing a data
marketplace and catalog to “shop for” all the data
(products) hosted on the data lake (lake house).
–– We also connected with the “business” users who were

the key stakeholders of the system. They were sup-
posed to use and consume data or analytics outcomes
from the platform. We had teams like data science,
business intelligence, C-suite executives, etc., who were
waiting to be onboarded onto the platform for different
reasons and rationales. We set up independent calls
with them to understand what “success” meant
for them.
–– Lastly, we wanted to quickly connect with our partner
teams. For example, the public cloud offering was from
AWS, and we wanted to connect with the AWS leads to
understand what was currently in discussion for this
implementation. Similarly, we connected with the
Collibra team that was providing the Collibra software
as an enterprise data catalog solution. Coming from a
consulting company, we have partnerships with both
vendors, and hence it was critical for us to be in sync
with them.
xvi
Introduction
With the key stakeholders identified and meetings set up, it was time
for business. Having dedicated sessions with each key member was critical
to get “buy-in” from each of them for the platform architecture (more on
this to follow in the coming chapters).
Understanding the Requirements

from Multiple Stakeholders’ Viewpoints
In general, implementing a greenfield data lake has many technical and
business challenges. The following are a few challenges that we needed to
think through:
• Establishing a clear understanding of the customer

requirements for a data lake implementation can be a
challenge because of the complexity of the area.
• It can be difficult to determine exactly what data

is required, as well as how it should be stored and
retrieved.
• It is difficult to understand the customer’s desired

outcomes and how they will use the data lake.
• It can be challenging to ensure that the data lake
is secure and conforms to industry standards and
regulations.
• Connecting the data lake with other systems can be a

challenge because of the complexity of the integration.
• It can be difficult to determine the best way to

structure the data lake, as well as how to optimize it for
performance.
xvii
Introduction
• It is difficult to ensure that the data lake is designed for

scalability so that it can accommodate future growth.
• Determining the most effective way to ingest data into

the data lake can be a challenge because of the volume
and variety of data sources.
• It can be difficult to ensure that the data is of high

quality, as well as how to monitor and maintain the
data within the data lake.
• Since the customer requirements will vary from one

organization to the next, it can be difficult to have an
accurate understanding of what is needed and build a
generalized solution.
• Understanding the customer’s security and privacy

requirements can be difficult to interpret, especially if
they are not adequately documented.
• Establishing the necessary data governance

frameworks and policies can be a challenge if there
is not sufficient detail regarding the customer’s
requirements.
• Understanding the customer’s desired access and
usage policies can be difficult to discern without
an appropriate level of detail in the customer’s
requirements.
• Establishing the necessary data quality requirements

can be a challenge if the customer’s requirements are
not met.
xviii
Introduction
The following diagram represents how “success” means different

things to different stakeholders. This illustration depicts an example
of what it means for this particular customer. This is to ensure that we
address and keep each of these success criterion in mind as we move
ahead and start the platform design.
Security was a key stakeholder.
Business was one key IT was another stakeholder who They wanted “right
stakeholder. They had CTO was another stakeholder wanted cloud native solution separation” of duties, manage
challenges to run business who wanted a modern data to keep a clean architecture and “blast radius” and ensure
insights over longer period of platform on-cloud minimize integration issues proper controls
data.
1.0 2.0 3.0 4.0
Data Scientists were key Data Engineering team were

stakeholders
holders who wanted a key stakeholders. They wanted CISO was another stakeholder.
seamless
less data access with a solution around re-usable, They want right data
ty to be ab
capability aable
le to perfo
f rm
perform repeatable &“low-code” fo fforr governancee via classifi
governanc classification
f cation
self-service
service withoutt IT plumbing
entire data plumb
m ing and Role based access control
7.0 6.0 5.0

Cloud Engineering Tea
T
Teamm were
Business Analyst were key key stakeholders. They wanted
stakeholders who wanted to right ‘guardrails” , process
analyze and build reports of controls and Operations
single source of truth
““single truth”” management
8.0 9.0
If we look closely, the first stakeholders are from the business side. For
them, the objective is outcome focused. The technology is secondary for
them as long as we continue delivering high-quality business insights in a
repeatable and predictable time frame.
Second are the stakeholders from the CTO’s office. They want to design
the platform (data lake) as a future-ready solution. For them it is important
to make the right technical decisions and adopt a cloud-first approach.
They want to focus on a modern data stack that centers around cloud-
native and software-as-a-service (SaaS) offerings.
Next, the customer’s IT organization is a key stakeholder. Their focus is
to incorporate technical solutions that are easy to maintain, cloud native,
and based on the principles of keeping the integrations minimal.
Next in line as a key stakeholder is the security office team. They
want to ensure that we design a system that has the right “separation of
concerns” and has the right security guardrails so that confidential and
personally identifiable information (PII) data can be safe and secure.
xix
Introduction
Next in line is the CISO’s team for whom the data access policies, data
governance and auditability, etc., are primary concerns. They want to
ensure that the data is available only to the right resources at the right time
through role-, tag-, and attribute-based access controls.
Next in line is the data engineering team who will eventually “own”
the applications and system for maintenance. For them it was important
that the data engineering solution built on the data lake has reusability,
extensibility, and customizability, and is based on a solid programming
framework and design that will be easy to manage and use in the long run.
Next in line is the data scientist community who needs the right access
to the data and right access to the tools to convert the data into insights.
They also want “self-service” as a capability where they have the right
permissions to work on ideas that can help the business get value.
Next in line is the business analyst community who want to be
onboarded into this new data lake platform as soon as possible with access
to a “single source of truth” so that they can start building the mission-
critical application that the business is waiting for.
Finally, the cloud engineering team is a key stakeholder. This team
wants the whole platform to be secure, controlled, user friendly, reliable,
and durable.
As you might have imagined by now, I will be using my experience to
explain the end-to-end process of designing and implementing a data lake
strategy in the following chapters.
This book will (in broad strokes) cover concepts such as how to
understand and document the business asks, define the security model,
define the organization structure, design and implement the data lake
from end to end, set up a production playground, and operationalize the
data lake. Finally, I will present some lessons learned from my experience.
Chapter 1 will focus on each of these points and how each resulted in
the design of a small part of the key problem (platform design) and how
little by little things fell into place for me and my team. Let’s get started.
xx
CHAPTER 1
Understanding “the
Ask”
Objective: Asking the Right Questions
In the introduction of the book, I set the stage for the project we’ll start
discussing in this chapter. When I took up the solution architect and
delivery lead role, I had no idea what vision my customer had, other than
a very general understanding of the final product my customer was after.
The intention was to build a modern, cloud-centric data and analytics
platform (called a lake house). So, at this point, it was important for me
and my team to ask the right questions, gather the requirements in detail,
and start peeling back the layers of the onion. In short, we needed to
understand “the ask.”
The first ask (for my team and me) was to be aligned to the customer’s
vision. To understand this vision, we set up a meeting with the VP of
engineering (the platform owner) to establish the direction of the project
and the key decisions that needed to be made.
© Nayanjyoti Paul 2023 1

N. Paul, Practical Implementation of a Data Lake,
https://doi.org/10.1007/978-1-4842-9735-3_1
Chapter 1 Understanding “the Ask”
The Recommendations
I used the following checklist as part of the vision alignment, and you can
use this for your project too. Also, be open to bringing your own questions
to the meeting based on your customer’s interests and their maturity.
• What are the migration path, modernization
techniques, enhancements, and cloud vendor that will
be used?
• What are the current challenges?
• Why is modernizing data platforms hard?
• What are the top five issues that we want to solve?
• What is available on-premise and on the cloud already?
• What meetings will be needed throughout the project?
• What common terms and jargon can we define?
My team and I started the first round of discussions with the key
customer stakeholders. We then understood the requirements better and
had a better appreciation of the direction our customer wanted to go in.
Each of the seven topics listed previously are detailed in the remainder of
the chapter.
ecide on the Migration Path, Modernization

D
Techniques, Enhancements, and the Cloud
Vendor
After the usual greetings and formal introduction, we sat down to start
documenting the vision. We understood that the requirement was to build
a cloud-native and future-proof data and analytics platform. Having said
that, the high-level objective was very clear. The data lake design was
2
supposed to be sponsored by the business, and they had strict timelines to

ensure we could get 25 highly important reports ready. Of the 25 reports,
most of them were to be built on business logic after bringing in data from
the system of records, but a few of them were to be powered by machine
learning predictive models. For us, that meant that the business had a very
specific “success criteria” in mind, and as long as we could deliver on the
business promise (through a technical capability), we could deliver value.
Even though the outcome was business focused, the enabler was
technology. We wanted to design the architecture “right” so that we
could have a sustainable and adaptive platform for data and analytics for
the future.
We started asking specific questions focused on whether the customer
had already chosen a cloud partner. This was critical as we wanted to be
cloud-native and leverage the capabilities each cloud vendor provided. In
this case, the customer already had decided on AWS. Questions around
whether the requirement was to modernize an existing architecture,
migrate a similar technology, or enhance an existing setup were important
for us to understand. Table 1-1 provides a quick reference for each
question we asked and why it was important.
These questions can add value to any project during the initial
understanding phase. Feel free to use Table 1-1 as a baseline for
documenting the basic premises of the offering you are planning to deliver
for your customer.
3
Table 1-1. Assessment Questions

Questions Why Was the Question Important? What Was Decided?
What cloud Each year cloud vendors introduce new The customer’s
platform to use? capabilities, features, and integrations. decision to go with
By being aligned to a cloud vendor’s AWS ensured (for
capabilities, we can understand the example) that we
“out-of-box” offerings versus gaps for could leverage its
that specific vendor. Also this means ML capabilities on
a correct estimation for time and cost Sagemaker, their
based on the maturity of the vendor and centralized RBAC and
the capabilities they currently offer. TBAC policies through
lake formation, and
many more (more on
those later).
(continued)
4
Table 1-1. (continued)
Questions Why Was the Question Important? What Was Decided?
Do you want Each of these solutions needs separate The customer was
to implement handling and enablement from a very clear that they
a lift-and-shift, technical point of view. wanted a data lake
modernization, or For example, lift and shift should focus in the cloud, which
migration solution on a path of least resistance to have the meant they were
strategy? same capability available in the cloud. ready to open up new
So, an Oracle system on-premise can possibilities, new
be deployed as an Oracle system on the personas, new kinds
cloud. of use cases, and new
opportunities for the
Migration is slightly different; for
whole organization.
example, the same Oracle system
can be migrated to a Redshift system
on the cloud leveraging native cloud
capabilities but keeping the basics
intact.
However, modernization can mean
replacing an on-premise system like
Oracle with a data lake or a lake
house architecture where we can
enable different personas such as data
engineers, analysts, BI team, and the
data science team to leverage the data
in different ways and with different
forms to get value.
5
Assess the Current Challenges

Even though the vision from the customer was for a new, modern data
platform, it is always important to understand why the customer has
decided to take that initiative now, including what challenges have become
important enough that they could not sustain the existing solution. Also,
documenting their current challenges provides a great way to evaluate
“success” and measure the outcomes. The following were some of the
critical challenges that were high priority for our customer in this example:
–– The current setup was costly. The software vendors for the
commercial off-the-shelf (COTS) products were charging a
license fee based on the number of machines. As the
organization was growing, so was their user base.
–– The current setup could not scale up based on the

organization’s needs, seasonality, user personas, etc.
–– As the data volume was growing, the current trend of analytics

was very slow and restrictive. There was no option for machine
learning, predictive modeling, unstructured data analysis, etc.
–– As the organization was gearing up for the future, they had

started investing in data scientists, data analysts, etc. The
organization had started recruiting new talent, and it was
important to build a platform that helped them bring in value.
–– Time to market was essential, and a process that can provide

“self-service” capabilities and quick prototyping features can
unlock a lot of capabilities for the customer.
–– They wanted to be future ready. Peer pressure is a huge

motivation. As other organizations in the same space were
adapting to the world of cloud-native and cloud-centric
solutions, it was important for our customer to not fall behind.
6
Some of these points were critical for the customer, and hence we ensured
that when we designed the solution, we considered the people who would
be using the platform and what capabilities the final platform should have.
nderstand Why Modernizing Data Platforms

U
Is Hard
Along with identifying the customer’s challenges and issues that they
were currently facing, it was important to have an open discussion on the
challenges other customers have faced (in similar domains) and what we
had learned through our experiences (lessons learned). Figure 1-1 provides
a quick reference for our past experience, which we thought would help
this current customer to see a pattern and help us avoid common gotchas.
Identify key
Identify Current What are the current Identify technical and
stakeholders and
Challenges limitations business issues
owners
Identify key Identify and create

Identify High Priority stakeholders and
Schedule meetings
buckets of
Assign prioritization to Identify Long and
with key stakeholders buckets Short-term priorities
Items owners requirements
Identify stakeholders Identify scope to Document key Identify critical decision

Identify Key within business, CISO, ensure outcome has technology decisions paths for each
Stakeholders security etc. business value and document debts stakeholders
Divide the project

Decide on Pilot/ POC
Time and Effort for scope and high-level
Identify project scope between Identify project
Project Plans structure business & technical management style
roadmap
requirements
Create actionable work Setup deployment

Start project Setup daily Document key
items and start Get sign-off on artifacts Build and test strategy along with
implementation whiteboarding sessions decisions
assigning work devops
Realign with key Plan for phase 2 with

Continuous Monitoring Stakeholders re-prioritization
Plan for user training
Figure 1-1. High-level step-by-step process of organizing the project

through different phases
Along with the Figure 2-1 pointers on what we should focus on while
delivering an enterprise-scale data platform solution, Figure 1-2 provides
guidelines for a target-state implementation as part of an end-to-end data
platform implementation.
7
Storage, zones based Roles, Permissions Right governance and

Design of right Design of Exploration
Platform & number of accounts
on data sensitivity and access
Zone for analytics and
team to manage the
and organizational (interactive vs
Infrastructure for data-lake
units automation)
data science
platform and manage
operations
Metadata, Data Rules Engine for

Data Grooming and Classification and Curation, Data Quality Engine
Pipeline scaling and Policy Driven Pipeline
Data Supply Chain automation
Scaling of Data integration with Transformation, for checks and
Configuration
Acquisition catalog systems, data Enrichment, validation
marketplace etc. Compliance etc.
ML
Framework around
Fit for purpose data Decision of right Operationalization –
ETL for data model management
models that support technology for data
Data & ML Products business usage products
processing
from identifying data
to building models @
and collaboration of
data science team
Scale
Integration with Scheduling and

Governance around
Operations & Operations automation around
approvals and
Cleanup, Purge and
Management management and error deploying pipelines
promotion
Archive
management (data and ml)
Figure 1-2. Holistic view of end-to -end capabilities needed for a

data strategy project (from a technical view)
At the minimum, we think that an enterprise-level implementation

should focus on four things: the platform and infrastructure, the data
supply chain, the data and ML product creation, and finally the operations
management. Within each of these four verticals, there are some specific
requirements and capabilities that need to be addressed.
For example, the platform and infrastructure should focus on the right
“account strategy,” data sensitivity issues, roles, permissions, zonal designs,
governance models, etc. The data supply chain should focus on pipeline
generation, scaling of data engineering processes, metadata management,
rules and the data quality engine, etc. The data and ML product creation
should focus on ETL, fit-for-purpose solutions, optimizations, etc. Finally,
the operations management should focus on scheduling, orchestration,
etc. Figure 1-2 is just a representation, but it provides a blueprint in
ensuring that we think through all these capabilities while designing and
implementing the end-to-end enterprise data platform.
8
Determine the Top Five Issues to Solve

This is similar to the previous point discussed. However, the key
differentiation is the process of collecting this information. To understand
the top five issues, we started interviewing almost 50+ key associates
and documenting the top issues they faced. We collected and collated
the answers to our questions across different organization units based
on where the key stakeholders were aligned. The result of the interview
process was a list of common issues faced across the organization.
When we looked at the report, we found many repetitive and common
challenges. Those challenges surely impacted a lot of business units and
hence were high on the priority list for us. Here are a few examples of the
common challenges:
–– The customer needed a central data hub. Different business

units had independent silos of data repositories, which were
either stale or out of sync.
–– There was no single 360-degree view of data. Every business

unit could see only their own side of data.
–– Governance was nonexistent. There was no central catalog

view of organization-wide datasets.
–– Time to market was slow because of the limitations of the
technology.
–– The cost of management and maintenance was a

fundamental issue.
What we discovered from this process was aligned to what we

expected from the previous step, but seeing a repetitive pattern gave
us the confidence that we had been following the right path so far and
documenting the right issues. Figure 1-3 gives a pictorial view of this.
9
Figure 1-3. A pie chart of what we saw were the driving factors for
the need to build a data lake solution
etermine What Is Available On-Premise vs. on

D
the Cloud
It is important to understand our current situation and assess customer
maturity before committing and undertaking any journey. Next, what we
did with our key stakeholders was to understand from them where they
stood and assess where they were currently in their vision. This would help
us to offer the right help and guidance to reach the goal.
First, we sat down with the cloud security and infrastructure team to
understand if the customer had started any journey toward AWS (their
chosen platform). Next, we wanted to understand if any guidelines,
corporate policies, and/or best practices were documented. Table 1-2
summarizes what details we got from the team. (Use this as a guide for
your project, but ensure you have a customer document these, as they will
become the rules of the game.)
10
Table 1-2. Maturity Assessment Questionnaire

Questions Maturity Assessment
Has the organization In this case, the customer had a well-established

started the journey cloud engineering practice. However, they had not
toward the cloud in implemented any large-scale implementation in the
practice, or is it still a cloud. It had only a few small proofs of concept for a
“paper exercise”? smaller group within the organization.
Does the organization The customer had documentation around cloud policies
have any standard and best practices. However, the customer wanted us to
security or cloud review them, find gaps, and propose a better approach
practices documented for the future.
already?
Who are the personas The customer wanted the platform to be built to be
(teams) with access to future proof and ready for other organizational units to
the current on-premise feel secure enough with it to onboard their analytics
data warehouse the workload. This meant that we had to think beyond what
customers are hosting? the current on-premise systems provided in terms of
role-based, attribute-based, and domain-based access to
Is the intention of the
data and build a solution that would provide a separation
customer to onboard
of concerns for each team who would use the platform
other personas in the
and onboard their data for analytics.
new data platform (when
ready), and will this imply
a different set of access
policies and practices?
(continued)
11
Questions Maturity Assessment
Have the consumption The simple answer was yes. A major focus was to
patterns changed? Are onboard the data science teams and enable them to
there new parties and build cutting-edge use cases to help do predictive
use cases that would insights on data rather than reactive ones. Similarly, a
be adopted on the new new kind of data analytics and BI teams would need
platform? instant and live access to the data to build and refresh
metrics for the business to help in quick decision-
making. Those personas and their set of use cases were
completely new and unknown and would surely need a
different design approach.
Do you want to be Most customers start with the idea of setting up a
provider agnostic or cloud-based system targeting a specific cloud provider
multicloud (from a for partnership. However, soon clients decide to have
strategy point)? a multicloud strategy that is provider agnostic. These
decisions do not impact the solution strategy in the short
to medium run, but they do have implications in the long
run. For this customer, they did not have any preference
about this, and we were supposed to focus on the AWS-
specific solution for now.
reate the Meetings Needed Throughout

C
the Project
Implementing a large-scale project is always challenging. Typically when
we have sprint-based programs and each sprint is 2 weeks, it is important
to think ahead and plan for the upcoming tasks. So, we wanted to
identify important meetings and get them on the calendar. This included
12
identifying the priority and ordering of tasks and ensuring we got calendar
time from each stakeholder so that we did not have to wait for any
important decisions from our customers.
We enabled three workstreams. I ensured we had dedicated teams for
each of the three workstreams, and each had specific responsibility areas,
as listed in Table 1-3. You can use this table to plan ahead for important
meetings with the right stakeholders.
Table 1-3. High-Level Workstreams with Their Typical

Responsibilities for a Technical Data Lake Implementation
Workstream Main Responsibilities
Business − Identify and prioritize source systems that need to be onboarded

analysis and into the new platform.
grooming − Identify which datasets from which sources need to be priority 1.
− For each source, “groom” the dataset based on data profile, types
of data, type of interactions, frequency of loads, and special data
handling needs (version of data versus snapshot, etc.).
− For each dataset, document basic data quality checks, common
issues, common standardization needs, and common enrichment
needs required.
− From a consumption point of view, clearly document the ask,
expected business outcome, and samples of output.
− From a consumption point of view, clearly document the business
logic for converting source datasets into the expected outcome.
(continued)
13

Data security − Work with the data security teams, CISO teams, and cloud
engineering teams, and have a common understanding of how
many AWS accounts are needed, how many environments are
needed (dev/UAT/prod), how to separate out the concerns of “blast
radius,” how to manage data encryption, how to manage PII data,
how to implement network security on data onboarding and IAM
policies, etc.
− Identify and document processes to define how to onboard a new
source system and what access and security should be in place.
− Identify and document processes to define a user onboarding process
through AD integrations, IAM policies, and roles to be applied.
− Have separate capabilities between interactive access and
automated access and have different policies, services, and
guardrails for both types.
− Understand and document life-cycle policies and practices for
data and processes.
− Understand and document a role-based matrix of who will be
getting access to this new platform and what will be their access
privileges.
− Define and document a DR strategy (hot-hot, hot-cold, cold-cold,
etc.).
− Define and document how third-party tools will be authenticated
and how they will access data within the platform (temp
credentials, SSO etc.).
− Define and document role-based, attribute-based, domain-based,
tag-based data access, and sharing needs.
− Define and document data consumption roles and policies, etc.
(continued)
14

Data − Design and document architecture for building a cloud-native

engineering and cloud-centric data lake strategy.
− Design a framework for a reusable and repeatable data
ingestion mechanism.
− Design and document ingestion patterns and processes based
on source types, source systems interactions, frequency (batch
versus streaming etc.), data formats, and data types.
− Design and document a framework for data cleansing, data
quality assessment, and data validation and checks in an
automated and reusable way.
− Design and document a framework for data enrichment,
data standardization, data augmentation, and data curation
in a reusable and repeatable way.
− Design and document a framework to capture the metadata
of a business, operational, and technical nature and sync up
with a catalog of choice.
− Design and document a data reconciliation and audit balance
framework for validating data loaded into the system.
− Design and document a framework for building a
data-reconciliation process for versioned datasets that might
have changing dimensions.
− Design and document a framework for building a business
outcome (ETL) process in an automated and reusable way.
− Define and coordinate with other teams to understand existing
and “to be” engineering processes for DR strategy.
− Define and coordinate with other teams to understand and
engineer processes for the data access in an automated way.
− Design and coordinate with third-party tools for data catalog, data
governance, scheduling, monitoring, etc.
15
Define Common Terms and Jargon

Probably the single most important activity to kick off any project is
the task that is needed to bring everyone on the same page. I have had
challenges in my previous projects where we did not have a chance to
align on the common terms and jargon. That always led to multiple issues
and challenges for any technical discussion and architecture process
throughout the project.
Here are a few examples where we wanted to align on this project:
–– A common definition of data storage zones. Examples are raw

versus curated versus provisioned, or bronze versus silver ver-
sus gold.
–– Clear responsibility and features for the zones. Examples include

what controls these zones should have versus what kind of data
and life-cycle policies should the zones have.
–– Common definitions for tenant versus hub versus spoke.
–– Common definitions for dev versus UAT versus prod versus

sandbox versus playground.
–– ETL versus ELT with regard to the cloud platform.

–– Common philosophy of loading data on-demand versus loading
all data and processing on an ad hoc basis.
–– Common philosophy for default personas and intended access

control to data within the data lake.
This was an important alignment where we as a team not only

interacted with customers for the first time, but we made great progress
in terms of clearly documenting what was to be delivered in the
subsequent weeks.
16
Key Takeaways
To recap, we met with all the key stakeholders including our sponsor
for the data strategy work. We interviewed key personnel and identified
key areas (to prioritize), and we understood the current landscape and
maturity. We devised a strategy to work on three workstreams and defined
key meetings and whiteboard sessions for the next few weeks (putting
meetings on calendars for key personnel). Last but not least, we defined
common terms and presented what our focus would be and the possible
measure of success for this project.
Based on the series of discussions, in general our goal for the next steps
were as follows:
Understand the customer’s requirements: The

first step is to understand the customer’s specific
requirements and goals to develop a plan to achieve
them. This includes understanding the data sources,
data types, data volume, and other factors that may
affect the design of the data lake.
Design the data lake architecture: After

understanding the customer’s requirements, the
next step is to design the data lake architecture.
This includes selecting the appropriate storage
technology, selecting the data ingestion and
transformation tools, and designing the data flow
and data management framework.
Develop the data lake: Once the architecture

is designed, the team can start to develop the
data lake. This includes setting up the necessary
infrastructure, building the data ingestion and
processing pipelines, and managing the data lake.
17
Test and deploy the data lake: After the data lake is
developed, it needs to be tested and deployed. This
includes testing the data lake to ensure it meets
the customer’s requirements and deploying it in a
production environment.
Monitor and optimize the data lake: Once the

data lake is deployed, it’s important to monitor
its performance to ensure it’s meeting the
customer’s goals.
18
CHAPTER 2
Enabling the Security

Model
Objective: Identifying
the Security Considerations
My responsibility as part of workstream was to define, design, and
implement a holistic security model for the data platform.
My fundamental objective was to work closely with the customer’s
security and cloud engineering teams and with the AWS team to define a
security blueprint that could help with the customer’s platform, data, and
application security considerations.
As we had already set up the important meetings ahead of time, we
started having initial one-on-one meetings with each of the key security
stakeholders (both internal and external) to document and design the
key decision points (through knowledge discovery in data [KDD]) needed
for designing the security blueprints. We eventually brought all the teams
together to agree on the common solution and socialized the outcomes.
This approach ensured we did not waste everyone’s time and ensured we
had targeted questions for specific groups and tangible outcomes designed
and approved by each group.
© Nayanjyoti Paul 2023 19

N. Paul, Practical Implementation of a Data Lake,
https://doi.org/10.1007/978-1-4842-9735-3_2
Chapter 2 Enabling the Security Model
The Recommendations
I used the following key design decisions to come up with a blueprint and
ensured that those KDDs addressed the needs of each stakeholder. The
objectives of the internal and external stakeholders were different. For
example, the internal teams wanted a security blueprint that focused on
a separation of concerns, the right access and security controls, and tight
integration with enterprise security principles and policies, whereas the
external stakeholders asked us to focus on cloud-native and best-of-breed
technologies and the right service model to build the solution.
The following checklist was part of the vision alignment, and you can
use this for your project too as a template. Be open to asking your own
questions based on your customer’s interest and their maturity (in other
words, use this as a starting guide).
• PII columns: RBAC, ABAC features
• Central access control
• SAML vs. PING, etc.
• Strategy for data obfuscation
• GDPR and other data privacy
• Ownership of the platform, interaction with other

stakeholders (CISO, legal teams, etc.)
• Legal/contractual obligations on getting/connecting

data from a third party on the cloud
Each of these is detailed in the remainder of the chapter.
20
PII Columns: RBAC, ABAC Features

As we were bringing in data from third-party sources and external vendors,
the chances of bringing in sensitive data is high. On top of that, the data is
owned by different organizational units, which begs the question, is it OK
for a group of people to have access to certain PII data that is specific to
that organizational unit but cannot be accessed by other units?
The following are both the challenges and the requirements for PII
column mapping from the requirements we received from the customer:
• Customers needed a single source of truth for all their

data and analytical needs. Currently the data across the
organization was siloed, which was one of the major
reasons for this customer to venture into a data lake in
the cloud. Hence, it was important for the data strategy
to have open access to the datasets. However, the PII
columns should be treated differently. The access to PII
data should be based on a “need-to-know” basis.
• Each dataset needs to be tagged with a classification

level, typically ranging from Confidential to Public.
Confidential-tagged datasets have different encryption
keys, and access to those datasets were on an on-
demand basic (not open for all).
• Each column has a sensitivity level (typically L001,

L002, etc.). These sensitivity levels should govern which
columns of which datasets can be accessed by default
versus which ones need special access.
• Datasets are organized as data domains (within business

domains). Some of these datasets should be handled
with the utmost care. For example, the HR team or
finance team can access salary information, but other
organizational/business units should not have access to it.
21
• Special roles and access grants should be allowed

for accessing sensitive data. As this is an open data
lake platform, personas such as data scientists or
data analysts can request access to certain sensitive
information based on business case and justification.
Policies and processes should be in place to enable
users and roles access to sensitive information for a
specific duration of time.
• Access permissions should also be controlled based on

the consumption and interaction pattern. For example,
automated access to data for processing might have full
access to all columns and datasets to ensure a quick
and repeatable way of building data transformation
and ETL jobs. However, the ad hoc and interactive
access should be restricted based on the role and
persona group the person/resource belongs to.
• Third-party tools that access data from the data lake

should also respect the access control and guardrails
defined in the data lake. They should impersonate
the user who needs access to data via the third-party
tools or have SSO integration for specific service role–
based access.
Figure 2-1 provides a glimpse of the overall process that was followed
for this customer based on the AWS stack selected for the project. The idea
was to have a data strategy design (more to follow in the next chapters)
of organizing the structure of data into Raw (or Bronze), Curated (or
Silver), and Provisioned (or Gold) for the Automated (ETL jobs, etc.) and
Playground (ad hoc or interactive) access perspective. For the interactive
access process, the access control was defined at a granular level (tag,
sensitivity, and domain level) and was based on AWS Lake Formation
22
(based on the AWS technology stack selected). Access to any “curated”

datasets had to be done based on automated policies registered while
onboarding data in the data lake through the data pipelines. All these
automated access policies were stored in the Lake Formation service of
AWS centrally, and access to the data through any service (like Athena,
Redshift Spectrum, etc.) was done through the Glue catalog managed and
governed by the Lake Formation service.
let’s
Figure 2-1. A high-level view (with AWS technology stack) for a

governed data lake
We started the security and access control design by taking baby steps
and handling the architecture on a use case by use case basis. We wanted
to have a baseline architecture first and then test our hypothesis by laying
out additional use cases and validating whether our architecture could
stand the test of the same.
23
The following were the measures we took and the guidelines we

followed to build the first iterations:
1. We created a multi-account AWS strategy to

maintain the blast radius. The multi-account
strategy can be based on three levels: ingestion
accounts, processing accounts, and consumption
accounts.
2. For the previous step, ingestion accounts were

isolated by source type such as external third-party
sources, internal on-premise sources (from within
organization), and cloud based (data loaded from
other existing AWS accounts).
3. For processing accounts, we kept a centralized

account but ensured those accounts did not store
any data. This processing account can assume one
of multiple service roles and process data.
4. The consumption accounts were more flexible.

We started by dividing AWS accounts based on
interactive access or automated access. However,
soon we had to expand automated access accounts
into multiple hub versus spoke architecture as
multiple organizational units wanted to own and
manage their own “data products.” Similarly, we
had to scale up interactive access into multiple AWS
accounts because of multiple independent teams
and their needs to have a “self-service” capability for
delivering business insights.
24
5. Once we decided on the AWS account setup, we

tried to finalize the data encryption strategies. Each
account had a bunch of AWS KMS CMK keys. We
divided the keys into tier 1 to tier 3 keys. Based on
the sensitivity of the datasets identified, we pushed
data into independent buckets that had the default
CMK keys associated with them. The service roles
had access to those keys.
6. Once the encryption strategies were in place, we

ventured into role-based, domain-based, and tag-
based access control policies. Each dataset when
being onboarded into the data lake was associated
with three tags: business domain tags (like finance,
marketing, general, etc.), data sensitivity tags
(confidential, public, etc.), and column-level PII
tags (L001, L002, etc., where L001 meant no PII, and
L002 meant it has partial or entire PII information
such as date or birth along with full name). We spent
considerable time and effort discussing these with
business and the CISO to come up with the tags.
7. Once the tags were in place, we introduced AWS
Lake Formation. AWS Lake Formation is a service
that allows a central access control and governance
platform to enforce data access policies. Typically,
Lake Formation ensures that the client applications
(like AWS Athena, etc.) authenticates itself to access
any data in S3. The authentication process grants
temporary credentials based on the user’s role.
Internally, Lake Formation then returns only those
datasets, columns, etc., that the user has “grants” for.
25
Hence, in our example, users who are from ROLE-A

that belongs to ORGANIZATION UNIT (or business
domain) B can query only those datasets that are
tagged for ORGANIZATION B usage (or tagged for
GENERAL usage). Additionally, the ROLE-A can
view only those columns of the mentioned datasets
that are tagged with either L001, L002, or L003 based
on the tags allowed for ROLE-A.
8. Once the tag and role-based access were set up,

we wrapped up the security and access control
based on the consumption pattern. In this case,
we focused only on AWS Redshift, and hence
we defined policies for Redshift access and data
sharing through IAM roles (more on Redshift in the
upcoming chapters). Redshift was used to register
data products that were owned by independent
domain/business organizations, and we ensured
the access control follows the same philosophy as
mentioned earlier.
9. Lastly, we enabled the Playground area as a logical

extension of the production setup. We enabled
guardrails and processes to access data and services
in the playground. This was mostly for data science
interactive access. Chapter 5 talks about enabling
the data science playground.
Figure 2-2 shows how the overall Lake Formation setup might look
(from a high level).
26
Another random document with
no related content on Scribd:
BILIBID.
As we are going to press, there comes to hand a little pamphlet
describing the industries and production of Bilibid.
Why not send our wardens who desire to do things to Bilibid?
Perhaps, it would be better to send our legislators, who after
observing the practical achievements of Bilibid may be induced to
authorize our wardens to inaugurate a sound industrial policy.
Where is Bilibid? Take the train for San Francisco, engage passage
on some leviathan of the deep and get off probably at the second
station which is Manila. Thence it is a short excursion to Bilibid, a trip
taken by twenty thousand visitors in a single year, not to mention
those who take involuntary trips thither.
Forty buildings, seventeen acres of ground, plan of main building like
Eastern Penitentiary, one of the best ever constructed if we consider
continual inspection as an essential factor. 2800 prisoners there; as
many others in prisons elsewhere in the islands but all co-ordinated
under a central administration.
The great aim is to prepare the inmates for “honorable position in the
community upon their release.”
The men work and play. We enumerate some of the industries.
1. Manufacture and repair of carriages, wagons, carts, trucks,

trailers, etc.
2. Household and Office Furniture made of mahogany and other
native beautiful woods.
3. Concrete work, construction of buildings.
4. Rattan and bamboo furniture. The famous fan-back chair.
5. Hand-wrought articles in silver, gold and other metals; shell-
work, horn specialties.
6. Tailoring. Service uniforms.
7. Steam laundry. All the work for the inside and a “considerable
civilian trade.”
8. Embroidery, lace making, crochet organized for the female
department.
Output per annum $350,000

Profit for the government 100,000
There are two penal colonies on large tracts of land, on one of which
the 1200 colonists practically have a government of their own.
The San Remon Farm is where the non-Christian convicts from
Moroland are confined. These war-like people have admirable
qualities when treated properly, and three-fourths of the inmates are
at work on the extensive farm without the presence of an armed
guard. Here is found the model prison of the Orient. “Built of
reinforced concrete, with grilled walls, dormitories, shower baths,
and with every modern feature for the comfort, health and
reformation of prisoners confined there, it has proved a wonderful
educational institution for the Moro.” The entire credit for this building
and the admirable system is due to the genius and sagacity of the
former governor, General John J. Pershing.
EMPLOYMENT OF PRISONERS FOR
THE GOVERNMENT.
Early in last July, a very earnest assemblage of wardens and
superintendents of prisons, and members of the American Prison
Association held a conference in Washington with a view of
mobilizing the prison industries so as to be helpful to the government
in these times of scarcity of labor.
There were forty delegates in attendance, mainly appointed by the
Governors of twenty-two States.
After discussions lasting for several sessions, the Conference was
unanimous in making certain recommendations.
1. Already the penal institutions possess enormous acreage. If the

government will accept the product, the crops may be greatly
increased. Over a half million acres are available and 75,000
workers.
2. In many States, where the law permits, selected inmates can be
paroled to labor for farmers at reasonable wages. Extensive
development of this method is possible.
3. Many of the prisons support industries the outcome of which
may be very serviceable. Among the chief industrial products
are socks, shirts, underwear, blankets, mattresses, boots and
shoes, overalls, harness, and army and navy equipment.
4. The executive order of 1905, providing that no prison-made
goods shall be purchased by the National Government, ought at
once to be rescinded.
5. “We are convinced that a very large majority of the inmates of
our prisons and reformatories are ready and earnestly desirous
of ‘doing their bit’ for the country.”
To accomplish these results, a bill has been introduced in the U. S.
Senate (S. 3076) and in the House of Representatives (H. R. 7353)
whose purpose is to utilize the labor of Federal and other prisoners
in manufacturing government supplies.
The prices paid are to be the market prices current in same locality
for same commodities.
The hours of labor are to be the same as the time prevalent in the
same vicinity for the same sort of labor.
The same rate of wages is to be paid to the prisoners, subject to the
necessary deduction for maintenance.
This means that the goods purchased by the United States from the
prisons shall be made by labor which is on a par with outside labor.
No goods are to be purchased by the Government from “any private
person or companies using the labor” of convicts. Thus the
Government utters its protest against any form of “contract labor.”
The Acting Committee of The Pennsylvania Prison Society has
urged the early passage of this bill.
A FEAST OF REASON AND A FLOW
OF SOUL.
Within the last two years, in the Empire State, some prison officials
and students of penology have occasionally met for an informal
conference on methods of penal management. No reporters attend,
they have no Secretary, there are no restrictive features, the
discussions are frank, free and open. They have been held at some
one of the institutions, and so the visitors have opportunity to
observe from the inside the methods and employments of the prison
where the meetings may be held.
Those who attend these conferences are the guests of the institution
which is visited. They have been held at the Elmira Reformatory, at
Great Meadow, at Blackwell’s Island and possibly at one or two other
prisons.
Last summer it was the good fortune of the Secretary to have the
privilege of attending the meeting held at Great Meadow at
Comstock, N. Y. Perhaps there were fifty ladies and gentlemen in
attendance, among whom were Jas. M. Carter, Superintendent of
Prisons for the State of New York; Warden Trombly, of Danemora;
William George, founder of the school which bears his name; ex-
Warden Geo. W. Kirchwey, O. F. Lewis, Secretary of the New York
Prison Association; Miss Katherine B. Davis, Commissioner of
Parole and Probation, New York City, and various officials connected
with the New York City prisons and other penal institutions of the
State.
Arriving at Comstock the guests were met by our genial host,
Warden Homer, whose conveyances soon brought us to the
Administration Building. There were excursions about the big farm
and the various buildings. Bounteous meals were served under a
tent on the grounds of the shady lawn. The ladies were lodged in the
administration building. It was at one time supposed that the
gentlemen would occupy cells in the regular prison department, but
for some reason this very interesting proposition was relinquished,
and a garage was converted into an airy dormitory for the
accommodation of the gentlemen. Prisoners brought iron bedsteads
and bedding and nothing was omitted for the comfort of the guests.
A shower bath was improvised, and in the morning the barber and
the shoe polisher appeared with all the proper accoutrements.
The guests arrived on Friday morning and departed the next
afternoon. There were three informal meetings, at which a variety of
penological subjects were both lightly and profoundly discussed.
Proceedings are not to be published, hence there is no feeling of
restraint.
The writer trusts he will not violate the confidence of his friends if he
may refer to one or two points in the discussion.
One of the prisoners on the farm—by the way an ex-member of the
New York State Legislature—addressed the company in emphatic
recommendation of The Honor System as employed by Warden
Homer. He stated that the prisoners there did not care for any
extension of the self-government plan. He was sure that Warden
Homer could govern them better than they could govern themselves.
They were as comfortable as any persons restrained within limits
could be supposed to be. They knew they could get a “square deal”
from Warden Homer, and they did not care to shift the responsibility
of government to any other shoulders.
An enthusiastic supporter of the self-government system was rather
inclined to look with disfavor on a benevolent despotism, such as the
system now in vogue at Great Meadow. The persons so governed
lost initiative and the power of thinking for themselves. They had no
opportunity of profiting by their own mistakes. They became mere
puppets, and were not learning the practical lessons which would fit
them for the life outside.
The company listened with interest to an expert dietitian who spoke
of the crude and ragged methods of preparing food in the penal
institutions. Sufficient food was provided as a rule, but it was ruined
in the preparation.
Mentally, morally, socially and gastronomically, we may state, the
meeting was a success.
Last October we enjoyed a similar conference at Sleighton Farm,
Pa., and we know of no reason why the experiment may not be
repeated in this commonwealth.
The Secretary is willing to suggest that there are in Pennsylvania
several country clubs where such a conference would be welcomed.
He is willing to mention The Reformatory at Huntingdon with its
splendid farm, the beautiful campus of the School at Morganza, the
Workhouse at Holmesburg and Hoboken, and the magnificent State
Prison Farm at Bellefonte. Perhaps Warden John Francies would
prefer to receive us when his institution is nearer completion.
When the invitation comes from any of the institutions mentioned,
there will be a response.
A. H. V.
Utilization of Prison Labor.
BY H. H. HART
(Suggestions to West Virginia)
With the tremendous demand for war supplies in addition to the

ordinary demand for domestic uses; with the cessation of
immigration which for 150 years has poured a steady stream of fresh
laborers into the United States; and with the immediate diversion of a
million men, and perhaps four or five millions, to the trade of war, we
are confronted with a scarcity of labor which compels us to utilize
every available worker to the limit of his reasonable capacity.
Able-bodied men, working under skilled direction and thorough
system, without loss of time from drink, strikes or voluntary holidays,
ought to earn more than their board and clothes. In the present state
of the labor market it is possible for prisoners, under proper
circumstances, to earn two dollars per day. In Vermont prisoners
from the common jail are earning two dollars per day on the adjacent
farms. In Windham county, Connecticut, prisoners are actually
earning $2.50 per day at common labor, and at Wilmington,
Delaware, short-term prisoners are earning $1.20 per day....
In those counties where the jail prisoners are not employed, we
would suggest the adoption of the Vermont plan under which the
jailer finds employment for individual prisoners with a nearby farmer
who pays for their labor at the ordinary rate for free labor. The
prisoner sleeps at the jail and has his breakfast and supper there—
two good hearty meals. He goes out in the morning, carrying a
dinner bucket, and returns at night. If the distance is too great the
farmer sends for him by team or automobile. If the prisoner fails to
make good or tries to run away, the farmer notifies the sheriff
promptly, who sends a deputy sheriff after him. The reports from
Vermont, Connecticut, Delaware and Wisconsin, where this plan
prevails, show very few escapes. No guard is necessary, as the
farmer looks after the prisoners, and thus the overhead charges are
done away with.
Thomas J. Tynan, of the Colorado State Penitentiary, wrote recently
as follows: “We are now doing work to the value of $2.50 per day by
prisoners on the highway at a cost to the taxpayers of the State not
to exceed 40 cents per day. The State could well afford to pay a little
wage besides the good time allowance.”
The difference between the old system and the new system lies
chiefly in the manner of dealing with the prisoners. Under the new
plan an appeal is made to the prisoner’s honor and good will. After
being tested within the walls, he is permitted to join a company of
workers outside the wall on his promise of good work and good
behavior, and on his promise not to run away. The prisoners work
without chains, and the guards in many cases carry no firearms.
Testimony comes from Ohio, from Oregon, from Colorado, from
Wisconsin, from Connecticut and from Canada that prisoners—even
low-grade prisoners and negroes—respond surprisingly to this
treatment; that escapes are few, and that there is a great
improvement in the industry and efficiency of the prisoners.
The incentive to the prisoners to make good is partly an increased
allowance for good time; partly, in some States, a small cash wage;
partly the desire of the prisoners for the freer life outside the walls,
and partly a response to the confidence shown in them by their
officers.
Success in employing prisoners on the roads by modern methods
depends first upon finding a road manager who is interested in his
men and believes in the possibility of exciting their interest and
loyalty for the work. It depends, second, upon creating such living
and working conditions as will keep the men fit for a good day’s
work. That means good food, good cooking, comfortable sleeping
quarters, opportunity for proper recreation, good laundry work and
bathing facilities, and, above all, the treatment of the prisoners by
their officers as reasonable human beings.
The plan of roadside camps, portable cages, chain gangs, ignorant
and cruel guards armed with shotguns and discretionary power will
not secure cheerful and efficient work.
THE INDIANA PENAL FARM.
[This article, by Henry A. Montgomery, Staff
Correspondent of the Detroit News, was incited by the
plan of Detroit to build a large and expensive House of
Correction. What other communities or States are doing
will be told by Mr. Montgomery.]
Greencastle, Ind., March 13.—“Reform them? No, you don’t
reform them. You can’t change a man’s real nature. But lots of them
are not naturally bad. We get the alcohol out of their systems, give
them all they can eat, make them keep regular hours and do a man’s
work, and the good in them has a chance to show itself.”
That is the way Charles E. Talkington, superintendent of the Indiana
State Farm, defined reformation.
The farm is situated in some of the most beautiful farming country in
western Indiana, about midway between Indianapolis and Terre
Haute. The site selected for the colony is particularly well adapted to
the needs of a penal institution, being rich in its deposits of
limestone, having plenty of tillable soil and considerable timber. The
beautiful hills and deep ravines lend the tract beauty and make it
possible for the landscape gardener with little effort to make it look
like the estate of a wealthy landowner. Although general farming is
engaged in, the products of the garden form so great a share of the
subsistence of the men that this has developed into one of the most
important features of the work. When present plans have been
realized there will be at least 160 acres devoted exclusively to
gardening.
It’s an unusual sight that one encounters on arriving at the top of the
long hill where is built the little village, the home of the 700 social
misfits. Two rows of long, squat frame buildings form the “street” of
this strange town. Nobody would guess from the appearance of the
buildings what they were intended for. They resemble the Billy
Sunday tabernacle type of structure. There is nothing attractive
about them, but they are temporary quarters and they are serving
their purpose well.
The first of these buildings is the office and sleeping apartments of
the officers. It is here that the prisoner is first taken. His history is
recorded, he is subjected to a medical examination, his clothes are
fumigated and stored away for his use when he is released, and he
is given a bath, shave and haircut, and a suit of clothes. The clothes
consist of the heavy working jacket and trousers, underclothing and
corduroy cap.
The first interview with the prisoner is considered important. The
assistant superintendent gives him detailed instructions as to his
own conduct and tells him with great care about the theories that are
being worked out. The number of men who, immediately following
this talk, are placed on their honor and given as much freedom as it
is possible to give is remarkable. Few of them violate the trust.
Often the man sent to the farm for six months walks forth from the
office, strolls over to the recreation room to await his work
assignment and never feels for one minute the influence of restraint,
except, perhaps, the realization that in the watch towers placed at
various points of vantage about the farm there is stationed a man—a
fellow prisoner—whose duty it is to notify headquarters if any
prisoner starts to leave the grounds.
The prisoner eats in a dining room and sleeps in a dormitory which
are kept spotlessly clean, and there is neither bar on the window nor
lock on the door. Each dormitory is occupied by about 200 men, and
one officer is all that is needed to maintain order.
“We do not say our plan is perfect,” said Superintendent Talkington,
“nor do we make any great claims about our ability to reform a man
during the short time he is here. But we do say this is the best
manner yet devised for handling them. We take a man from the
gutter, and at least make it possible for him to improve. We give him
health, and direction enough to get him into some employment at
which he can earn his living. Although we refuse to put forth any
claims about how much good we do for the man, we at least know
that we do not injure him. And that is more than can be said for any
jail or prison. We aren’t running any school for crime here. We do
know that. We also know that we can make this institution self-
supporting and a means of revenue for the State. What more can
you ask?
“The wide-open policy of freedom, I believe, has been carried to the
extreme here. Although the great majority of men can be handled
and trusted in absolute freedom, there are, in a population of 700
men, some who can never be given liberty. There is need for not
more than 50 cells. Any farm colony ought to have them even if the
cells are never used. Even so, we are getting along very nicely
without them, and it shows to what great extent this policy can be
carried successfully.
“We never had even punishment cells until a few days ago when four
were completed. We aren’t going to have to use them much, either.
Confinement on bread and water is the only form of punishment
permitted in this colony—no flogging, no dungeons, no ball and
chain, no stripes.
“We have prisoners living down on the lower end of the farm working
under a prisoner-foreman. We see them only when we are making
the weekly round of inspection.”
One could not help but feel, in discussion with Mr. Talkington, that
one was listening to a practical man who is anything but the dreamer
or idealist usually found advocating so revolutionary a plan as the
one on which the superintendent is working. He made no claims to
super-knowledge in the handling of men. He had no illusions about
the matter. He knew the faults of the plan and he knew the virtues.
When he undertook the present work, his only assets were his
experience as a farmer and school teacher.
“I feel,” said Mr. Talkington, “that your officials, before spending more
than a million dollars on the old type of prison, should see this farm
and the one at Guelph, Ont. I’m confident they would change their
plans. This may be a new thing in this country, but it is not untried in
the old. The most famous of the European farm colonies is the one
at Witzwil, Switzerland. It has solved all the problems of handling
men, it pays thousands of dollars annually into the treasury of the
canton Berne and there has been no trouble experienced in
competition with free labor.
“The farm colony has come to America to stay, and I hope Detroit
won’t take any action which will postpone for perhaps half a century
an improvement they are entitled to now.”
One of the bad features of Indiana’s temporary arrangement is the
lack of opportunity to segregate prisoners into classes or groups.
The dormitories are too large and the facilities for recreation are very
limited.
“The men should be divided into smaller groups,” said the
superintendent, “and I believe the recreation room should be a part
of the dormitory. I would not place more than 25 to 50 men in each
group. That would give a chance to segregate the youths from the
older men and permit keeping apart the more dangerous type from
the man who is here on some comparatively trifling charge.
“Another of our greatest needs is the establishment of industries to
supplement the work on the farm. We are going to get these. There
should be a furniture factory, a canning factory, brick yard or other
suitable industries where the men can be worked when weather
conditions are bad or outside work slack.”
The greatest factor in the maintaining of discipline is the use of the
honor system. There are good jobs on the farm and bad ones. And
the good jobs go to the men who have the best records and have
shown their ability to take positions of responsibility.
“You can’t tell me that you can run any prison with any such sort of
discipline,” a prison superintendent recently told me. “There are
some men who must be strung up and there are some who must be
spanked. If we didn’t resort to extreme methods at times we would
have a riot on our hands all the time.”
The best answer to this is found in the record of the Indiana farm.
There hasn’t been a strike or a serious riot since the institution was
founded. There are no guards standing or sitting around idle. The
guards are working foremen who perform as much actual labor as
any prisoner. The employed guards have guns in their pocket, but
the guns are never used and some of them aren’t even loaded.
There is a provision in the State law of Indiana which permits the
drafting from the penitentiaries of trusties to take jobs as foremen,
sentinels and lookouts. Of course, this probably could not be done in
Detroit, because the house of correction is a city institution. But in
Indiana it assists materially in keeping down the payroll. It makes this
difference—the farm colony at Occoquan, Va., has a payroll of about
$5000 a month; the Indiana institution gets along with $1700. And
the two institutions are very much alike.—From The Delinquent,
March, 1917.
THE CRIMINAL CODE OF
[A]
PENNSYLVANIA.
William E. Mikell, Member of State Commission to Revise
the Criminal Code.
Perhaps, in the true sense of the term, there is no criminal “code” in

Pennsylvania. The whole body of the criminal law has never been
reduced to a written code in this state in the sense in which this has
been done in some of the States of the Union in which jurisdictions
there are no crimes except those specifically prescribed. * * *
At the common law, crimes were classified as felonies and
misdemeanors. Without going into nice historical questions we can
fairly say that the term “felony” was applied to the more heinous
crimes, “misdemeanors” to the more venial ones. In the statutory law
of both England and of this country these terms have in general
been similarly employed. In the Pennsylvania code the legislature
has in the majority of cases in defining each crime designated the
crime a felony or a misdemeanor; and following the general principle
of the common law, affixed the stigma of “felony” to the graver
crimes. Viewing the code, however, as a whole, there is an utter lack
of principle in the grading of crimes as felonies or misdemeanors,
either according to the moral heinousness of the offense, or the
severity of the punishment.
Bigamy, with its attendant disgrace and illegitimacy, is a
misdemeanor, while embezzlement by a servant is a felony. For a
clerk or agent to embezzle—by the code called larceny—is a felony;
for a banker, trustee or guardian to embezzle, is only a
misdemeanor. * * *
Administering a narcotic with intent to commit larceny, is felony;
assault and battery endangering the life of an infant, is a
misdemeanor. Blackmailing is only a misdemeanor, while receiving
stolen goods is a felony. If one in the heat of a fight, intending to
disable or maim his antagonist, should cut him ever so slightly, he is
guilty of a felony, but, if he “on purpose, and of malice aforethought
by lying in wait, shall unlawfully cut out the tongue, put out an eye,
cut off the nose * * * or cut off any limb” of his victim, he commits
only a misdemeanor. Also if he “voluntarily, maliciously and of
purpose bite off the * * * limb or member of another,” he is guilty of a
misdemeanor. Truly, there must have been giants in those days. The
effect of these two sections is to make it a graver offense to attempt
mayhem and fail, than to succeed.
The Grading of Penalties.
The work of the commissioners who framed the Code of 1860 shows
an utter lack of any consistent theory not only of grading the crimes
as felonies and misdemeanors, but also in grading the punishment
fixed for the various crimes. It may not be easy to do this in all cases.
Persons may intelligently differ as to whether perjury should be more
seriously punished than assault and battery, and whether larceny or
bigamy be deserving of the greater penalty. But it is difficult to see
why embezzlement by a consignee or factor should be punished with
five years’ imprisonment and embezzlement by a person
transporting the goods to the factor should be punished by one
year’s imprisonment. * * *
Under the Act of 1860, having in possession tools for the
counterfeiting of copper coin is punished by six years’ imprisonment,
while by the next section the punishment for actually making
counterfeit copper coin is only three years, though it cannot be made
without the tools to make it. * * *
The distinction just mentioned is, however, no stranger than that
made by the code between a councilman on the one hand and a
judge on the other, in the provisions against bribery. Section 48 of
the Act of 1860 provides that if any judge * * * shall accept a bribe,
he shall be fined not more than $1000 and be imprisoned for not
more than five years. But by Section 8 of the Act of 1874, a
councilman who accepts a bribe may be fined $10,000, ten times as
much as a judge, and be imprisoned the same number of years—
five years. The statute also provides that the councilman shall be
incapable of holding any place of profit or trust in this
Commonwealth thereafter. But the convicted judge is placed under
no such disability.
Relations of Fine to Imprisonment.
In the case of almost every crime denounced by the code fine and
imprisonment are associated. In most cases the penalty provided is
fine and imprisonment, in some it is fine or imprisonment. In a few
cases imprisonment alone without a fine is prescribed, and in a few
others it is a fine alone without imprisonment. We seek in vain for
any principle on which the fine is omitted, where it is omitted; or for a
principle on which it is inflicted in addition to imprisonment in some
cases, and as an alternative to imprisonment in others. Thus the
penalty for exhibiting indecent pictures on a wall in a public place is a
fine of $300, but no imprisonment, while by the same act the drawing
of such pictures on the same wall carries a fine of $500 and one
year’s imprisonment. Manslaughter carries a fine of $1000 as well as
imprisonment for twelve years, but train robbery and murder in the
second degree involve no fine, but fifteen and twenty years in prison
respectively. It cannot be the length of the imprisonment that does
away with the fine in this latter case, for the crime of aiding in
kidnapping may be punished with twenty-five years in prison, but
also has a fine of $5000.
More striking still, perhaps, is the lack of any relation between the
amount of the fine and the length of the imprisonment provided in the
code. In the case of some crimes the fine is small and the
imprisonment short, as in blasphemy, which is punished by a fine of
$100 and three months in prison, extortion and embracery punished
with $500 and one year. In a few the fine is large and the
imprisonment long, as in accepting bribes by councilmen, $10,000
and five years, and malicious injury to railroads, $10,000 and ten
years. But in others the fine is small while the imprisonment is long
and in others the fine large and the imprisonment short.
Incomplete Crimes.
It is a general principle of criminal jurisprudence that “incomplete

crimes,” as they are called, such as attempt, and conspiracy to
commit a crime, should not be punished as severely as the full,
completed crime. It was on this principle that at common law an
attempt to commit even the gravest felony, such as murder, was only
a misdemeanor. Other codes maintain this principle. * * *
The Pennsylvania code has no general section on attempts, but in a
haphazard manner, in providing for some crimes, provides for the
attempt to commit the same, and in some cases has no provision for
such attempts. A study of those cases in which provision for
punishing the attempt is made, shows an entire absence of any
theory or principle in assessing the punishment. Thus the penalty for
the attempt to commit arson is the same as for the crime of arson
itself; for the attempt to commit robbery, the same as for the
completed robbery; but the attempt to commit murder is not
punished with the same penalty as murder, viz.: death, or twenty
years’ imprisonment, but by seven years’ imprisonment only.
Instances of Lack of Co-ordination in Drafting.
Two strikers separately determine to wreck a passenger train: one

removes a rail from the road over which a train is scheduled to pass;
another cuts the telegraph wire to prevent the train dispatcher from
stopping the train from running into a wreck. The first striker would
come within the terms of Section 7 of the Act of 1911 and could be
sentenced to pay a fine of $10,000 and suffer imprisonment for ten
years; the second man would come within the terms of Section 147
of the Act of 1860 and could not be fined more than $500 or
imprisoned more than twelve months. * * *
If the executor made way with a horse belonging to the estate, his
maximum imprisonment would be still two years; but if the butler
made way with another horse he might receive ten years as a
penalty. If a mule would serve the butler’s purpose as well as a horse
he had better take the mule, for then he could not be sentenced for
more than three years; if the mule were not swift enough, however,
he might choose an automobile, for the maximum imprisonment for
stealing an automobile is the same as that for larceny of the mule,
being less than one-third of that for larceny of a horse.
If the driver of a public “coachee” by “wanton and furious driving or
racing” unintentionally breaks a chicken’s leg he may be punished by
five years’ imprisonment, the same punishment provided for
attempted rape, for mayhem, for counterfeiting, and for robbery; but
if the driver of a taxicab is guilty of the same assault on a member of
the feathered tribe he is not even indictable. If the driver of this
“coachee,” while so driving, should accidentally inflict the slightest
personal injury on another, he would be liable to greater punishment
than if he deliberately stabbed that other with intent to maim him, or
wilfully and maliciously exploded dynamite under him, thus doing him
serious bodily harm. This violates one of the cardinal principles of
criminal jurisprudence, viz., that crimes of negligence are not so
grave as crimes done with deliberate intent, a principle recognized in
other parts of the code in providing for murder and involuntary
manslaughter. * * *
The writer has attempted to point out in this paper some of the more
glaring and interesting defects in the code. He has by no means
exhausted them. There is a great need for a complete revision of the
code. It is a jumble of inconsistent theories; a great many sections
are badly drawn, others are obsolete; many are inconsistent, many
are in conflict; there is much overlapping due to different acts having
been passed at different times covering in part the same subject
matter, so that it cannot be told whether a given crime should be
punished under one section or another prescribing a different
punishment.
Governor Brumbaugh has appointed the following on the

Commission to revise the Criminal Code of the Commonwealth:
Edwin M. Abbott, Chairman, Philadelphia; Wm. E. Mikell, Secretary,
Philadelphia; George C. Bradshaw, Pittsburgh; Clarence E.
Coughlin, Wilkes-Barre; Rex N. Mitchell, Punxsutawney.
FOOTNOTES:
[A]From an article by Mr. Mikell in the University of Pennsylvania
Law Review, January, 1917. The article clearly indicates the
urgent need of revision of our criminal code.
AMERICAN PRISON ASSOCIATION.
ATTENDANCE.
The meetings of the American Prison Association in 1917 were held

in New Orleans, November 19-23. Outside of the State of Louisiana,
the registration of delegates and attenders amounted to 261, of
whom seven were from Canada, one from Mexico, one from Cuba
and one from Guatemala. One hundred and thirty-seven registered
from Louisiana. Outside of this State, Massachusetts enrolled the
largest number, thirty-two being accredited to the Bay State. Then
followed New York, with twenty-nine, and Pennsylvania was third
with twenty-two, of whom nine were Official Delegates. It must not be
forgotten that there were many attenders at these meetings who had
not received appointment as Official Delegates, but who were active
and welcome participators in the discussions. In 1916 and 1917, the
Commonwealth of Pennsylvania has been well represented, but for
several years previous the attendance from the Keystone State had,
from a numerical point of view, been rather insignificant.
THE PRESIDENT’S ADDRESS.
Dr. David C. Peyton, Superintendent of the Indiana Reformatory at

Jeffersonville, ably officiated as presiding officer of the various
sessions of the Association. He is evidently a believer in strict
discipline administered by the officials who are legally appointed as
the guardians of the offender, yet no one who visits the institution
over which he presides, would assert that the regulations are harsh
or unreasonable. His view of the Honor System would not involve
government by the inmates.
“In prison management there have developed two colossal evils.
One was peculiar to the past and the other in a measure
characterizes the present. Most of the evils that are associated with
prison work have come from these two roots. They are antipodal as
east and west. They are cruelty and sentimentality. As the first was
the child of ignorance, the second springs from half knowledge and
is not the less reprehensible. True reform will come, not by a
softening and relaxation in prison discipline, not by imputing to
criminals qualities which their whole activities have proven them to
lack and the very absence of which is the cause of their
incarceration, not by making their pathway smoother and easier, nor
yet by touching it with the magic of romance. If a little of the leaven
of common sense were allowed to permeate the situation it seems to
me that the clouds in our pathway would lift somewhat.
“Of course cruelty, the other bête-noir, is only named to be
condemned, and thanks to our even half-knowledge it has no place
in modern prisons except in isolated spots. But I doubt if ever cruelty
was any more cruel than a regime which threatens to become
popular today. It seems to me that prisons should be run for the
purpose of training men for sane living. If that is true, then they
should in fact train these men for sane living.
“A modern prison should be a beehive of industrial activity and
should be more than self-supporting. Indeed, men should be able to
serve their sentences and earn enough overtime money during their
terms to support their dependents—in part at least. A trade should
be taught when practicable, but even more important than a trade is
the idea of inculcating industrious habits. It is not a misfortune for
men to have to labor, but it is a blessing both for them and for us.
“Discipline should be strict but not arbitrary. The rules should be
based on experience and should be obviously sound.
“Punishment has a place in prison, but it should be logical; should,
as far as possible, flow as a natural consequence from the
transgression according to the pedagogical rule of Spencer.
“The industrial training should be correlated with the didactic
instruction and the prison library. The three should form the tripartite
educative force of the institution.”
THE HONOR SYSTEM.

As was naturally to be expected, the so-called Honor System
received a large share of attention, especially in the Wardens’
meetings. The Wardens generally are opposed to that feature of the
Honor System which involves placing the discipline to any great
extent in the hands of the convicts. The experiments of Mr. Osborne
at Auburn, Sing Sing and Portsmouth are regarded as sporadic
efforts largely affected by the personality of a masterful, though
sentimental, empiricist. Men who have never governed themselves
should not be elected to govern one another. Mr. Erskine, of
Connecticut, argued that it was wrong to base any system on
emotional appeal. “Twenty per cent. of the men in prison are entirely
bad and vicious; 20 per cent. would wield a good influence if they
had the opportunity, and the remaining 60 per cent. could be swayed
by either the good or the bad element.”
On the other hand it was stoutly contended that the prison was the
proper place for men to learn to govern themselves. Necessarily they
were confined to a limited area, and still subject to watchful care by
big-hearted, efficient advisers. Let them learn that discipline in life is
an essential feature of any community. Let them learn this lesson by
personal experimentation. Thus they may recover some sense of
self-respect. They will rejoin the outer world with some measure of
responsibility. They will return to freedom with a different
understanding of life.
Dr. Bernard Glueck, director of the psychological clinic at Sing Sing,
asserted that in general prison officials, through limitations to their
work, were not fully qualified to pass judgment on the subject, and
had not availed themselves of opportunities to carry out a
comprehensive system of self-government. He stated that such a
system had proven to be distinctly successful at Preston, California,
and asked Mr. Calvin Derrick, the founder of the School of Industry
at Preston, to present some account of this institution.
Mr. Derrick informed the Association that this California school has
developed its honor system to the point where 250 of the boys were
sent to the Sacramento fair alone, traveling through the country 250
miles, and returning without the loss of a single one. The boys have
a complete republic system of government, and conduct all of the
work of the school. A football team plays regular school and college
teams of the State, traveling without supervision.
It appeared to some of us who listened with intense interest to these
discussions that the difference in sentiment was rather one of degree
than of principle. Wherever any privileges are allowed and wherever
the trusty system is permitted, there is involved some measure of
self-government. One warden permits the prisoners to mingle
together on the base ball field. These men are on their honor.
Another warden might say that he would allow his prisoners to play
and observe games and leave the regulation of their conduct while
on the field to the men themselves. They are still on their honor and
doubtless are aware that their regulation of conduct must meet the
approval of the warden. Suppose we allow the men to impose
penalties for minor delinquencies. The warden still exercises his
judgment on the punishment awarded. The warden governs. The
men may have more or less privileges, but they are granted by the
warden. It resolves itself into a question as to how far such privileges
may be granted. And no two wardens in the world will agree
precisely on this point.
For the last twenty years the trend has been getting away from the
brutality of the former systems, from the petty rules, from degrading
and humiliating treatment, and the avowed object of confinement has
been reiterated again and again to aim at reformation. We admit that
some wardens are more successful than others in accomplishing
desired results, and yet we must not expect them to adopt an entirely
uniform program. We must make due allowance for the personal
equation involved, for the individuality of the ruling authority. The
warden who claims that his system is perfect, and that he has
nothing more to learn, is recommended for removal.
CLINICAL WORK.
Dr. Bernard Glueck exhibited a number of charts showing much that

was deeply interesting in regard to the mentality, environment and
parentage of the criminals sent to Sing Sing.
“Sing Sing is being reorganized and rebuilt to receive all of the
criminals of New York for examination immediately after they have
been convicted. We are trying to get at the man behind the crime
rather than the crime itself. Two facts are very evident from our work.
The social fact is that 66 per cent. of the prisoners we have received
are previous offenders. The medical fact is that 59 per cent. of the
prisoners can be classified for mental diseases or mental deviation;
and thousands of this class can be treated and cured by means of
vocational training and other modern prison methods.
“Sing Sing is being remodeled so that we can devote as much as
four months to an intensive study of each prisoner who enters the
institution. From this examination we can learn which men should be
sent to the insane hospitals, which to the intensive vocational
schools, and can outline the most effective method of treatment
necessary to prepare the men for the future.
“The indeterminate sentence is essential to the proper working out of
our plans, and the criminal courts of the State are working in
harmony with this idea. Most offenders can be restored to a normal
life and good citizenship after they have finished a term under proper
treatment, and criminal judges in New York city tell us that fewer men
come before them for a second time since we have adopted the
present methods.
“It is the aim of the prison to turn the men into citizens with an
understanding. They are allowed many liberties, and are made to
take an active part in community life in prison. They have their own
social organization, a system of self-government, including even a
charitable society. In their charity work they aid prisoners who are
leaving the prison, make it possible for poor people to visit
imprisoned relatives, send the bodies of prisoners home for burial,
and many other things of that nature.”
PRISONERS AND THE WAR.
Some problems arising as a direct result of the war received serious

attention. There was the question of additional food production by
prison labor; whether paroled men shall enter the army or navy; and
whether conditional pardons may be granted, contingent upon
military service.
Some delegates asserted that a general restlessness was noticed
among most prisoners. Many of them are exceedingly anxious to get
into the war, and in some States prisoners are being paroled so that
they may enter the army or navy. Prison officials know that many of
their wards are fit for military service just as well as they are aware
that other prisoners are unfit. After prolonged discussion of the
subject, the Wardens’ Association unanimously adopted the
following resolution:
“Resolved, That the Wardens’ Association of the American Prison
Association suggest to and request of the President of the United
States the modification of paragraph 849 of the Regulations of the
Army and paragraph 3686 of the Articles for the Government of the
Navy of the United States so as to permit the enlistment in the
military and naval forces of the United States of men who, in the
judgment of the proper military and naval authorities, are physically,
mentally and morally qualified, despite the fact that such persons
may have been convicted of the offenses set forth in the regulations
and articles above referred to and imprisoned therefor, upon their
being duly and honorably paroled or discharged from such
imprisonment.”
At Guelph, Ontario, the great prison has been practically
depopulated. The prisoners have gone to war, and the institution has
been taken over as a hospital for convalescents returned from the
scenes of war. The Superintendent, Dr. J. T. Gilmour, declared that it
is only a step from prisoner to patriot.
“We have learned a great deal about prisoners during the three
years we have been in war. We have learned that the prisoner’s
sense of patriotism is not dead because he is behind the bars; that
he is just as anxious to serve his country as the man who is not
being punished, and if given an opportunity the chances are that he
will make a good soldier. It has come to my notice that men have
exchanged prison uniforms for army uniforms in three hours after
their discharge from prison.”
He made the further statement that thousands of men had been
released from Canadian prisons to permit them to serve in the army,
and thousands of others were “doing their bit” by making hospital
supplies during their imprisonment.
PRISONERS CONDEMNED TO SLAVERY.
In at least two Southern States, the infamous lease system, whereby

prisoners are leased for an annual stipend to work in the mines or in
the turpentine forests or in other work, prevails. Isadore Shapiro, a
member of the Legislature from Alabama, and President of the
Committee on Prisons, vigorously lambasted the government of
Alabama for tolerating and continuing such venal disgrace. The
Alabama legislature had made an effort to abolish the lease system
but the governor had interposed so as to prolong the infamy. The
prisoners could profitably and healthfully be put to work on the State
farms but instead they are offered for sale to the highest bidder, and
employed in mills, coal mines, lumber and turpentine camps. All of
the women prisoners in one county were leased recently for the term
of two years at the rate of fifteen cents a day. Mr. Shapiro produced
a leather strap six feet long and an inch and a half wide with which
prisoners are flogged.
Recently in the State of Florida 598 prisoners were leased at an
average of $360 per head by the year. It is a fact that most prisoners
who work in the turpentine industry are so broken down in health
after a few years that for the remainder of their days they are unfit for
any manual employment. Of course it is granted that this work must
be done, but we insist that it must be done under humane
regulations. We have yet to learn of any leasing corporation or
individual that has treated his serfs with merciful consideration.
Georgia, after a long fight, has entirely repudiated the system.
The Association, while insisting that employment should be given to
prisoners, unanimously adopted a resolution condemning in the
strongest terms a system whereby men and women are sold into
bondage in order to enhance the revenue of the State.
THE INDETERMINATE SYSTEM.
There is no longer any debate about the Indeterminate Sentence.

The principle is written upon the statutes of nearly every State of the
Union, tho in a debilitated and illogical form in the Commonwealth of
Pennsylvania. Amos W. Butler, Secretary Board of Charities in
Indiana, in an address delivered in one of the churches declared that
we had brought very little understanding to bear upon our treatment
of criminals until recent years. He compared our knowledge of
smallpox, yellow fever and other diseases with crime and concluded
that we made as many blunders in considering the offender as we
formerly made in our attitude toward these mysterious and dreaded
diseases.
“Prisons are the visible signs of our failures. It is now within the
power of man to abolish many diseases from the earth, and so with
crime. Criminals are not sent to prison for punishment, as many
seem to believe, but prisons exist for the confinement of prisoners
for the safety of society and for the reform of the man or woman
there. They should, if possible, be reformed and returned to society.”
The speaker favored the indeterminate sentence. He said you would
not send a diphtheria patient to the hospital for a definite time, say
two or three weeks. Complications might appear and more time may
be required to effect a cure. The same is true of criminals. They
should be sent there until reformed, until fit to be returned.
Indiana has an indeterminate sentence law and in the past twenty
years 11,000 men and women have been released under that law.
Seventy-five per cent. of them succeeded, or made good. Prisoners
there have earned about $3,000,000 for themselves.
The absolutely indeterminate sentence is not yet in vogue in any
State. A criminal under such a sentence would be sent to prison as
to a hospital to remain till cured of his malady. Perhaps, in some
cases he would be subject to some detention as a deterrent to
others contemplating entering upon a criminal career. Specialists
would determine when he was ready to enter the community. We
may at some time adopt such a system when there are enough men
and women having the skill and training necessary to pass on the
mental and moral characteristics of such patients.
Perhaps the best form of sentence is by statutes which fix the time
for any given offense. The time for arson, for instance, may be
placed from two to twenty years. It is the function of the judge or jury
to determine whether the accused is guilty. If found guilty, the
defendant is placed in care of a judicious board of control who will
release the criminal at such time as they may deem best for him and
the community.
Judge Willis, of St. Paul, said that no physician would send a patient
to a hospital for a specified number of days or weeks, yet this very
thing is done daily by judges who send mentally and morally sick
men and women to jail. “A doctor would not presume to predict just
how many days it would take to cure a disease, but a judge daily
uses his prerogative as a diagnostician in sending morally diseased
people to jail, although the records of trials show that no two judges
think alike in the matter. * * * Society no longer tolerates vengeance
in the criminal code. The desire of intelligent people of today is to
restore the criminal to a place in society—an honorable place—and
not only to restore him, but to make him a more valuable member of
society than he was before his incarceration.”
NATIONAL PRISONERS’ AID ASSOCIATION.
This body is an auxiliary of the main organization. Two sessions

were held at which various phases of the work were earnestly
discussed.
The address of the President, Rev. James Parsons, of Minneapolis,
presented a summary of what has been accomplished, and also
some cogent reasons for the existence and maintenance of
organizations having constantly in view the rehabilitation of those
who have violated law. This address is given in another part of the
Journal.
Rev. Charles Parsons, of Des Moines, called attention to the
increase of crime in time of war. “A celebrated doctor declared that

Practical Implementation of A Data Lake Translating Customer Expectations Into Tangible Technical Goals 1St Edition Nayanjyoti Paul All Chapter

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Practical Implementation of A Data Lake Translating Customer Expectations Into Tangible Technical Goals 1St Edition Nayanjyoti Paul All Chapter

Загружено:

Авторское право:

Доступные форматы

Practical Implementation of a Data

Lake: Translating Customer

ISBN-13 (pbk): 978-1-4842-9734-6 ISBN-13 (electronic): 978-1-4842-9735-3

Copyright © 2023 by Nayanjyoti Paul

About the Technical Reviewer�������������������������������������������������������������ix

Chapter 1: Understanding “the Ask”����������������������������������������������������1

Chapter 2: Enabling the Security Model���������������������������������������������19

Authentication and Authorization (SAML vs. PING, etc.)��������������������������������31

Chapter 3: Enabling the Organizational Structure������������������������������53

Chapter 4: The Data Lake Setup���������������������������������������������������������63

Chapter 5: Production Playground���������������������������������������������������151

Chapter 6: Production Operationalization����������������������������������������159

AWS Accounts Used for Delivery�����������������������������������������������������������������176

After completing this book, you will understand how to implement a

What You Will Learn

• Understand the challenges associated with

• Explore the architectural patterns and processes used

• Design and implement data lake capabilities

• Associate business requirements with technical

Who This Book Is For

–– We needed to know the owner of this platform. This

–– We wanted to know the team that was handling all the

this customer and what considerations we needed to

–– Next we set up a call with the “cloud engineering” team.

–– Next was the DBA team. The DBA team currently

–– Next was the data governance team. One of the key

“governance-­first” approach, and our customer

–– We also connected with the “business” users who were

Understanding the Requirements

• Establishing a clear understanding of the customer

• It can be difficult to determine exactly what data

• It is difficult to understand the customer’s desired

• Connecting the data lake with other systems can be a

• It can be difficult to determine the best way to

• It is difficult to ensure that the data lake is designed for

• Determining the most effective way to ingest data into

• It can be difficult to ensure that the data is of high

• Since the customer requirements will vary from one

• Understanding the customer’s security and privacy

• Establishing the necessary data governance

• Establishing the necessary data quality requirements

The following diagram represents how “success” means different

1.0 2.0 3.0 4.0

Data Scientists were key Data Engineering team were

7.0 6.0 5.0

© Nayanjyoti Paul 2023 1

• What are the current challenges?

• Why is modernizing data platforms hard?

• What are the top five issues that we want to solve?

• What is available on-premise and on the cloud already?

• What meetings will be needed throughout the project?

• What common terms and jargon can we define?

 ecide on the Migration Path, Modernization

supposed to be sponsored by the business, and they had strict timelines to

Table 1-1. Assessment Questions

Table 1-1. (continued)

Questions Why Was the Question Important? What Was Decided?

Assess the Current Challenges

–– The current setup could not scale up based on the

–– As the data volume was growing, the current trend of analytics

–– As the organization was gearing up for the future, they had

–– Time to market was essential, and a process that can provide

–– They wanted to be future ready. Peer pressure is a huge

About the Technical Reviewer��ix

Chapter 1: Understanding “the Ask”��1

Chapter 2: Enabling the Security Model��19

Authentication and Authorization (SAML vs. PING, etc.)��31

Chapter 3: Enabling the Organizational Structure��53

Chapter 4: The Data Lake Setup��63

Chapter 5: Production Playground��151

Chapter 6: Production Operationalization��159

AWS Accounts Used for Delivery��176

What You Will Learn

Who This Book Is For

“governance-first” approach, and our customer

Understanding the Requirements

ecide on the Migration Path, Modernization

Assess the Current Challenges

nderstand Why Modernizing Data Platforms

Determine the Top Five Issues to Solve

etermine What Is Available On-Premise vs. on

reate the Meetings Needed Throughout

Business − Identify and prioritize source systems that need to be onboarded

Data − Design and document architecture for building a cloud-native

Define Common Terms and Jargon

PII Columns: RBAC, ABAC Features