2012 Third International Conference on Services in Emerging Markets

Hadoop Compatible Framework For Discovering Network Topology And

Detecting Hardware Failures
Anusha Ventrapragada
ventrapragada.anusha@iiitb.org

Sheeba Samuel
sheeba.samuel@iiitb.org

Varsha Raja Vidya

raja.vidyavarsha@iiitb.org

Prabha Satya Manepalli V

satya.manepalli@iiitb.org

Shwetha Muralidharan
shwetha.m@iiitb.org

Shrisha Rao
shrao@ieee.org

International Institute of Information Technology - Bangalore

each of the DataNodes in the cluster. This collected information is serialized and stored in the Hadoop Distributed
File System (HDFS) in the form of log files.
The Apache Pig tool, which is a data analytics tool for
the Hadoop framework, is used to parse and analyze the
hardware-related information [3]. The log files for each
of the DataNodes which are stored in HDFS are given
as inputs to the Pig Latin [4] script. These Pig Latin
scripts are used to get the most recent values for each
hardware component considered in each of the DataNodes,
and the output so obtained is again stored in the HDFS.
Now the NameNode takes these most recent values, parses
them, and compares them with the threshold values of
the corresponding hardware component. In case for any
DataNode if it is found that any hardware component
value has exceeded the threshold value, a message is
sent to the corresponding DataNode about the suspected
failure. This signalling system to the DataNode works with
the assumption that there is a robust network connection
between the NameNode and the DataNodes.
In the context of high-volume data processing, the
limiting factor is the rate at which we can transfer data
between the nodes [5] or the bandwidth which is also
dependent on the physical topology of the network. In
order to configure Hadoop clusters to meet varying requirements in terms of size, processing power, reliability,
availability [6], etc., it is important to keep track of the
current topology of the Hadoop network and track the
status of the network services in Hadoop.
For the network topology discovery module, we get
the IP addresses of all the live nodes in the cluster by
observing what are all the DataNodes the executor process
is able to reach and storing the addresses of those nodes
in a file. This file containing the addresses of the live
nodes is given as input to the network monitoring tools,
OpenNMS and Zenmap. Zenmap provides a topology
map of the network [7]. OpenNMS monitors network
services like ICMP, SMTP, SNMP, SSH, etc., of the
Hadoop cluster [8]. It can also detect outages among the
DataNodes of Hadoop.
The detection of hardware failures module can be used
for large number of nodes in a Hadoop cluster. Since the
DataNode is signalled about the suspected failures in the
hardware components, the DataNode can take appropri-

AbstractWe have implemented a Hadoop compatible

framework that helps in detection of suspected hardware
failures in DataNodes within a Hadoop cluster and in
signalling the nodes accordingly. Based on the status of the
various hardware components of the DataNodes, the master
node signals the DataNode so that appropriate actions can
be taken. In a Hadoop cluster, the knowledge of the network
is important for the master node to configure itself to meet
varying needs of the DataNodes. It is important to keep track
of the current topology of the Hadoop network and track
the status of the network services in Hadoop. We have also
added the functionality to discover the network topology in
a Hadoop cluster. This discovered topology information will
be useful in performing load balancing in the network, or
in making intelligent decisions in data replication in case of
node failures.
Keywords-Hadoop, NameNode, DataNode, Network Topology

I. I NTRODUCTION
Apache Hadoop is an open source framework [1] used
to develop scalable distributed software systems. It can
scale up to large number of nodes, each offering local
computation. Hadoop performs distributed processing of
large data sets operating on clusters of computers consisting of commodity hardware.
We propose an extension to the Hadoop framework
which signals the nodes in the Hadoop cluster about suspected hardware failures. This becomes extremely useful
in large clusters of nodes where failures are quite frequent
due to the fact that Hadoop uses inexpensive commodity
hardware. Hardware failure detection is made possible by
collecting the status of the hardware components from
each of the DataNodes in the network. This information is
analyzed and compared with existing threshold values. In
case the values are exceeded, the corresponding DataNode
is informed by the NameNode or the master node. We also
discover the network topology by generating the network
topology map of the Hadoop cluster and monitoring the
network services. Thus the robustness of the network in
the Hadoop cluster can also be analyzed through our
extension.
A hardware failure monitoring package, FailMon [2]
which is a built-in functionality within Hadoop, is used to
collect various hardware status related information from

978-0-7695-4937-8/12 $26.00 2012 IEEE

DOI 10.1109/ICSEM.2012.16

58

ate actions to avoid the failures to occur. The network

topology map of Hadoop cluster shows all the available
live nodes in the cluster and helps the Namenode to take
correct decisions regarding data replication in the case of
a node failure.
For any high-performance computing cluster of nodes,
failure prediction can be done by the application of data
mining techniques like decision tree classifiers [9], as well
as machine learning methodologies like support vector
machines to classify the collected system log information
[10]. There is also an architecture proposed which allows
failure detection by mutual monitoring of the liveness of
the cluster nodes and recovery of a failed node using
remote access of the system resources of the failed node
[11]. There exist a lot of networking tools for topology
detection even outside the context of Hadoop. Our work
helps bridge the gap between these networking tools and
Hadoop. Also our approach detects failure of a node
and intimates the corresponding node so that appropriate
actions may be taken accordingly.

Figure 1. System Architecture Of The Hadoop Compatible Framework

uniform record format. Each record in the uniform format

is serialized and appended to a file on the local disk. Hence
this file on the DataNode is uploaded into the HDFS.
Finally, we run a Pig Latin script to analyze the data
uploaded onto the HDFS.
The Pig tool takes the log files of all the nodes in
the Hadoop Cluster uploaded in the HDFS as input, and
for each DataNode, a Pig script filters the data according
to the type of hardware component. Then the Pig script
groups the data according to timestamp and stores the
most recent timestamped hardware information back to
the HDFS. This is performed for NIC, CPU information,
hard disks and sensors, and the results are placed in the
corresponding slots for each DataNode in the HDFS.
From the HDFS, the NameNode retrieves the output
of the Pig script which consists of four files for each
DataNode. The four files correspond to information regarding NIC, CPU, hard disks, and sensors. The NameNode compares each value in these files with the threshold
value for failure. In case any value of the hardware
components exceeds the threshold, it displays a message
in the User Interface and also signals the DataNode about
the suspected failure so that it can take appropriate action.
In Network topology discovery module, proposed architecture uses the scheduler nodes hosts list and does
SSH to every node in the hosts list. The nodes that are
network-reachable are added to the live nodes list and fed
to the network topology discovery tools, OpenNMS and
Zenmap.
OpenNMS and Zenmap were selected from the evaluated tools since they were scalable and provided detailed
information about the network status.
Zenmap provides a dynamic topology map of the network, and OpenNMS monitors the network and displays
status information of services like SSH, SNMP, etc.,
running on each of the alive DataNodes. When FailMon
is executed, all the IP addresses of active nodes in the
Hadoop cluster are redirected into a file and this file is
given as an input to OpenNMS and Zenmap.
Both the OpenNMS and Zenmap tools also show the
status of a node, i.e., whether the node is up or down.

II. S YSTEM D ESIGN

The Hadoop framework has two classes of nodes, NameNode and DataNode. The NameNode and the DataNodes of the Hadoop framework follow the master/slave
architecture [12]. For the failure-monitoring functionality
in our system, said functionality is decoupled from the
applications running over Hadoop by maintaining one
specific node as an ad hoc scheduler node. After setting up
the Hadoop multi-node cluster [13], NameNode is made
as the scheduler node.
Figure 1 depicts the system architecture of a Hadoopcompatible framework for hardware failure detection and
network topology discovery. For the purpose of collecting hardware component information and uploading into
HDFS, the FailMon package [14] of the Hadoop framework, configured according to our requirements, is used.
The FailMon package schedules an executor thread
on all the DataNodes at specified time intervals. This
task makes a DataNode perform data collection followed
by updation of the local data collected into the HDFS.
We maintain a local configuration file locally on each
DataNode, which specifies what local data are to be
collected for that node and how to collect it.
The executor process on the DataNode reads the local
configuration file and executes the job of collecting the
local data by parsing the system log files, Hadoop log files
and operating system utilities and updates the local data
collected on to HDFS [2]. Local data are collected from
the log files by the SystemLog parser, the HadoopLog
parser, and parsers for various operating system utilities
implemented in FailMon. These parsers run on each of
the DataNodes.
Various operating system utilities like ifconfig,
lm-sensors, smartmon tools and cpuinfo are required in the DataNodes to enable the executor process
to populate the log files. To analyze all the data in the
same way, the local data collected are converted into a

III. I MPLEMENTATION
A seperate Hadoop user account is used to start Hadoop.
The user hduser and the group hadoop are added to
each local system. SSH access is required by Hadoop to
manage its nodes. For a single-node setup of Hadoop,
it is required to configure SSH access to localhost for
59

the hduser user created previously [15]. A SSH key is

generated on each machine. Hadoop is installed and the
permissions to access all the files in Hadoop are changed
to the hduser user and hadoop group.
The Hadoop filesystem HDFS is formatted by the
NameNode. The Hadoop filesystem is implemented on
top of the local filesystem of the cluster [15]. This is to
be done initially when we set up a Hadoop cluster. After
formatting, the single-node cluster starts the NameNode,
DataNode, JobTracker and a TaskTracker on the machine.

C. Failure Monitoring
A Hadoop cluster is started and a failure monitoring
script is run on the NameNode. The script schedules logs
collection tasks on all the DataNodes in the cluster. A
configuration file is maintained in each of the DataNodes.
These files are given as an information source to the
executor process about what local data it has to collect
and from where it has to collect. It also specifies the
upload path of HDFS. The information is collected from
the system log files, Hadoop log files, and various OS
hardware diagnostic utilities.
Log entries about checksum errors imply either a hard
disk failure or a memory failure. Various operating system
utilities give information about the status of the hardware
components, and ifconfig provides information about
the identity and metrics of various network interfaces
present in the system. It also provides information about
the status of the network interface. By using txerrors
and rxerrors parameters, we can obtain the transmit
and receive error rates of packets respectively.
The file /proc/cpuinfo lists all the processors
and cores present in the system, along with information
about their status. smartmontools and lm-sensors
are two other utilities supported by the Linux operating
system which we made use of in order to collect status
information of the hardware components.
1) Smartmontools: SMART (Self-Monitoring, Analysis
and Reporting Technology) is a system to monitor the
health of computer hard disk drives . It detects and
reports conditions that indicate that there could be some
failures [16]. Depending upon the statistics provided by
SMART, the user is advised to replace the hard drive in
order to avoid loss of data. It is also used by hard disk
manufacturers to discover various failures on their devices.
A variety of different metrics are reported across different types of disks. These metrics are collected by the
FailMon package to monitor the hard disk status of the
DataNodes of the Hadoop cluster.
Table I gives some sample vendor-specific values for
the hard disk parameter values.

A. Setup Of Multi Node Cluster

Here we describe how to set up a multi-node Hadoop
cluster using the Hadoop Distributed File System (HDFS)
on Linux systems. Single-node clusters are installed in
each Linux system and these clusters are merged into one
multi-node cluster. In this cluster, one Linux system is
the designated master and the other systems are its slaves
[13]. Figure 2 depicts a setup for multi-node Hadoop
cluster.
Prerequisites
Configuring single-node clusters on every machine
first.
Network connectivity.
Also the /etc/hosts file has to be updated on all
machines for including the IP addresses of master and all
the slaves.

Figure 2.

Hadoop Setup For A Multi-Node Cluster

Table I
C RITICAL VALUES F OR H ARD D ISK PARAMETERS U SING
S MARTMONTOOLS

B. SSH access
The user hduser on the master (also known as
hduser@master) should connect:
to its own user account on the masteri.e., ssh
master; and
to the hduser user account on the slave (also known
as hduser@slave) via a passwordless SSH login
[13].
The hduser@masters public SSH key (which
should be in $HOME/.ssh/id_rsa.pub) has to
be added to the authorized keys file of hduser@slave,
i.e., users $HOME/.ssh/authorize_keys. This
is done manually or by using hduser@master:$
ssh-copy-id -i $HOME/.ssh/id_rsa.pub
hduser@slave SSH command [13].

Attribute
Seek Error Rate
Raw Read Error Rate
Reallocated Sector Count
Temperature Celsius

Hitachi
67
62
5
60

Western Digital
0
51
140
65

Toshiba
50
50
50
51

2) lm-sensors: lm-sensors is a hardware health

monitoring package for Linux systems [17]. Information
from temperature, voltage and fan speed sensors can be
accessed with the help of this package. Available sensors vary across different devices. This package contains
programs to help set up and read data. It determines the
temperatures of the cores present on the system.
60

The values are reported and the output format depends

heavily on the underlying hardware. Local data is periodically collected on each and every node by parsing the
system log files and the Hadoop log files, as well as by
parsing the outputs of various operating system utilities.
Parsing of different log files are done at different time
intervals. Each piece of information gathered locally is
converted into a uniform record format so that processing
of gathered data can be done in a uniform way. Each
uniform record is serialised and appended to a temporary
file on the local disk. All the local data are then uploaded
to a stable storage HDFS.

to show the availability of network services among

the nodes in the cluster to be determined.
OpenNMS and Zenmap were selected from the evaluated tools.
1) OpenNMS: In the maps-adapter configuration
file, the input can be specified in three ways:
Giving specific IP addresses.
Giving a range of IP address.
Giving a list of IP addresses through URL.
As we obtain a list of IP addresses, we choose the third
option.
2) Zenmap: As previously noted, the scheduler thread
of the failure monitoring module detects the live nodes in
the Hadoop cluster and inputs their IP addresses to a file.
This file is given as input to the Zenmap and OpenNMS
tools. Zenmap gives the topology map of the live nodes
in the Hadoop cluster.
Figure 3 shows a sample network topology map for
Hadoop cluster of five nodes in Zenmap.

D. Data Analysis Using Pig Tool

The Pig tool takes the log files of all the nodes in
the Hadoop cluster uploaded in the HDFS as input, and
for each DataNode, a Pig script filters the data according
to the type of hardware component. Then the Pig script
groups the data according to timestamp and then stores the
most recent timestamped hardware information back to the
HDFS. This is performed for NIC, CPU information, hard
disks and sensors, and placed in the corresponding slots
for each DataNode in the HDFS.
The HDFS can be accessed from any of the nodes using
the URL hdfs://master:54310/. For each of the
DataNodes, space is allocated in the HDFS to store the
outputs of the FailMon and Pig tool in the HDFS. The
merged log file is the output of FailMon which is stored
in the HDFS. This output merged file of the FailMon
serves as input to the Pig tool.
E. Messaging
The NameNode takes the Pig output and compares each
value with the threshold. In case of any value nearing the
threshold, it displays a message on the User Interface and
also sends this message to the failed DataNode to take
appropriate actions. The messaging to the failed DataNode
is successful provided there is a proper network connection
between the NameNode and the DataNode.

Figure 3.
Zenmap

Network Topology Map for Hadoop cluster of five nodes in

IV. P ERFORMANCE M ETRICS

Various experiments were conducted in the Hadoop
framework in order to measure parameters like response
time of the system and core temperature of the NameNode
and the DataNodes. The time taken to parse the log files is
an important indicator of the time taken for data collection.
The execution time of Pig in the cluster is an estimate
of the time the system takes for data analysis. Also the
execution time of the total system was measured separately
in order to compare the time taken for the individual
modules with the total execution time. The temperature
rise in the NameNode and DataNode while running the
entire system are captured and also compared.
Our implementation is based on the assumption that
there exists a robust network connectivity among the nodes
of the Hadoop clusters. In case of extra network traffic,
the time taken to upload and analyze the logs collected
increases.
As we are using Hadoops own failure monitoring
package for log collection and uploading of these logs
into HDFS, our implementation does not interrupt the
applications running over Hadoop. While analyzing the
data collected using Pig scripts, only a minimum overhead

F. Discovering Network Topology

When FailMon is executed in the scheduler node, it tries
to connect to all the nodes listed in the hosts file using
SSH to schedule the log collection task. For each node
to which it is able to connect, we added a functionality
to FailMon to write the IP address of that node to a file.
This file is fed as input to the network monitoring tools.
The tool for discovering the network topology was chosen
such that the basic criteria mentioned below are satisfied:
Scalability
The network topology discovering tool must be scalable for a large number of nodes in the Hadoop
cluster.
Map
The network topology discovering tool must be able
to display the network topology map of Hadoop
cluster.
Detailed Network Status
The network topology discovering tool must be able
61

Table III
P REDICTED T IME TAKEN TO PARSE L OG F ILES U SING FAILMON
PACKAGE O N L ARGE N UMBERS OF N ODES

is experienced by the applications. This minimum overhead exists because Pig scripts are internally scheduled as
MapReduce jobs, similar to Hadoop applications.

Number of Nodes

Time taken in seconds

0
50
100
500
1000

0
51.93
454.23
2242.23
4477.23

A. Response Time
In order to measure the responsiveness of our implementation, the time taken to parse the log files, the
execution time of Pig in the cluster, and the execution
time of the total system were measured separately. These
values were taken by varying the number of DataNodes
in the Hadoop cluster.
1) Time Taken To Parse The Log Files Using Failmon
Package: The NameNode schedules jobs on all DataNodes
available in the Hadoop cluster. The DataNodes collect
data from various sources like Hadoop log files, system
log files, and various operating system utilities and upload
the same to HDFS periodically.
Table II shows the response times for collecting the information and storing in HDFS depending on the number
of nodes.

Table IV
T IME TAKEN T O RUN P IG O N T HE C LUSTER

Table II
T IME TAKEN T O PARSE L OG F ILES U SING FAILMON PACKAGE
Number of Nodes

Time taken in seconds

0
1
2
3
4
5

0
17.23
18.78
23.85
24.98
25.65

Number of Nodes

Time taken in seconds

0
1
2
3
4
5

0
29.2
35.64
42.9
50.81
63.36

From Figure 5, it can be seen that as the number of

nodes in the cluster increases, the time taken for the Pig
tool to complete its task also increases.

From Figure 4, it can be seen that as the number of

nodes increases, the time taken to collect log files from
each DataNode increases.
Figure 5.

Time Taken To Run Pig On The Cluster

This is because, as the number of nodes increase, the

data collected also increases, and hence the increase in
time.
The observed data follows linearity and has, as before,
one independent variable, the number of nodes. Here too
we use linear regression model to predict the response
times for larger numbers of nodes.
The predicted response times in seconds for large nodes
is given in Table V.
Figure 4.

Time Taken To Parse The Log Files

Table V
P REDICTED T IME TAKEN T O RUN P IG O N L ARGE N ODES

The observed data shows linearity, and there is one

independent variable, the number of nodes. Here we use
linear regression model to predict the response times for
larger numbers of nodes.
The predicted response time in seconds for large numbers of nodes is given in Table III.
2) Execution Time For Running Pig On The Cluster:
Table IV gives some sample values of the time taken to run
Pig on the cluster. The time taken to run Pig are recorded
with increasing numbers of nodes in the cluster.

Number of Nodes
0
50
100
500
1000

Time taken in seconds

0
120.31
1120.21
5564.21
11119.21

The time taken to collect, parse and upload the local

data into HDFS is very small compared to analyze the
62

Table VIII
T EMPERATURE I N N AME N ODE AND D ATA N ODE AT VARIOUS T IME
I NSTANCES

data collected using Pig scripts. This observation can be

observed using the tables above.
B. Execution Time Of The Entire System

Time
instant

Temperature on
NameNode

Temperature on
DataNode

1
2
3
4
5
6
7
8

75
72
75
60
64
71
80
72

61
62
61
61
62
63
62
63

The test is run by integrating the two modules:

Network Topology Discovery.

Hardware Failure Monitoring.

It includes starting of Hadoop, collection of Hadoop

and system log data from all DataNodes, analyzing the
data using Pig, uploading the output on HDFS, messaging
to a failed DataNode about the failure in its hardware
component, and discovery of the network topology of the
Hadoop cluster.
Table VI gives the values of the response times of the
integrated system. The response times are recorded with
the increasing nodes in the cluster.
From Figure 6, it is evident that as the number of nodes
in the cluster increase, the times taken to execute both
FailMon and Pig on the cluster also increase.
Table VI
E XECUTION T IME O F T HE E NTIRE S YSTEM
Figure 7.
Number of Nodes

Time taken in seconds

1
2
3
4
5

64.85
80.19
99.26
122.36
145.89

Change In Temperature In NameNode Vs DataNode

C. Change In Hard Disk Temperature

The sample readings for temperature were collected
from the NameNode and the DataNode which had the
same manufacturer of the hard disk, as given in Table VIII.
Figure 7 shows that as there is an overhead of controller processes in the NameNode, the temperature on
the NameNode was recorded to be higher than that of the
DataNode.
V. C ONCLUSION

Figure 6.

The probability of hardware component failures increases as the number of nodes in a cluster increases. Since
Hadoop is widely used in scalable computing systems
where the network topology is dynamic and potentially
unknown, locating hardware failures is also a challenge.
In this scenario, the hardware failure detection schema
that we propose gives a predictive ability for detecting
failures and ensures reliability. It also paves the way for
future work directed towards developing mechanisms to
perform smooth handoff of data and running tasks from
the signalled DataNode to another working node in the
cluster. In the network discovery module, a topology
map that displays only the live nodes in our Hadoop
network is generated, and also helps track the network
services and detect recent outages. This information helps
network administrators to ensure highly available services
in Hadoop.

Execution Time Of The Integrated System

Table VII
P REDICTED T IME TAKEN F OR E XECUTION O F E NTIRE S YSTEM O N
L ARGE N ODES
Number of Nodes

Time taken in seconds

0
50
100
500
1000

0
245.54
2084.24
10256.24
20471.24

VI. ACKNOWLEDGMENT
The predicted response times in seconds for large numbers of nodes are shown in Table VII.

We thank Prof. Debabrata Das for his continual support

and guidance throughout the execution of this work.
63

R EFERENCES
[1] Hadoop, The Apache Software Foundation, Dec. 2011.
[Online]. Available: http://hadoop.apache.org/
[2] I. Koltsidas, K. Gupta, and P. Sarkar, Log collection and
failure monitoring in large clusters running hadoop, IBM
Almaden Research Center, July 2008.
[3] Pig Tool. [Online]. Available: http://pig.apache.org
[4] Pig
Latin.
[Online].
Available:
http://pig.apache.org/docs/r0.8.1/piglatin ref1.html
[5] T. White, Hadoop: The definitive guide.
2010.

Yahoo Press,

[6] Hadoop
tutorial-YDN.
[Online].
Available:
http://developer.yahoo.com/hadoop/tutorial/module7.html
[7] A. M. Marques, Zenmap, 2007. [Online]. Available:
http://nmap.org/book/zenmap.html
[8] Opennms, OpenNMS Official Website, Jul. 1999. [Online]. Available: http://www.opennms.org/wiki/Main Page
[9] N. Nakka, A. Agrawal, and A. Choudhary, Predicting node
failure in high performance computing systems from failure
and usage logs, in Parallel and Distributed Processing
Workshops and Phd Forum (IPDPSW), 2011 IEEE International Symposium on, may 2011, pp. 1557 1566.
[10] E. W. Fulp, G. A. Fink, and J. N. Haack, Predicting
computer system failures using support vector machines.
[11] F. Sultan, A. Bohra, Y. Pan, S. Smaldone, I. Neamtiu,
P. Gallard, and L. Iftode, Nonintrusive failure detection
and recovery for internet services using backdoors.
[12] K. Shvachko, H. Kuang, S. Radia, and R. Chansler, The
hadoop distributed file system, in Mass Storage Systems
and Technologies (MSST), 2010 IEEE 26th Symposium on,
may 2010, pp. 1 10.
[13] M. G. Noll, Running Hadoop on Ubuntu Linux (MultiNode Cluster), Aug. 2007, My digital moleskine. [Online].
Available: http://www.michael-noll.com/tutorials/runninghadoop-on-ubuntu-linux-multi-node-cluster/
[14] Failmon, Hadoop Releases, Aug. 2010. [Online].
Available: http://hadoop.apache.org/common/releases.html
[15] M. G. Noll, Running Hadoop on Ubuntu Linux (SingleNode Cluster), Aug. 2007, My digital moleskine. [Online].
Available: http://www.michael-noll.com/tutorials/runninghadoop-on-ubuntu-linux-single-node-cluster/
[16] Smartmontools.
[Online].
Available:
https://help.ubuntu.com/community/Smartmontools
[17] lm
sensors.
[Online].
Available:
https://launchpad.net/ubuntu/+source/lm-sensors/1:3.3.12ubuntu2

64