2011 Frontiers of Information Technology
2011 Frontiers of Information Technology
DOFUR: DDoS Forensics Using mapReduce
Rana Khattak, S hehar Bano, Shujaat Hussain, Zahid Anwar
SEECS, NUST,
Islamabad, Pakistan.
{10msccsmkhattak; 10msccsabano; shujaat.hussain; zahid.anwar} @seecs.edu.pk
Abstract-Currently we have seen a very sharp increase in
network traffic. Due to this increase, the size of attack log files
has also increased greatly and using conventional techniques to
mine the logs and get some meaningful analyses about the
DDoS attackers location and possible victims has become
increasingly difficult. We propose a technique using Hadoops
MapReduce to deduce results efficiently and quickly which
would otherwise take a long time if conventional means were
used. The aim of this paper is to describe how we designed a
framework to detect those packets in a dataset which belong to a
DDoS attack using MapReduce provided by Hadoop.
Experimental results using a real dataset show that
parallelising DDoS detection can greatly improve efficiency.
Keywords - DDoS, mapReduce
I. INTRODUCTION
Denial of Service (DoS) attack is launched to make an
internet resource unavailable often by overwhelming the
victim with a large number of requests. DoS attacks can be
categorized on the basis of single source and multi source.
Multi source attacks are called distributed Dos or DDoS
attacks.
Today computers and networks are attacked very
frequently and these attacks can be very expensive and hard
to recover from. This leads to need of computer forensics,
which can help us identify the following: are we under
attack? Who is attacking us? And which incoming traffic is
malicious or a part of the attack? Evidence of such
intrusions is required in case the affected wants to pursue
the court and legal action is to be taken against the
adversary.
These forensics investigations are very hard and often
done manually in many cases [1]. Information about
network, web and different protocols and applications are
saved in log files. These log files usually save everything
and anything indiscriminately. An intelligent attacker can
mask his attac ks by mixing it up with legitimate requests.
Figure 1: DDOS attack
978-0-7695-4625-4/ 11 $26.00 2011 IEEE
978-0-7695-4625-4/
DOI 11 $26.00 2011 IEEE
10.1109/FIT.2011.29
DOI 10.1109/FIT.2011.29
The type of attack that we will be dealing with is DDoS
attack. Distributed DoS attacks are a major security concern
these days. DDoS attacks are launched to compromise the
availability of a system or a network like DoS but unlike
DoS. the attack is launched by the adversary by creating
zombies that send several requests to the victim and
overwhelm it with large amount of requests. This creates a
bottleneck and the victim can no further entertain requests
from legitimate users, denying service to them.
During DDOS attacks, the log files swell up to huge
sizes. These log files if analysed properly and effectively
can help detect and recover from a DDoS attack. These log
files can take a long time if processed through conventional
means thus delaying the results due to which the recovery
phase could be delayed. As the attacker is progressing and
devising new ways to commit crime and intrude, our
solution to the problem emphasis is on the fact that the
forensic community also needs to get smarter and instead of
using traditional ways for detecting advanced attacks,
should focus on using advanced techniques that are efficient
and helps detect intrusions in real time.
MapReduce is a model provided by Hadoop that is used
for parallel processing of distributed data[1]. MapReduce
often consists of several distributed computing machines
working together in the form a cluster. There is a cluster
head or master machine and several mapper and reducer
machines. Tasks are assigned and managed by the master.
MapReduce is a 2-step process. The map phase is the first
processing step in which the input hadoop file system
(HDFS) is broken into several splits and each split is
assigned to a mapper. These splits are the n parallelly
processed by the mappers. In the Reduce Phase the
intermediate results provided by the map phase are
summarized and associated records are processed by single
reducer.
Since in DDoS attac ks large data sets have to be analyzed
quickly to take decision about suspicious activities,
MapReduce can prove to be a very good solution. We tested
our forensics system with the data set provided by Lincoln
labs[1] and showed that DDoS forensics on Hadoop
MapReduce can produce results much efficiently than a
serial analyser. We propose distributed forensics for
distributed attacks.
Section 2 represents work of others related to ours.
Section 3 describes the design and architecture of our work.
Section 4 gives the details about the imple mentation of our
work. Section 5 is related to the evaluation of our strategy.
Section 6 concludes the report and defines areas of future
work on the project.
117
117

Figure 2: Analyzing data collected at firewall through
MapReduce Hadoop
II. RELATED WORK
Lincoln laboratories carried out similar forensics [5].
They created an example attack scenario, implemented it,
carried out a DDOS attack and captured the attack traffic at
Eyrie Air Force Base, the site where zombies were created
by the adversary. Their attack scenario consisted of 5
phases.
The attacker sends ICMP echo requests to all the subnets
of the Air force Base to determine which hosts are alive.
The hosts that were found up in the previous phase are then
probed to determine which hosts are running the Sadmind
program. This helps the attacker to know which hosts might
be vulnerable to the buffer overflow vulnerability present in
the Sadmind remote administration tool. This is done by
asking the hosts at Air Force Base for the port number of
Sadmind service and then trying to connect to the port
number supplied in response. Successful connections to
hosts are shortlisted.
The hosts that were shortlisted in phase 2 are now
exploited by buffer overflow vulnerability present in them.
Several attempts are made in order to create a new user
named hacker 2 on the remote machines. The attacker now
installs the mstream tool on the hosts where hacker 2 was
successfully created. Each client informs the master when it
is alive. The master keeps a record of live clients.
The attack is launched by overwhelming the target victim
with a huge number of connection requests by all the
mstream clients using a spoofed source address.
Processing and Analyzing Large Scale Data is a project
in which intrusion detection has been carried out through
Hadoop MapReduce using the TRW algorithm. The
algorithm only identifies host IP address to be malicious or
not. It cannot detect spoofed IPs and does not detect those
packets that are part of the attack.
[5] has used fuzzy logic and expert systems to detect
networks intrusions. Their method is efficient, automatic
and reliable. Though their results show quite an increase in
the correct classification rate of log, compared to the
118
118
traditional means of evidence ga thering but still it is not
100% and there is room for improvement.
A tracing back technique has also been proposed in [7]
for digital forensics. Both of these technique s need to be
appreciated for making an effort and improving the
methodology of network forensics but they still lack behind
in comparison to the adversarys speed and intelligence.
We, in our project have used the same datasets and attack
situation assumptions as provided by the Lincoln
laboratories. And correctly identified those packets that
belong each phase of the attack.
III. DESIGN AND ARCHITECTURE
The MapReduce cluster was provided with a Distributed
File System (DFS) containing a large set of attack data from
the experiments carried out by the Lincoln Lab in 2000 for
DARPA. The master distributes different sets of data among
different mappers and the intermediate results are stored at
the mappers. The master then assigns the task of extracting
information regarding the attack data to the reducers. And
the final results are stored to the distributed file system.
The data files contain the information of all the packets
that has been sent from or received by a network. Each
packets information included the time stamp, packet
length, source IP, destination IP, source port, destination
port, flag (if the packet is a syn/ack/fyn/reset), and the
protocol (if the connection is tcp or udp) for each packet.
All DDoS attacks follow a similar lifecycle with minor
variations:
1- IP Sweep
2- Port scan
3- Vulnerability exploit
4- Dos attack
We maintain a data structure in which each row
represents each host on our own network. The number of
objects in each row is variable. Each row object represents a
connection (IP address + port) to an outside host and other
information about it (e.g. connection state, outbound and
inbound direction).
No. Time Source Destination
Protocol Info
1 0.000000 202.77.162.213 172.16.115.20
TCP 49212 > telnet [SYN] Seq=0 Win=32768 Len=0
MSS=1460 WS=0 TSV=190935 TSER=0
Frame 1: 74 bytes on wire (592 bits), 74 bytes captured (592
bits)
Ethernet II, Src: 3com_9c:b2:8e (00:10:5a:9c:b2:8e), Dst:
Oracle_89:ba:28 (08:00:20:89:ba:28)
Internet Protocol, Src: 202.77.162.213 (202.77.162.213),
Dst: 172.16.115.20 (172.16.115.20)
Transmission Control Protocol, Src Port: 49212 (49212),
Dst Port: telnet (23), Seq: 0, Len: 0
Figure 2: Pcap file converted to a text file

Figure 3: Data Flow and Architectu
The dataset provided is in binary fo
converted into text format as shown in
converting into text for mat the file was g
the Hadoop cluster as shown in figure 3.
We populate the connections table by an
packets one by one. If a particular IP ad
'horizontal threshold' or 'vertical threshol
ping requests, we label it as suspicious a
input to the next phase to further observe it
Horizontal threshold is a limit on th
addresses allowed to ping a single host
within a fixed time window. Vertical thres
the number of ping requests an external ho
our internal hosts within a time window
If an external host crosses the 'horizon
or the 'vertical port threshold', it can qu
phase 3. Horizontal port threshold is a lim
of ports an external host tries to connect
interna l hosts within a time window. Vertic
a limit on the number of inter nal hosts to t
a single external host may connect to withi
In the phase 3 if the IP addresses that
phase 1 and phase 2 are found to be inv
incoming traffic to a single internal host,
are malformed, it might be a sign of buffe
Such IP addresses are marked.
In phase 4 if the outbound connections
addresses (i.e. none of the source IP addre
network) and the destination is the sa
packets, it might be the case that our ne
been used as zombies to launch dos a
victim. We may deduce that the IP addres
Phase 4 IPs are the main culprits behind th
IV. IMPLEMENTATION
A. Implementation strategy:
We used Java as programming langua
the traffic analyzing technique. Karmasph
environment for creating Hadoop jobs. Ra
with low-level Hadoop management, Karm
provides cluster monitoring, job monitorin
management through a high-level user e
not only saves time learning Hadoop, b
productivity when using Hadoop.
As is the case with most o
also used string matching fo
identified the packets which s
in the five phases and wrote t
was used to accumulate the m
We analyzed the data
laboratories, exported them t
Hadoop to identify packets in
attack phase. Each mapper
e mits a key, value pair
information). Some part of ou
1) for(int i=0;i<3;i++){
2) word.set(tokenizer.next
3) for(int j=0;j<IPs.length
4) if(pkt.indexOf("ICMP")
5) &&
ure diagram
6) {
ormat and it was 7) pkt=pkt.substring(0, co
n figure 2. After 8) word.set("\n\n*******
iven to HDFS for Phase 1: IP Sweep from
*******************
nalysing the dump \n No.Time Source Des
ddress crosses the 9) keyval.set(1);
ld' for acceptable 10) context.write(word, key
and pass it on as 11) }
ts behaviour.
he number of IP This is the code for ident
t on our network phase 1. We start our work
shold is a limit on address as the attacker and
ost sends on all of showing his participation in th
searches for all those packets
ntal port threshold' echo requests. We see that a
ualify as input to a huge number of such req
mit on the number strengthened by proving his
to on any of our the attack. As we can see, the
cal port threshold: phase and a count of one
the ports of which suspicious packet.
in a time window. Each reducer gets the sorted
came down from i.e, attack phase and sums the
volved in a lot of greater than the threshold va lue, the reducer emits ( attack
and such packets phase, no. of packets) sum. A
er overflow attack.
is used as a combiner on the m
s have spoofed IP Some part of our code for th
sses belong to our checks if the number of ICM
ame for all such particular threshold:
etwork hosts have 1) int sum=0;
attack against the 2) while (values.hasNext( ()) {
sses short listed in 3) sum+=values.next().ge et(); }
his attack. 4) if(sum>=threshold)
5) while(values.hasNext() () ) {
N 6) context.write(key, sum) m)
7) }
age to implement
here is a graphical B. Challenges In Implem mentation:
ather than dealing
The datasets provided by L
masphere services
form of Pcap files. We neede
ng, and file system
so we converted the Pcap file
environment. This
file among mappers but the p
but also increases
that the input reader splits th
119
119
of the programs in Hadoop, we
or malic ious packets. Mappers
satisfied the conditions de fined
them in the file. Reduce phase
map phase outputs.
asets provided by Lincoln
to a text file and programmed
n the dataset that belong to each
takes a packet as input and
for (attack phase, packet
ur code for the mapper is:
tToken());}
h;j++)
")>0
&( pkt.indexOf("(ping)")>0)
ount);
*************************
m a remote site
***********************\n
st Protocol Info\n"+pkt);
yval);
tifying packets that belong to
by suspecting a particular IP
then prove him as culprit by
he attack. In line 4, the mapper
that are participating in ICMP
particular IP address produces
quests. This suspicion is then
participation in each phase of
e mapper then creates a pair for
e for each occurrence of a
d records according to the key
e counts for each phase, if it is
As an optimization, the reducer
.
map outputs
he reducer is shown below tha t
MP packets is greater than a
Lincoln laboratories were in the
ed the input to be in text form
es to .txt. MapReduce splits the
problem we faced initially was
he file by size. We didn't want
any packet to be spread across several mappers. To
overcome this we created our own custom reader which
could treat each packet as a single whole object, taking file
splitting into consideration. Our code for the custom reader
is:

1) long start = split.getStart();

2) end = split.getStart() + split.getLength();
3) fsin.seek(start);
4) if (start != 0) {
5) readUntilMatch(endTag, false);
6) }
Figure 4: Comparison
7) public boolean nextKeyValue() throws IOException
{
VI. FUTURE WORK AND CONCLUSION
8) if (!stillInChunk) return false;
9) boolean status = readUntilMatch(endTag, true); As the current work converts the binary files to text files
10) value = new Text(); and does processing on them, so the next step would be
11) value.set(buffer.getData(), 0, buffer.getLength()); implementing binary input module which will take binary
12) key = new LongWritable(fsin.getPos()); files as input. We will also further generalize the solution to
13) buffer.reset();} detect the attacker and victim IPs from a data set based on
entropy calculations and also add a module to detect packets
with spoofed IPs.
In line 1, the variable split is assigned the start of our
packet, which in our case was marked by the No. token. R EFERENCES
The end of the split is calculated by adding the length of the
[1] Hadoop, Wiki, http://wiki.apache.org/hadoop/PoweredBy , date
split to the start of the split. Line 5, is the start of a function accessed: 5 June 2011.
that makes a check for the end of the file. [2] J. Lin and C. Dyer, Data-Intensive Text Processing with MapReduce,
Morgan & Claypool Publishers, 2010.
1) private boolean readUntilMatch(byte[] match, [3] T . White. 2009. Hadoop: The Definitive Guide. O'Reilly Media, Inc.
boolean withinBlock) throws IOException { June 2009.
[4] J. Dean and S. Ghemawat, MapReduce: Simplified Data Processing on
2) int i = 0; Large Cluster, OSDI, 2004
3) while (true) { [5] Niandong Liao *, Shengfeng T ian, T inghua Wang , Network forensics
4) int b = fsin.read(); based on fuzzy logic and expert syst em.
[6] Lincoln Laboratotry, Massachusetts Institite of T echnology, accessed
5) if (b == -1) return false;
on:1st June 2011,
6) if (withinBlock) buffer.write (b); http://www.ll.mit.edu/mission/communications/ist/corpora/ideval/data/200
7) if (b == match[i]) { 0/LLS_DDOS_1.0.html .
8) i++; [7] H. Achi, A. Hellany & M. Nagrial. Network Security Approach for
9) if (i >= match.length) { Digital Forensics Analysis.
[8] Apache Hadoop, ht tp://hadoop.apache.org/ , accessed: 2 June 2011.
10) return fsin.getPos() < end;
11) }
12) } else i = 0; }}

V. EVALUATION AND RESULTS

We compared the performance of our program written in
java and distributed through Hadoop and a sequential java
code having the same input file. The tests were carried out
against both the programs with different data sizes. It was
found that the sequential program performed better than
Hadoop up until a certain size because Hadoop has an
additional complexity of distributing tasks which becomes
an issue for small data sizes. But as the input sizes grew,
the performance of Hadoop grew exponentially. This was
because of the face that in Hadoop the nodes could perform
parallel and produce results in muc h less time.
These were two more large data sets we experimented
on. The size of one dataset was 911 MB. The dataset result
showed 3 malicious hosts. The 6.9 GB data did not contain
any malicious hosts. There was significant improvement in
execution time.
Both time and number of processors involved, are the
factors that can greatly speed up the execution time and
hence performance of the Hadoop MapReduce system.

120
120