Platform based on an embedded system to evaluate the intrusion detection system

Mohammed SABER, Mohamed EMHARREF, Toumi BOUCHENTOUF

Department of Computer Science, National School of Applied Sciences Mohammed First University Oujda, Morocco mosaber@gmail.com, m.emharraf@gmail.com, tbouchentouf@gmail.com
AbstractThe evaluation of an IDS (Intrusion Detection System) is a difficult task. We can distinguish between the fact to test the effectiveness of the entire system and therefore all the characteristics of an IDS [2] and to test a component of an IDS. In this paper, we present an evaluation approach for IDS based on measuring the performance of its various components to evaluate its characteristics. For this we made a detailed design, then we have implemented hardware platforms based on embedded systems. And to measure the performance of the components of an IDS, we have integrated the SNORT IDS to these platforms. The obtained results show that the performance characteristics of an IDS depends on the performance of its components. Keywords; Evaluation; IDS; Performance; Embedded System; FPGA; ARM; DSP; SNORT

Abdelhamid BENAZZI
Department of Computer Science, High School of Technology Mohammed First University Oujda, Morocco benazzihamid@yahoo.fr

components of IDS. In our case, we measure the performance of these components, which allows subsequently to evaluate its characteristics. In practice, most IDS suffer from several problems, taking into consideration the large number of false positives and false negatives, and the evolution of attacks. All these problems increase the need for an evaluation system for the IDS; in this context, many attempts to assess took place [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]. They are based on the classification of attacks, which aims to classify the attacks to simplify detection, either by technology or detection range, or by the generation of attack scenarios to understand the behavior of attacks and by other criteria. But the great weakness of these assessments is that they do not cover all the characteristics of an IDS as cited in [2]. In this paper, we propose a new approach EIDS (Embedded Intrusion Detection System) for evaluating IDS mounted on a hardware platform based on embedded systems to test the performance of the various components of an IDS, which can allow later to evaluate its characteristics. For this we set up two platforms, the first FPGA-based EIDS system FPGA (Field Programmable Gate Array), the second EIDS-ARM based on a complete system LYRTECH SFF SDR, and we set up the platforms on a possible SNORT IDS [ 15]. In the remaining sections of this article, we quote the old IDS evaluations with their weaknesses in Section 2. We discuss the new approach with the design of such a platform in Section 3. In Section 4, we present our prototypes with their components. Then we describe the implementation of SNORT with our platforms in Section 5. In Section 6, we present the results of our evaluation. Finally, in Section 7, we draw a conclusion and pinpoint future works. II.
THE IDS EVALUATION

I.

INTRODUCTION

The evaluation of the intrusion detection systems is a difficult task, demanding a thorough knowledge of techniques relating to different disciplines, especially intrusion detection, methods of attack, networks and systems, technical testing and evaluation. And what makes the evaluation more difficult is the fact that different intrusion detection systems have different operational environments and can use a variety of techniques for producing alerts corresponding to attacks [1]. The task is even more difficult as the IDS must be evaluated not only under normal conditions, but also and especially in a malicious environment, taking particular account of unexpected and sometimes even unknown patterns of use (this is true of almost all the tools dedicated to security such as firewalls, IPS (Intrusion Prevention Systems) and antiviruses). All these considerations make it difficult to build representative data for evaluation. So normally before beginning any experimental test, it is extremely important to identify clearly the objectives of the evaluation. First, it is important to distinguish between tests aim at evaluating the effectiveness of the entire system, or the whole characteristics of IDS [2], which are interested in testing

In this section we present initially the important characteristics for the evaluation of IDS, and we cite research on the evaluation of IDS, before concluding with a discussion.

978-1-4673-1520-3/12/$31.00 2012 IEEE

In the work [2], we identify a number of features which would be interesting for the evaluation of an IDS; we refer to the cover, the rate of detection, false alarm rate, resistance to attacks against IDS, the ability to manage traffic, high bandwidth, capacity correlation, detection of unknown attacks, identifying attacks and finally the ability to distinguish between attacks and intrusions. In recent years, numerous attempts of evaluation have taken place [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]. These evaluations have a particular purpose, characteristics, detection techniques (for scenario or behavior), architecture or detection range, the classifications of attacks or behavior of the attacks. Among the problems of these evaluations are platform tests, either private or public, but complicated to implement; another problem is the experimental conditions that depend on the traffic handled and the different types of attacks without forgetting the evolution of attacks. After citing the old assessments and weaknesses, we present next section the new approach to test the characteristics of the IDS. III.
A NEW APPROACH FOR EVALUATING IDS

Decision: When there is correspondence with are of detection rules, we connect to the corresponding alert action.

Figure 1. SNORT Components.

In this section, we discuss some considerations in the design and implementation of our approach for an efficient system and evolutionary intrusion detection. A. Motivation for a hardware based IDS Despite all the problems mentioned in the previous section, and the nature of the software found in the market, we can not make an assessment of the performance characteristics of an IDS [2] through its components. This is why we choose to use a hardware platform to understand, represent and evaluate the performance of components of an IDS. B. Characterization of the components of an IDS Before we can design and implement our IDS platform, we must first analyze the performance characteristics of the analysis of an IDS. A real-time IDS monitors network traffic by sniffing network packets (capture) and analyzing the traffic data according to the rules of intrusion detection. We use SNORT [15] as an example to describe the main steps of packet processing and analysis of IDS as shown in figure (Figure 1). In the software SNORT, each captured packet goes through the following steps: Packets decoding: is decoded information in the header of packets at different layers and stores information in data structures. All packets pass through this stage. Pretreatment: Each preprocessor function is calls in order; we begin with IP fragments and then we go back to process the TCP or UDP flows. Detection: First, the values of the header of a packet are used to select an appropriate subset of rules for a deeper detection. This subset consists of all the rules that apply to this package. Second, the selected rules are evaluated sequentially.

Thus, an IDS can be considered as a waiting system where the packet buffers are the queues and the IDS is the engine of the service. Obviously, if the IDS processes the packets with a slower rate than that of their arrival, the buffer can be filled and newly arrived packets will be dropped (ie not stored). Therefore, the IDS may not have enough information to accurately analyze traffic and detect intrusions. Therefore, it is very important to design and implement a system to minimize dropped packets by the IDS. C. Considerations for the hardware architecture

It is clear from the foregoing discussion that there are

potential performance gains if the components are implemented in hardware IDS. To achieve this, we will use a pipeline of DSP (Digital Signal Processor). Intrusion detection is an interesting application of hardware architecture point of view because of its need for substantial resources. The analysis of intrusion detection requires much more computing cycles and memory accesses per packet than required by traditional network applications such as IP routing. The analysis consists of several tasks with different patterns of resource use; certain tasks need to load and others have access to the memory bound. Furthermore, the amount of work done for each packet is not constant; it depends on the packet size and network conditions at the moment.

Figure 2. Task allocation to hardware caption.

When designing the architecture of an IDS-based material, we consider both the task requirements of different tests, and the capabilities of each hardware component. Based on these properties and experimental tests, our goal is to determine the best allocation of tasks to hardware resources. Figure 2 summarizes our criteria for mapping tasks into elements of processing equipment. On the ARM TMS, the treatment requires relatively simple operations, to be applied to high-

speed data can be implemented on the DSP. We provide packet capture, filtering, decoding and preprocessing on DSP. Treatments that require complex calculations on data from low flow are better achieved by the ARM processor. Thus, for the decision of the IDS we are on ARM. However, IDS tasks require both a complex calculation and broadband as the detection task. For such cases, our approach is to map the calculation of dynamically reconfigurable hardware, which is able to achieve high performance by optimizing the simultaneous calculation in question. We use an FPGA (Field Programmable Gate Array) co-processor to handle this type of task. In our system, the FPGA co-processor handles the function of pattern-matching. After using our approach for the evaluation of performance indicators of an IDS, we describe our prototype in the next section. IV. OUR PROTOTYPE OF PERFORMANCE OF IDS
CHARACTERISTICS

Figure 4. EIDS based on a ARM TMS (EIDS-ARM).

The main components are an ARM and a TMS 320 Virtex5 Xilinx FPGAs. The FPGA co-processor board is attached to a PCI slot and communicates with the ARM via an internal bus 32-bit, 66 MHz PCI bus with a maximum throughput of 1.5 Gbps. However, the overhead imposed by the PCI interface limits the type of tasks that can be offloaded to the coprocessor. The long latency of PCI transactions means that large data transfers are more efficient than small transfers. The role of the FPGA will be detailed in the section on pattern-matching. We couple an ARM-FPGA interface to improve performance and allow a wider class of tasks to be sent to the coprocessor. It would also allow the system to adapt to changing conditions of traffic by dynamically reallocating tasks between the ARM and FPGA Virtex5 TMS. After quoting the prototypes with their functional architectures, we move to the implementation of SNORT on two prototypes in the next section. V.
OUR PLATFORMS SNORT IMPLEMENTATION

In this section, we describe the components of our prototypes, the implementation of two IDS platforms based on an embedded system EIDS (Embedded Intrusion Detection System), one (EIDS-FPGA) based on an FPGA (Figure 3 ) and one (EIDS-ARM) based on an advanced (ARM TMS) (Figure 4). A. EIDS platform based on FPGA (FPGA-EIDS) A diagram of the hardware platform is shown in Figure 3. It is based on Xilinx Spartan 3E START KIT with a 100 Mbps Ethernet port.

Figure 3. EIDS based on a FPGA (FPGA-EIDS).

Given the difference in complexity between the two hardware platforms EIDS, we use SNORT dedicated implementations for each platform. We use a set of libraries and development tools that are essential for the implementation of SDI platforms. These include a library of memory management, a management queue, a packet capture for multithreaded and filtering and one for IP fragmentation and reassembly. In this section we detail SNORT two implementations. A. SNORT implementation of FPGA-EIDS We implement the main components of SNORT with simple algorithms using Embedded C libraries SNORT who describe the operation simple SNORT (light version) because of physical limitations of Xilinx Spartan 3E platform. In figure 5, for the first Blaze (Slave) is responsible for operations capture frames using the library Libpicap [18], decoding, preprocessing and storage in the SRAM. The second Blaze (Master) is responsible for operations management of the entire system and the decision. As to the processing unit, it is responsible for the operation of detection (pattern-matching).

The main components (Figure 5) is a Xilinx Spartan 3E FPGA [16], where we set up our EIDS-FPGA, which includes a processing unit to detect two virtual chips (Blaze with a frequency of 50MHz and a bus of 32 bits) one for communication and another for the management of EIDS, the SRAM (one part for storing pattern-matching rules and the other for packet processing) and SDRAM (used for processors operations). B. EIDS platform based ARM-TMS 320 (EIDS-ARM) The diagram of our second hardware platform is shown in Figure 4. It uses the SFF SDR LYRTECH [17], with a 100 Mbps Ethernet port.

Figure 5. Components of the EIDS-FPGA.

B. SNORT implementation on EIDS-ARM We implement a complete version of the main components of SNORT, the basis of our second-ARM platform EIDS because it is loosely coupled and easy to customize. Here, we briefly describe the main components of SNORT. According to the figure (Figure 5), the operations of capture and filtering is based on Libpcap [17] are realized by the DSP, the packets are transmitted to the decoder to process their headers. Each packet is then passed through a series of preprocessors, including IP fragmentation and reassembly of TCP flows. Then the packets are controlled by the detection engine (ARM) (Figure 6).

We have a computer connected to our platform and a server running a standard SNORT network hub. We also set a different computer with the software traffic generation on the same hub (Figure 8). The traffic generator was used to send a packet of varying sizes (128, 512 and 1024 Bytes) that contain data and other attacks. Newspapers output of each sensor IDS were compared, and found that EIDS ARM-generated almost the same set of warnings that the software standard SNORT but the first system has operating difficulties. The results of treatment of EIDS, EIDS-ARM and FPGA are sent directly to the supervisor, which stores then in log files.

Figure 8. Network test.

Figure 6. Pipeline Detection.

One of the tasks requiring more intensive calculations made by SNORT is the pattern-matching [19, 20] on the packet contents. To manage the pattern-matching, we have developed an FPGA design (Figure 7), which compares the contents of a packet to any rule SNORT (using models of SNORT rule) simultaneously [21] stored in a memory.

B. RecievePerformance As there is a significant overhead for processing headers of each packet, the greatest influence on performance is due to receive the packet size, which determines the number of arrivals of packets per second. We tested this module with a range of different packet sizes and determined the rate achievable by the number of clock cycles required for each package. The results are presented in Table 1.
TABLE I. Packet size (byte) 128 512 1024 RECEIVE PERFORMANCE Throughput Max (Mbps) 183 246 291

Cycles / Packet 1832 3584 6245

Figure 7. Pattern-matching FPGA.

C. Defragmentation IP Performance The critical factor in the treatment of IP defragmentation is the number of fragments per packet. We find that the performance decreases as the number of fragments increases. Table 2 shows the rate and number of cycles for calculating a packet of 512 bytes with varying numbers of fragments.
TABLE II. Number of fragments 1 4 8 16 DEFRAGMENTATION IP PERFORMANCE Cycles / Packet 654 867 964 1265 Throughput Max (Mbps) 380 280 143 54

VI.

RESULTS AND DISCUSSION

We have implemented both platforms EIDS-FPGA and EIDS-ARM to measure the performance of the components of an IDS, performing functional verification at the system level. The results are presented and discussed in this section. A. Functional verification To verify that our system provides correct results, we compared it with the software distribution standard SNORT.

D. Detection performance The detection rate strongly depends on the time required to transfer a package of ARM in FPGA via the PCI bus. As

expected, the performance is better for large packages, and for small packets. Once the data reaches the FPGA, the processing is completed very quickly. However, the PCI interface limits the overall performance of this module. As mentioned earlier, we hope to reduce this limitation by developing a more effective interface between the ARM and the FPGA. Table 4 shows the performance in the worst of cases, and that's when all incoming packets reach the phase detection.
TABLE III. Packet Size 128 512 1024
DETECTION PERFORMANCE

Throughput Max (Mbps) 20 37 65

After the comparison, we recommend to have a powerful IDS by taking a set of precautions. First, to handle a maximum of traffic, there must be at the entrance to the IDS, a fast system to capture the maximum of frames and avoid the problem of dropping frames. Second, should a system that allows the decoding of frames (frames of decapsulation and separation of headers) in parallel (a Pipeline) and in real time to avoid the problem of overloading the queue at the decoding . Third, we must ensure high processing capacity of parallel patternmatching to increase the speed of identification and detection of attacks. Thus, it would be necessary to use a learning system to identify unknown attacks. And finally, a high-performance decision system is essential to address the complexity of treatment to manage the entire system and make decisions. G. System Benchmarks We have run several system-level benchmarks to determine how the components of the detection pipeline work together. The test environment was the same as described in Section 6.A. These tests are designed to measure system performance under different load levels. We used a traffic generator to send different rates of fixed-size packets at IDS sensor. For each packet size and rate, we measured the percentage of packets that the sensor was able to process and determined the maximum speed at which the sensor could work without losing any packets. Due to the limitations of software and hardware packages in our generation of computer, we were not able to run tests at maximum speed with minimum size packets. As there is a fixed processing overhead for each packet, the tests using small packets generally have a lower yield performance, because there are more packets per second. The most significant factor is the result of detection. If the header of a packet corresponds to the values of some fields in one of the SNORT rules, it must be checked by the phase-matching pattern. Otherwise, no further treatment is necessary; the results of these tests are presented in Table 6.
TABLE VI. Packet size 128 512 1024 Rate of 25 Mbps 70% 100% 100% CHANGE IN TRAFFIC FLOW AS THE PACKET SIZE PERFORMANCE Rate of 50 Mbps 40% 100% 100% Throughp ut of 75 Mbps 26% 100% 100% Flow rate of 100 Mbps 13% 100% 100% Max (Mbps) 8% 100% 100%

E. Pattern-matching performance The important parameters for the FPGA pattern-matching are the number of characters it can store and its flow. Generally with the FPGA, an increase in the use of logic resources causes a delay of increased interconnection and a reduction in maximum operating frequency. Table 4 shows the speed supported by the FPGA for each set of rules, but the actual throughput is limited by I/O PCI connection.
TABLE IV. Number characters 1500 3000 10000 of
FPGA PATTERN-MATCHING PERFORMANCE

Resource Used 15% 25% 60%

Frequency (MHz) 120 115 102

Throughput (Mbps) 980 920 830

F. Comparison of EIDS-FPGA and EIDS -ARM In this part, we make a comparison between the two platforms EIDS-FPGA and EIDS -ARM with SNORT as component performance results above. Table 5 summarizes our comparison:
TABLE V. Comparison criteria Traffic analysis Reassembly of the information Analysis of alerts, and pattern-detection ability mattching Resources used Price of the platform Coverage Number of packets dropped Ability to manage traffic Detection rate Detection of unknown attacks Ability to correlate COMPARISON OF EIDS-FPGA AND EIDS -ARM

EIDS-FPGA
Generates delays Difficulty, the problem of limited material Slow processing

EIDS-ARM
Real time Continuous use of DSP to operations by Pipeline. Very fast processing, by using an FPGA that accelerates this type of treatment More resources Expensive 90% 10% of traffic Very processing 100% fast

Less resources least cost 15% of traffic Almost 85% of traffic Slow processing Problems for detection because the weaknesses of the treatment unit No treatment No treatment

For each of the components of ARM, we used the simulator based on the library that operates with MBDK MATLAB SIMULINK and tests the system components and measures its performance in real time. These tests show that our platform EIDS-ARM, running on an ARM and Xilinx FPGAs TMS 320 Virtex5, was able to achieve performance roughly equal to that reported by the test SNORT software running on a server. VII. CONCLUSION AND FUTURE WORK In this paper we proposed a new approach for evaluating an IDS with these characteristics, based on tests to measure

performance indicators of the components of an IDS. For this we chose a hardware solution based on embedded systems. Our solution is a hardware platform that gives the hand to measure the performance indicators of the components of an IDS. For this we set up two platforms, the first FPGA-based EIDS system FPGA (Field Programmable Gate Array), the second EIDS-ARM based on a complete system LYRTECH SFF SDR. We also describe our experience in the implementation of two prototypes, based on SNORT, the first is a Xilinx Spartan 3E FPGA and the second is an ARM and a TMS 320 Virtex5 Xilinx FPGAs. We also used several libraries that are essential for the construction of the components of IDS on both prototypes. Then we mounted the platform in a network with a snort and traffic generator. We conducted experiments to study the comparative performance of the components of an IDS. The EIDS-FPGA has difficulty operating; it is limited by the hardware side. For EIDS-ARM, it generated nearly the same results as SNORT. We have provided a better understanding of design principles and implementation techniques at high speed, reliable and scalable to test the performance of an IDS. The current work is to improve the libraries used to increase performance at the level of EIDS-ARM, and put on a real network to test more. Another work that is underway is to establish a learning system to increase the performance of pattern-matching. And finally implement the EIDS-ARM to assess IPS (Intrusion Prevention Systems). REFERENCES:
[1]. M. Ranum. Experience Benchmarking Intrusion Detection Systems. NFR Security White Paper, December 2001. [2]. Peter Mell, Vincent Hu, Richard Lippmann , Josh Haines , Marc Zissman, An Overview of Issues in Testing Intrusion Detection Systems. [3]. N.J. Puketza, K. Zhang, M. Chung, B. Mukherjee, R. A. Olsson, "A Methodology for Testing Intrusion Detection Systems", IEEE Transactions on Software Engineering, Vol. 22, Number 10, pp. 719729, 1996. [4]. Nicholas Puketza, Mandy Chung, Ronald A. Olsson and Biswanath Mukherjee, "A Software Platform for Testing Intrusion Detection Systems", Journal of IEEE Software, Vol. 14, Number: 5, pp. 43-51, 1997. [5]. Herv Debar, Marc Dacier and Andreas Wespi, "A Revised Taxonomy for Intrusion Detection Systems", Annales des Telecommunications, Vol. 55, Number: 7-8, pp. 361-378, 2000. [6]. Lippmann R.P., Fried D.J., Graf I., Haines J.W., Kendall K.R., McClung D., Weber D., Webster S.E., Wyschogrod D., Cunningham R.K., and Zissman M.A., Evaluating Intrusion Detection Systems: The 1998 DARPA Off-Line Intrusion Detection Evaluation, in Proceedings of the 2000 DARPA Information Survivability Conference and Exposition

[7].

[8].

[9].

[10].

[11]. [12].

[13].

[14].

[15].

[16].

[17]. [18]. [19]. [20].

[21].

(DISCEX), Vol. 2, 12-26, 2000, IEEE Computer Society Press: Los Alamitos, CA, http://www.ll.mit.edu/IST/pubs/discex2000-rplpaper.pdf. Lippmann R.P. and Haines J., Analysis and Results of the 1999 DARPA Off-Line Intrusion Detection Evaluation, in Recent Advances in Intrusion Detection, Third International Workshop, 20 RAID 2000 Toulouse, France, October 204, 2000 Proceedings, H. Debar, L. Me, and S.F. Wu, Editors. 2000, Springer Verlag, 162-182, http://link.springer.de/link/service/series/0558/bibs/1907/19070162.htm. Herv Debar and Benjamin Morin, "Evaluation of the Diagnostic Capabilities of Commercial Intrusion Detection Systems", International Symposium Recent Advances in Intrusion Detection (RAID 2002), LNCS 2516, Zurich, Switzerland, Springer, LNCS 2516, pp. 177-198, 2002. Jacob W. Ulvila, John E. Gaffney, Jr. Evaluation of Intrusion Detection Systems , Journal of Research of the National Institute of Standards and Technology, 108, 453-473 (2003). Dominique Alessandri, "Attack-Class-Based Analysis of Intrusion Detection Systems", Ph.D. Thesis, University of Newcastle upon Tyne, School of Computing Science, Newcastle upon Tyne, UK, 2004. Davide Balzarotti, Testing Network Intrusion Detection Systems, Ph.D. Thesis, Politecnico di Milano, 2006. Mohammed Saber, Toumi Bouchentouf, Abdelhamid Benazzi and Mostafa Azizi, "Amelioration of Attack Classifications for Evaluating and Testing Intrusion Detection System" Journal of Computer Science 6(7): 716-722 , 2010 Mohammed Saber, Toumi Bouchentouf, Mohammed Ghaouth Belkasmi and Abdelhamid Benazzi "Generation of attack scenarios by modeling CSP for Evaluating and Testing Intrusion Detection System", IJCSNS International Journal of Computer Science and Network Security, VOL.10 No.11, November 2010, page 93 98. Mohammed Saber, Toumi Bouchentouf and Abdelhamid Benazzi, "Generation of attack scenarios by modeling algorithms for evaluating IDS", The 2nd International Conference on Multimedia Computing and Systems (ICMCS'11), April 7-9 2011, Ouarzazate, Morocco. 978-161284-732-0/11/$26.00 2010 IEEE. M. Roesch, SNORT - Lightweight Intrusion Detection for Networks, USENIX LISA Conference, Nov 1999. Software available at http://www.SNORT.org. START KIT FPGA SPARTAN 3E http://www.xilinx.com/products/boards/s3esamplepack/files/S3Eusergui de.pdf. LYRTECH SFF SDR http://www.lyrtech.com/products/sff_sdr_development_platforms.php S. McCanne, C. Leres, and V. Jacobson, libpcap, 1994. Available at ftp://ftp.ee.lbl.gov M. Fisk and G. Varghese, Fast Content-based Packet Handling for Intrusion Detection, Technical Report CS2001-0670, UCSD, 2001. S. Staniford, C.J. Coit, and J. McAlerney, Towards Faster String Matching for Intrusion Detection, DARPAInformation Survivability Conference, 2001. C.R. Clark and D.E. Schimmel, Efficient Reconfigurable Logic Circuits for Matching Complex Network Intrusion Detection Patterns, International Conference on Field Programmable Logic and Applications (FPL), Sept 2003.