76 (IJCNS) International Journal of Computer and Network Security,

Vol. 2, No. 1, January 2010

A Comprehensive and Relatively Fast Web

Archiving Application for Offline Browsing
Mr. B.Vijaya Babu1 and Prof M.S.Prasad Babu2
1
Professor, Dept. of Computer
Science,
Usha Rama College of Engineering &
Technology , TELAPROLU, Near Gannavaram,
Krishna(D.t) vijaymtech28@gmail.com

2
Dept of
CS&SE,
A.U.College Of Engineering, Andhra
University Visakhapatnam
msprasadbabu@yahoo.co.in

Abstract: The task of archiving or mirroring the websites has
become more difficult in the recent years, because of the rapid
devolopments in designing technologies of websites. The users
and organizations are facing many challenges in term of
restoration of all types of links originally present on the
websites. More over preservation, accessing and interpretation
of the data present on the websites for future references poses
new problems. This paper focuses on the design
,implementation and analysis of an optimized, multi threaded,
website specific, application that saves the websites at a
relatively faster time on to a user defined location of a local
disk in the form of a file ,which is useful for offline browsing.
The problems and limitations of existing state of art archiving
tools in terms of comprehensive retrieval of the links, and speed
of archiving have been addressed. This application is compared
with the existing open sourced utilities like web eater 0.2.1 and
winHTTrack Website copier 3. 43-6 versions.
Key words: Web archiving, open source tool, running time,
active threads.
1. Introduction
Web archiving or mirroring is the process of collecting
portions of the World Wide Web(WWW) and ensuring the
collection is digitally preserved for the future references
and interpretation of research scholars, scientists, business
people, various government and private organizations of
different countries.The web is a pervasive and ephemeral
media where modern culture in a large sense finds a
natural form of expression. Publication, debate, creation,
work and social interaction in a large sense: many aspects
of the society are happening or reflected on the Internet in
general and the web in particular [7].
Archiving websites is important for three main reasons.
Firstly, archiving web sites is justified by the documentary
value the website possesses themselves. Archived websites
are a necessary evidence material for research into the and
the evolution of this medium itself history. Secondly, the
websites them selves are having large informational value ,
it is impor- tant to realize that the websites can be frozen as
on-line sources one way or the other. Thirdly, websites also
have some cultural value, because they belongs to the
digital heritage and they are the material witness of our
society and with out proper archiving policy, they will be lost
for the future[12].
Since a website is a unit of linked files including text
files, sound files, data bases and scripts etc., the author or
the organization that has created the site is protected in
exactly the same way as the author of a classical literary
work. An organization or the author of the site has to
consider the legal copy right implications, when drawing up
an archiving strategy for their websites. Libraries of different
countries and various organizations that deal with the
reproduction of the information see copy right as an
obstruction. The current copy right laws do not allow
enough possibilities to make reproductions of publications,
with out the authors permission. A lot information on the
sites is being shielded by copy right, making its
reproduction subject to a number of clearly defined
conditions [12][7].
2. Study of literature on related work
An extensive study has been done on the literature of
various web archiving, mirroring utilities, applications and
tools, both licenced as well as open sourced . The study of
the state of art technologies on the related work has
provided enough base and platform for the development of
our application.
J. Cho et al..have described the design and performance
of WebBase, a tool for Web research. The system includes
a highly customizable crawler, a repository for collected
Web pages, an indexer for both text and link-related page
features, and a high-speed content distribution facility. The
distributed model enables the researchers world-wide to
retrieve pages from WebBase, and stream them across the
Internet at high speed [9]. Masanes.J has presented various
crawling algorithms and approaches undertaken today by
different institu- tions; it will discuss their focuses,
strengths, and limits, as well as a model for appraisal and
identifying potential comple- mentary aspects amongst
 (IJCNS) International Journal of Computer and Network Security, 77
Vol. 2, No. 1, January 2010

them. following diagram.

He stated that the completeness of a Web archive can be
measured in horizontally by the number of relevant entry
points found within the designated perimeter and vertically
by the number of relevant linked nodes found from this
entry point.[7,10]. It was shown that, not all the pages are
stored, but some of the hidden pages are missing and he also
presented models for extensive and intensive archiving
mechanisms. B.Vijaya Babu et al. in their comparative study
have concluded that the comprehensive archiving or
retrieval of the links of the website also depends on the
technologies that were used to design the individual pages of
the websites[1].
We have also gone through the literature of existing open
source archiving tools which are useful in web site mirroring
and other re-creation activities. Out of all those mirroring
tools, cURL[3], HTTrack [9]and Stanford WebBase[9,18] are the
most optimized tools, but are having their own advantages
and disadvantages. curl is fast, extensible, and fully featured
as a web-downloading tool. It can use telnet, ftp and http
protocols. It has build-in encryption for SSL connections. It
fills in forms, providing passwords to websites if requested. It Figure 1. Screen shots of web eater
follows all redirects and collects cookies automatically. 0.2.1
cURL has a command line interface .Curl supports resumed
transfers both ways on both FTP and HTTP. The main The winHTTrack Website copier 3. 43-6 on the
drawback of cURL is it lacks spidering ability. By itself, it other hand is a GUI tool and is easier for implementation.
cannot mirror whole websites. A wrapper must extend cURL The screen shot of the home page is as shown below.
to spider sites and rotate through links[9].
Tarmo Robal has designed a web crawler software
component which is able to map Estonian Internet domain
structure, and index its web page source[15]. Vladislav
Shkapenyuk ans Torsten Suel in their work have described the
design and implementation of a distributed web crawler
architecture that runs on a network of work stations. The
scalar scales to several hundred pages per second, and is
resilient against sytem crashes and other events, which can be
adapted to other applications [16]. Allen Heydon and Mark Figure 2. Screen shots of winHTTrack Website copier 3.
najork have described about the Mercator[17], a scalable and 43-6
extensible Web Crawler and discussed about the alternative
trade-offs in the design of web crawlers. Paulo Boldi et al.
In our study we have found that HTTrack web site
have reported the implementation of Ubi Crawler, another
copier[9] is better than cURL, as far as the issues of
scalable and distributed web crawler with the features like
recreating the links, as it employs the most professional
platform independence, fault tolerance[18].
interface of the evaluated crawlers. It actually has two
interfaces, one GUI, one command line. It has many options
2.1 Study of Existing Open sourced Tools on the and accurately downloads im- ages and rewrites links. It has
Related Work SSL capability [9]. It spiders a prescribed depth into the
Number of tools and utilities are available in the form of site. It can spider several sites in series. But it lacks the
licensed with copy rights and as open source applications on extensibility of cURL. Difficulties in the core code can lead to
the related work, but most of them are lacking the ease of awkward workarounds. CDL, for instance, might not be able
to build form guessing into HTTrack, which is required to
implementation and comprehensivness.More over, they are
reach much of the deep web [9].
not meeting basic requirements of archiving.
Stanford’s WebBase[ 11] mirrors sites with almost
The open sourced tools like cURL, winHTTrack Website
copier 3 . 43-6, and web eater 0.2.1 have been studied. Both perfect recall of mirrored html. Simple sites with little
cURL and web eater are command line tools and web eater dynamic content can be mirrored with ease. But Stanford
0.2.1 is not comprehensive in terms of retrieval of links WebBase does not rewrite links within pages. It does not
and pages. The archiving efficiency of Web eater 0.2.1[13] is download files required in an EMBED tag as used by Flash
not up to the mark. The screen shots for running and and other programs. It does not parse or even scan
implementation of Web eater 0.2.1 are as shown in the JavaScript for links or images. WebBase may be extended
78 (IJCNS) International Journal of Computer and Network Security,
Vol. 2, No. 1, January 2010

only with great difficulty. The C++ code is not ANSI }

compliant, and cannot be compiled with a modern
compiler[9][11]. Rui Cai et al. have studied about an 4. Architecture Diagram
intelligent crawler with the main idea of learning about the
site map of a forum site with a few pre-sampled pages, and
then decide how to select an optimal traversal path to
avoid duplicates and invalids[2].

3. Pseudo Code
According to Shalin Shah [19], part of the pseudo code
summary of the algorithm that can be used to implement
the application is given below :

Add the URL to the empty list of URLs to

search

While not empty(the list of URLs to

search)
{

Take the first URL in from the list of

URLs
Mark this URL as already searches
URL Figure 3. Architecture Diagram of standard Web
Crawler
If the URL protocol is not HTTP
then break;
go back to 5. Flow Diagram of the Web Archiving
while Application

If robot.txt file exist on the site

then
If file includes “ Disallow” statement
then break;
go back to while

Open the URL

If the opened URL is HTML file

then break;
go back to while
Iterate the HTML
file

While the HTML text contains another link

{ If robot.txt file exist on URL/site then

If file includes” Disallow” statement
then break;
Go back to
while
Figure 4. Flow Diagram of the Web Archiving Application
If the opened URL is HTML file
then
If the URL is not marked as searched then
Mark this URL as already searched
URL

}
 (IJCNS) International Journal of Computer and Network Security, 79
Vol. 2, No. 1, January 2010

6. Implementation of the Application and Case

Study
This application is implemented on a Dell T100 power edge
Model server system with 4GB RAM, 250 GB SATA Hard
Disk and 1 Gbps Network card(Ethernet) . The system has a
2 Mbps dedicated leased line internet connectivity .The
website www.usharama.com is taken as the reference site
with prior permission, for the implementation of the
application.
Figure 6. Screen Shots for Threads=1 and Depth=5 with
BFS Algorithm

Figure 5. Screen Shots of Home Page of Web Archiving

Application

The implementation results in observations for different

active threads and the running time, for BFS and DFS
algorithms are shown.

Figure 7. Screen Shots for Threads=1 and Depth=5 With

DFS algorithm

The page size of 3000kb and a depth of 5 are set for the
implementation. Some interesting results have been
observed for different active threads selected. As the number
of threads increases the running time for saving the website
is observed decreasing. This running time also depends on
the factors like the bandwidth of the Internet connectivity,
speed of packet transfer rate, congestion at that particular
moment and also on the configuration of the system.
80
The content of the website is archived in the folder
UR on the desk top and it consists the following folders.
a) Images
b) Includes(Jscript and CSS Files)
c) Jsfiles
d) Photos
e) Pages and to browse the website in offline, the
index.html also has been created. The observations are
tabulated inTable1 and Table2 as shown below.
Table1: Observations for BFS algorithm
Table2: Observations for DFS algorithm
From the results, the BFS crawling efficiency is better than
DFS, at the cost of running time and vice versa.The values of
running time for various active threads are depicted as
shown in fig 8.
Figure 8. Comparison of running time for BFS and DFS
7. Comparison of Open sourced tools web eater
0.2.1 and winHTTrack Website copier 3. 43-
6 versions
The open source tools winHTTrack website copier 3. 43-
6 and web eater 0.2.1 are implemented and compared with
our application. The winHTTrack website copier 3.43-6 is
compre- hensive in terms of restoration of the links but
takes much longer running time of 22 minutes and 21
(IJCNS) International Journal of Computer and Network Security,
Vol. 2, No. 1, January 2010
seconds.
Figure 9. Output screenshot of winHTTrack website copier
3. 43-6
But the web eater 0.2.1 tool has given poor results in terms
of restoration of the links as well as the running time. It has
retrieved only the home page or the index page.
Figure 10. Output screenshot of web eater 0.2.1
8. Conclusions
This archiving application saves all the links, except
external links of the website originally present in the site
and a l so retrieves the pages written in either aspx,html, or
PHP etc., in to the archived folder. The application is
compared with the existing state of art open sourced
archiving tools like web eater 0.2.1 and winHTTrack
Website copier 3. 43-6 versions . It has shown a much
improved performance in terms of running time and number
of links visited. As the ultimate objective of our work is to
develop a comprehensive, open sourced web archiving tool,
this work provides a platform and scope for future
enhancements like the retrieval of external links and
consequently to the problems related to memory
requirements.
9. Acknowledgements
I extend my sincere thanks to the management of Usha
 (IJCNS) International Journal of Computer and Network Security, 81
Rama College of Engineering and Technology,
Telaprolu, Krishna(D.t), Andhra Pradesh, INDIA for
giving me the permission to use the website
http://www.usharama.com to conduct the case study
and I also thank the staff and
colleagues for their cooperation in completing the
work.
References
[1] B.Vijaya Babu, Prof M.S Prasad Babu “ Performance
Evaluation and Comparative study of Web Archiving
Tools “ International Journal of Computer Engineering
and Information Technology( ISSN 0974-2034), Volume
01,Number 01,page 100-106, Nov 2008-Jan 2009.
[2] Rui Cai, Jiang-Ming Yang, Wei Lai, Yida Wang, And Lei
Zhang 2008. iRobot: An Intelligent Crawler for Web
Forums, In the proceedings of WWW 2008 / Refereed
Track: Search – Crawlers, April 21-25, 2008 · Beijing,
China.
[3]. Ricardo baeza-yates, Aristides Gionis, Flavio p. Junquira,
Vanessa Murdock, Vassils Plachouras and Fabrizio
Silvestri 2008 Design Trade-Offs for Search Engine
Caching, In the ACM Transactions on the Web, Vol. 2,
No. 4, Article 20, Publication date: October 2008
[4]. Seung Hwan Ryu, Fabio Casati, Halvard Skogsrud,
Boualem Benatallah And Re´Gis Saint-Paul 2008
Supporting the Dynamic Evolution of Web Service
Protocols in Service- Oriented Architectures, In the
ACM Transactions on the Web, Vol. 2, No. 2, Article 13,
Publication date: April 2008.
[5] http://www.curl.haxx.se.
[6] Robert C. Miller and Krishna Bharat SPHINX: A Framework
for Creating Personal, Site-Specific Web
Crawlers . Proceedings of the Seventh International
World Wide Web Conference (WWW7), Brisbane,
Australia, April 1998 and Printed in Computer Network
and ISDN Systems v.30, pp. 119-130, 1998. Brisbane,
Australia, April 1998.
[7] Web Archiving-Julien Masanès, Springer- ISBN-10 3-
540-23338-5,Verlag Berlin Heidelberg 2006, ,European
Web Archive.
[8] http://www.httrack.com
[9] Junghoo Cho, Hector Garcia-Molina, Taher Haveliwala,
Wang Lam, Andreas Paepcke, Sriram Raghavan, And
Gary Wesley 2006 “ Stanford WebBase
Components and Applications “ ,In the ACM
Transactions on Internet Technology, Vol. 6, No. 2,
May 2006, Pages 153–186.
[10] Julien Masanès, “Towards Continuous Web Archiving“.
D-Lib Magazine, Volume 8 Number 12, December 2002.
[11] www-iglib.stanford.edu
[12] DAVID—Archiving Websites , Version 1.0, Antwerp -
Leuven, July 2002,
[13] http://freshmeat.net/projects/webeater/
Vol. 2, No. 1, January 2010
[14] Fitch, K. Web site archiving: an approach to recording
every materially different response produced by a
Website.” AusWeb 2003: the Ninth Australian
World Wide Web Conference, Hyatt Sanctuary Cove,
Gold Coast, Australia, 5-9 July 2003.
http://ausweb.scu.edu.au/aw03/papers/fitch/
[15] Tarmo Robal “ Agile Web-Crawler : Design
and Implementation, 2007
http://www.scribd.com/doc/100903/Agile-webcrawler-
design- and-implementation
[16] Vladislav Shkapenyuk and Torsten Suel “ Design
and Implementation of a high performance Distributed
Web crawler”
[17] Allen Heydon and Mark najork “ Mercator : A
Scalable, Extensible Web Crawler
http://www.mias.uiuc.edu/files/
tutorials/mercator.pdf
[18] Paulo Boldi, Bruno Codenotti, Massimo santini and
Sebastino Vigna “ Ubi Crawler : A scalable fully
distributed web Crawler, 2003
http://eprints.kfupm.edu.sa
[19] Shalin Shah “Implementing an Effective Web Crawler
“eInfochips, Dash Board September 2006.
Authors Profile:
Prof. B.Vijaya Babu received B.Tech
(ECE).,and M.Tech(Computer Science).,
degrees from JNTU, Hyderabad in 1993 and
2004 respectively. Presently he is working as
Professor and Head of CSE/IT Departments
in UshaRama College of Engineering &
Technology, Telaprolu, Near Gannavaram,
Krishna(D.t), Andhra Pradesh, INDIA.
During his 16 years of experience in
teaching and research, he attended many National and
International Conferences/ Seminars in India and also contributed
no. of research papers to various International journals.
Prof. Maddali Surendra Prasad Babu
obtained his BSc., M.Sc., M. Phil., and Ph.D.
degrees from Andhra University in 1976,
1978, 1981and 1986 respectively. He was the
Head of the Department of the Department
of Computer Science & Systems
Engineering, Andhra University from
2006-09.During his 30 years of
experience in teaching and research, he
attended about 30 National and
International Conferences/ Seminars in India and contributed
about 60 Research papers either in journals or in National and
International conferences/ seminars. He received the ISCA Young
Scientist Award at the73rd Indian Science Congress in 1986.