You are on page 1of 6

Tracking Hardware Evolution

Author Name1, Author Name2, Author Name2


1
Company One, 123 Main Street, Cincinnati, OH USA
2
Company Two, 42 Gerbrekarstrasse, Brussels, Belgium
1
E-mail: name@domain.com

• Bug Tracking Systems (BTS): Store information


Abstract
about all reported software bugs. Popular open-
Software evolution is the term used to describe the
source systems are Bugzilla [6] and Mantis [7];
process of developing and updating software systems.
Software repositories such as versioning systems, and bug • E-mail lists or software documentation.
tracking systems are used to manage the evolution of In the last years, software evolution research focused
software projects. Mining this information is used to support mainly on [8]: estimation of developer effort, impact of
predictions and improve design and reuse. Integrated circuit changes in source code and defect prediction. In order to
development can also benefit from these techniques. facilitate software evolution research, several frameworks
Nowadays, both software and hardware development use were proposed [9]. These frameworks are suitable to track
repositories and bug tracking systems. There are many historical and software complexity metrics, but unfortunately
hardware open source projects such as SUN's OpenSparc do not offer support to extract and visualize Hardware
and designs at Opencores.org. We propose a methodology to Description Language (HDL) specific metrics.
track specific HDL metrics in order to improve design Software evolution techniques can be used to
quality. Our results present a case study that correlates HDL complement and deeply improve hardware verification
metrics to bug proneness of Verilog HDL modules. We also methodologies. For example, large industrial System-on-a-
present EyesOn, an open source framework designed to chips (SoCs) often reuse IP (Intellectual Property) blocks.
automate historical and complexity metrics tracking of HDL Many high-value IP blocks are developed over long periods
projects. of time, being expected to live on for a decade [10]. Such
systems are designed by large teams, generating repositories
Keywords that contain millions of RTL lines of code and thousands of
HDL, design flow, quality, metrics, validation bug reports. Open source hardware also plays an important
1. Introduction role on making hardware evolution research possible.
Verifying industrial scale designs is still a challenging Designs like SUN’s Opensparc [11] and OpenRisc [12] offer
task that consumes most part of total development efforts. repositories that have plenty of information about how these
International Technology Roadmap for Semiconductors designs where implemented and verified.
reported that verification stage is a bottleneck that has Tracking hardware evolution is a new paradigm that
actually reached crisis proportions [1]. Considering that offers many advantages. As happens in software
there is no formal or simulation tool capable to stimulate all construction, hardware development can also benefit from
possible states in an industrial scale design, verification team hardware evolution. It allows hardware bug prediction,
must work smarter, wisely spending their resources. In order mitigating hardware development complexity and
to handle the still increasing design complexity, verification issues. Furthermore, it may provide hardware
improvements in current verification methodologies are module complexity profile, facilitating development
needed. A step in this direction is to benefit from software management.
engineering experience. Nowadays hardware and software The contributions of this work are threefold. First, we
development methodologies have many aspects in common: propose the use of information available in HDL repositories
requirements specification; use of description languages; to track hardware evolution. Second, we provide EyesOn, an
identification of functional design errors; use of versioning open source, extensible framework designed to extract, store,
systems and bug tracking tools. visualize, and export HDL complexity and historical metrics.
Software evolution is a mature and well established Third, we present scenarios where hardware evolution
research area in software engineering [2,3]. Software tracking is inserted in traditional integrated design flows,
evolution research relies on extracting and processing providing accurate and visual information.
information from: This paper is outlined as follows. Section II reviews the
• Version Control Systems (VCS) repositories: These main software evolution tracking tools. In Section III, we
systems store all revisions of the software source present the methodology to track hardware evolution, while
code. Freely available platforms are Subversion in Section IV details of the framework are provided. In
(SVN) [4] and Concurrent Version System (CVS) Section V some hardware evolution key visualization
[5]; possibilities are presented and a case study is discussed.
Finally, in Section VI, we conclude our remarks and present
future work directions.
1-xxxx-xxxx-4/10/$25.00 ©2010 IEEE 12th Int’l Symposium on Quality Electronic Design
2. Related work While these tools have their own metrics and capabilities,
While software evolution has been studied for at least the EyesOn is the first to integrate environments for hardware
last three decades [13, 14], research on hardware evolution evolution research.
is still incipient. Therefore, there are no tools or frameworks
3. Methodology
designed to track HDL evolution. A preliminary work was
In this Section, we describe the methodology to track
proposed by [15]. In this work, some historical metrics and
hardware evolution. Basically, two different application
bug reports of HDL repositories are retrieved and stored on scenarios are described. These application scenarios are
a database. The main contribution of that work is to define a
illustrated on Figure 1.
language to describe bug reports. This language is used to
In the first scenario, hardware evolution tracking, the
attach bug reports on the commit messages, making possible
metrics are retrieved from the Repository and displayed on
to automatically retrieve and display this information. The
different formats that include:
most complete tools and frameworks designed process
repositories are legacy from software evolution research • Bar and line charts: present metrics evolution
community. The most importants are CVSAnalY, Release overcommits or periods of time;
History Database (RHDB), Kenyon, and Alitheia. • Kiviat diagrams: useful to compare the evolution of
Unfortunately, none of them are capable to handle HDL metrics on different axis and periods of time;
specific complexity metrics. • Heatmap diagrams: help the visualization of
CVSAnalY [16] is a repository data extractor. It was module’s specific metrics, such as modification rate
initially built to collect revision data from CVS repositories. and bug proneness.
They made GNOME project analysis as their case study and
presented statistics about inactivity, commiters, commits,
lines changed, lines modified, lines removed, first and last
commit. Based on these statics, inferences were presented
about modules and commiters.
RHDB [17] joined historical information from CVS and
Bugzilla, correlating their relationships. The experiments
were based on Mozilla web browser, a well known open
source project. Internal function calls were tracked and
mapped to call graphs, registering their evolution with
collected modification and bug reports.
Kenyon [18] is a software evolution tracker. It was
designed to facilitate software evolution research providing
an environment with repositories connectors, metric
extractors, and API points that could be extended. The
collected data is stored on a relational database through
Object-Relational Mapping.
Alitheia [19] tool aggregates similar functionalities, but
is designed to be a service oriented architecture and focused
in having good performance on large scale data extraction.
They are the most complete tools designed to facilitate
software evolution tracking. A detailed work comparing
different software evolution trackers is presented in [9].
The great number of metrics, the large scale of objects
being analyzed, and the context where they could be
Figure 1: Hardware evolution application scenarios.
observed require different viewpoints. RelVis [20] is a
visualization approach that presents multiple metric data
It’s important to note that hardware evolution tracking
about software entities and their relationships during the
can include also metrics extracted by commercial Electronic
software releases. It uses kiviat diagrams, also known as
Design Automation (EDA) tools.
radar charts, that are good to depict different values.
GEVOL [21] is a system that use graphs and colors to The second application scenario facilitates the use of
represent Java classes evolution and the contribution of their advanced data mining and statistics techniques. This is
developers. Different viewpoints like simple bar and pie where hardware evolution research starts, experimenting the
charts, and more sophisticated diagrams like network, matrix effectiveness of different software evolution techniques on
and cityscape views were presented in [22]. HDL hardware projects.

1-xxxx-xxxx-4/10/$25.00 ©2010 IEEE 12th Int’l Symposium on Quality Electronic Design


The most important function performed by Repository
Extractor is the first step of VCS preprocessing. During this
4. Framework architecture stage, repository revision information is retrieved and stored
Finding integrated environments for hardware evolution
in a database. The acquired revision information will guide
research is a key problem. Having that in mind, we designed
metrics extraction which is the second EyesOn preprocessing
and implemented EyesOn, an open source, extensible
step.
framework. Our main contribution allows extracting, storing,
visualizing and exporting HDL complexity metrics, 4.2. Metrics extractor
facilitating hardware development and verification. Our Metrics Extractor is the module responsible to extract
framework is organized as illustrated on Figure 2. and store different design metrics: HDL, External, Bug, and
Repository. With revision information
gathered by Repository Extractor it is
possible to observe project specific
state points on timeline.
4.2.1. HDL metrics.
In order to extract static metrics
from source code, the desired version
is retrieved on demand by Source
Extractor. A revision file and
directory structure is mounted on a
temporary file system path, providing
an environment that is a mirror of the
design’s evolution state.
Despite the fact that some metrics
are simple to implement, as occurs
with LoC (lines of code), others, such
as complexity and metrics that
quantify internal structures, depend of
HDL parsers.
EyesOn provides a rich framework with grammars and
Figure 2: Framework architecture. HDL parsers. The metrics taken are mostly based on source
There are two storage components, namely Database and code analysis. At this version, we developed HDL parsers
Repository. In addition, we have modules for data that extract:
manipulation (extractors, parsers, metrics, visualization,
• McCabe [23]: Also known as Cyclomatic
exporter).
Complexity, relies on the premise that source code
The architecture was designed to allow the integration complexity does not depend on its size, but in its
among repositories and metric extractors, facilitating the structure. It measures the number of independent
researchers work. During the hardware development process, program’s execution paths.
designers insert into the repository messages about every
• NPath [24]: Counts the acyclic execution paths
module revision. From those revisions, data are extracted by
through a function. This metric was originally
the Repository Extractor and inserted into the Database. The
proposed to measure maintainability effort.
framework provides support for researchers to retrieve
project files and extract desired metrics. Those can be • Halstead [25]: Composed by a set of complexity
visualized through the Data Visualization module, which measures derived from the amount of operands and
offers many options, such as bars, lines, heat map and radar. operators.
Through a GUI data visualization, it is also possible to build • Other simpler metrics like LoC, statements, I/O
queries and to exchange data. signals, operators, and operands.
Next, we explain the main modules and features 4.2.2. External metrics.
presented in the framework architecture. In order to increase EyesOn flexibility, it is possible to
4.1. Repository extractor include metrics extracted by external tools. Any commercial
Repository Extractor is a module that connects to the EDA tools offer statistics about design’s modules like flip-
most popular VCS tools (CVS, Subversion, and Git) and flops, equivalent gate count, finite state machines, latches,
retrieves data about the revisions of the HDL designs. registers, etc.
Although VCS tools store revision information in different
ways, key data like tags, commits, comments, and authors
are standardized.

1-xxxx-xxxx-4/10/$25.00 ©2010 IEEE 12th Int’l Symposium on Quality Electronic Design


4.2.3. Bug metrics. also the case in hardware evolution. We provide a set of
In addition, EyesOn is also capable of extracting bug data charts, allowing the data visualization in many formats.
from BTS and VCS repositories. Simple VCS transaction
4. Results
detection heuristics are provided based on similar commit
As a case study, we used revision history information
messages and intersections of VCS with BTS modification
from a proprietary IP core. This core was divided into 17
intervals.
modules that were developed during a semester. All the
4.2.4. Repository metrics. modifications on these modules were reported and stored at
Repository metrics are useful to contextualize the other the repository as commit messages.
metrics in time. This information provides support to study In order to observe when bugs were discovered during
the relation among different classes of events like commit or the development process, we built a bi-weekly chart depicted
bug reports. Examples of Repository Metrics are: in Figure 3. The bars represent the accumulated found bugs
• Commit comments and its type; by the time interval.
• Contribution by author; Figure 4 depicts extracted metrics changing as hardware
• Revision differences; designers commit new versions of Module #1. Keywords,
operands and Lines of Code present a similar behavior. One
• Transactions;
can notice variations when source refactoring occurs (at
• Coupling among modifications. commit intervals [9:11] and [17:21]). The Lines per
4.3. Extension points Comment values reduced as the module was being
Our framework extensibility relies on its capability to developed. It happened because some commented
adapt to new tools, metrics and to accept parameterized data instructions were also removed from source code as it was
from external applications. In order to support these new being implemented. Despite refactoring, the module
features we provided extension points for metrics,
repositories, and external extractors.
4.4. Flexible database
Despite making metric extraction more agile, other
motivation to maintain a database is to facilitate queries. Due
to the capabilities of Hibernate, an Object-Relational
Mapping (ORM) framework, different Database
Management System (DBMS) can be configured to store the
manipulated data. Hibernate Query Language (HQL) is used
to retrieve database information based on mapped classes.

cyclomatic complexity grew, but, at the end, diminished a


little. It could be an indicative for future modifications.
Figure 4: Metrics changing between commits.
A simplified view of metrics evolution is shown in Figure
5. This chart is composed by LoC, Cyclomatic Complexity
and Accumulated Fixed Errors. The values were normalized.
This visualization is fragmented on time, showing only three
evolution steps. They were composed of commits #13, #20
and #26, respectively colored by red, blue and green. It
presents the growth of these metrics.
Figure 3: Number of errors per time interval.
4.5. Data Visualization
The data visualization module is a GUI workspace that
supports users to build queries and to exchange extracted
data. The usage of plug-in metrics classes on query builder
are facilitated by Java language Reflection API that discover
available mapped attributes on new registered metrics.
Efficient data visualization to track files and software
evolution has become a specific research area [26]. This is

1-xxxx-xxxx-4/10/$25.00 ©2010 IEEE 12th Int’l Symposium on Quality Electronic Design


As future work, we intent to extract HDL metrics in order
to build HDL quality predictor models. We are also
improving the set of EyesOn supported metrics as well as
visualization options.
6. References
[1] “International technology roadmap for semiconductors
- design,”. 2009. [Online]. Available:
http://www.itrs.net
[2] H. Kagdi, M. L. Collard, and J. I. Maletic, “A survey
and taxonomy of approaches for mining software
repositories in the context of software evolution,” J.
Softw. Maint. Evol., vol. 19, no. 2, pp. 77–131, 2007.
Figure 5: Radar visualization.
[3] K. H. Bennett and V. T. Rajlich, “Software
maintenance and evolution: a roadmap,” in ICSE ’00:
Figure 6 shows a heat map that presents modules that are Proceedings of the Conference on The Future of
colored according to the error prone value. Green (light gray Software Engineering. New York, NY, USA: ACM,
in grayscale copies) means low error proneness, while read 2000, pp. 73– 87.
(dark gray) means high chance of error. Each module was [4] B. Collins-Sussman, B. W. Fitzpatrick, and C. M.
represented by a rectangle where its area is proportional to Pilato, Version Control with Subversion. O’Reilly
its LoC size. The adopted prediction model was based on the Media, 2004.
Most Frequently Fixed cache replacement algorithm [5] P. Cederqvist, Version Management with CVS. Free
presented on [28]. It was incorporated by EyesOn as an Software Foundation, Inc., 2004.
external application. As observed on the Figure, among 17 [6] The Bugzilla Guide, 2010. [Online]. Available:
modules, 3 were subject to high error prone. They are not http://www.bugzilla.org
necessarily the 3 largest ones. [7] Mantis Bug Tracker, 2010. [Online]. Available:
http://www.mantisbt.org
[8] M. DAmbros, H. C. Gall, M. Lanza, and M. Pinzger,
Software Evolution. Springer, 2008, ch. Analyzing
software repositories to understand software evolution,
pp. 37–67.
[9] G. Gousios, “Tools and methods for large scale
software engineering research,” Ph.D. dissertation,
Athens University of Economics and Business, jul
2009.
[10] W. Stapleton and P. Tobin, “Verification problems in
reusing internal design components,” in DAC ’09:
Proceedings of the 46th Annual Design Automation
Conference. New York, NY, USA: ACM, 2009, pp.
209–211.
[11] D. L. Weaver, OpenSPARCTMInternals, 2008.
[12] OpenRISC 1000 Architecture Manual, 2004. [Online].
Available: http://www.opencores.org
[13] “I J. P. Cavano and J. A. McCall, “A framework for
Figure 6: Radar visualization. the measurement of software quality,” in Proceedings
of the software quality assurance workshop on
Functional and performance issues, 1978, pp. 133–139.
5. Conclusion and future work [14] V. R. Basili, L. C. Briand, and W. L. Melo, “A
Tracking hardware evolution offers many benefits. It can validation of objectoriented design metrics as quality
improve the quality of electronic design. We proposed the indicators,” IEEE Trans. Softw. Eng., vol. 22, no. 10,
use of information available in HDL repositories to track pp. 751–761, 1996.
hardware evolution. We presented EyesOn, an extensible, [15] “Blind review.”
open-source framework designed to automate hardware [16] G. Robles, S. Koch, and J. M. Gonzalez-Barahona,
evolution tracking. We presented a case study where “Remote analysis and measurement of libre software
hardware evolution tracking were used to correlate HDL systems by means of the CVSAnalY tool,” in
metrics and bug proneness of Verilog HDL modules. Proceedings of the 2nd ICSE Workshop on Remote
Analysis and Measurement of Software Systems

1-xxxx-xxxx-4/10/$25.00 ©2010 IEEE 12th Int’l Symposium on Quality Electronic Design


(RAMSS), Edinburg, Scotland, UK, 2004. [Online].
Available:
http://libresoft.urjc.es/html/downloads/cvsanaly-
icse.pdf
[17] M. Fischer, M. Pinzger, and H. Gall, “Analyzing and
relating bug report data for feature tracking,” in WCRE
’03: Proceedings of the 10th Working Conference on
Reverse Engineering. Washington, DC, USA: IEEE
Computer Society, 2003, p. 90.
[18] J. Bevan, E. J. Whitehead, Jr., S. Kim, and M. Godfrey,
“Facilitating software evolution research with kenyon,”
in ESEC/FSE-13: Proceedings of the 10th European
software engineering conference held jointly with 13th
ACM SIGSOFT international symposium on
Foundations of software engineering. New York, NY,
USA: ACM, 2005, pp. 177– 186.
[19] G. Gousios and D. Spinellis, “Alitheia core: An
extensible software quality monitoring platform,”
Software Engineering, International Conference on,
vol. 0, pp. 579–582, 2009.
[20] M. Pinzger, H. Gall, M. Fischer, and M. Lanza,
“Visualizing multiple evolution metrics,” in
Proceedings of the 2005 ACM symposium on Software
visualization. ACM, 2005, pp. 67–75.
[21] C. Collberg, S. Kobourov, J. Nagra, J. Pitts, and K.
Wampler, “A system for graph-based visualization of
the evolution of software,” in Proceedings of the 2003
ACM symposium on Software visualization. ACM,
2003.
[22] S. Eick, T. Graves, A. Karr, A. Mockus, and P.
Schuster, “Visualizing software changes,” IEEE
Transactions on Software Engineering, vol. 28, no. 4,
p. 412, 2002.
[23] T. J. McCabe, “A complexity measure,” in ICSE ’76:
Proceedings of the 2nd international conference on
Software engineering. Los Alamitos, CA, USA: IEEE
Computer Society Press, 1976, p. 407.
[24] B. A. Nejmeh, “Npath: a measure of execution path
complexity and its applications,” Commun. ACM, vol.
31, no. 2, pp. 188–200, 1988.
[25] M. H. Halstead, Elements of Software Science. New
York: Elsevier, 1977.
[26] L. Voinea, A. Telea, and M. Chaudron, “Version
centric visualization of code evolution,” in Proceedings
of Eurographics/IEEE-VGTC Symposium on
Visualization. Citeseer, 2005.
[27] A. Hassan and R. Holt, “The top ten list: Dynamic fault
prediction,” in Software Maintenance, 2005. ICSM’05.
Proceedings of the 21st IEEE International Conference
on, 2005, pp. 263–272.

1-xxxx-xxxx-4/10/$25.00 ©2010 IEEE 12th Int’l Symposium on Quality Electronic Design