Quality Engineering Helps You Clean Up Your Act

By Roger Nessier, VP, Client Management, Estuate, Inc. - January 14, 2013
Trends such as cloud based software delivery and emphasis on uptime create even greater demands on software quality. Independent software vendors (ISVs) and software-enabled businesses must forego their usual pre-release quality control approaches and instill quality principles into every aspect of product development, from whiteboard to long term maintenance. Users no longer accept the tradeoff of whiz-bang technology for buggy, hard to use software. A recent example of this was the release disaster of Apple Maps, from a company which normally produces high quality innovative software. Software providers, software enabled businesses, must focus on high quality, secure, and reliable deliverables. The question is how will they address this? Software development organizations must respond to this changing market dynamic by placing a greater emphasis on quality engineering (QE). QE imbues quality principles into every aspect of the software product development lifecycle (PDLC), as well as technology and architecture decisions. The impact of taking a QE approach to software development can be huge. Catching process or design flaws early on reduces the impact and cost of remediating those issues as you get closer to the release. The most mature QE centric development organizations clearly understand the impact of quality on their PDLC process and software design. They use this information to ensure that final deliverables are of the highest quality. For some organizations this can be a big cultural shift where architects and developers were the power players. In some organizations, QA is a place where those who aspire to development roles attempt to prove themselves, or those who cant cut it as developers are relegated to lesser roles. In a QE centric organization the Quality Engineer is an equal to the architect or developer. In communicating this change, quality should be regarded as the mutual goal for both the developer and quality engineer. Teamwork is the hallmark of a QE organization. The quality engineer should work to help development engineers preemptively identify issues and address them early in the development cycle. The quality engineer needs to be a highly capable engineer that combines the problem solving skills of a developer with the eye for quality of a QA engineer. The first step in adopting a QE centric approach to development is to understand your Quality Engineering Maturity Level (QEML). QEML is a framework we have developed to help you better understand where your development organization is in QE adoption and what level of QE adoption your organization aspires to achieve. Once you understand your QEML, you set incremental QE goals over times that are supported by business benefits. What is your Quality Engineering Maturity Level? Feedback mechanism from stakeholders to refine design/process on an ongoing basis. Strong reliance on metrics, surveys and interviews to identify quality trends. Use of trend data to take action to optimize technology, design and process decisions to meet the changing needs of stakeholders. Leverage predictive mechanisms to better understand weak

links. Recognition that quality extends beyond catching defects early to include QE as part of design and development processes. Every aspect of development is analyzed and scrutinized to ensure it optimally aligned towards achieving quality goals. Technology, architecture and design decisions are made with an eye on quality. Stakeholders are tightly integrated into design and process decisions to ensure their needs are met. QE oversight to processes ranging from requirements gathering to long term support. Empower QE to stop the line should defect trends show a need to reassess design, process or technology decisions. Extensive use of metrics to understand quality trends. Culture of quality is instituted. QA and development activities are quite coordinated through the PDLC. No large backlog of bugs. Test cases derived from use cases and user stories early in development cycle. Developers responsible for unit testing. Limited use of metrics to better understand problem areas. Throw it over the wall pre-release QA, little emphasis on unit testing, test cases developed late in development cycle, releases regularly go out with several S1/S2 defects. Heavy reliance on point

releases to improve product quality. Development organizations should assess their QEML from two perspectives: Process and technology. It is possible to rank differently on the QEML scale from a technology and process standpoint. For example, you may have excellent development processes with strong stakeholder involvement, but are limited to how well you are able to serve your stakeholders because your applications are built on an inflexible architecture. Conversely, your application may be built on a modern web framework, but, your development processes are immature. Once an organization understands their technology and process QEML, then it sets goals over time for increasing their QEML. QEML goals should be supported by business cases that have strong linkages to financial goals. For example, if no severity 3 defects in releases doesnt have a business justification because the application isnt used that often and users are tolerant of severity 3 defects, then that should not be a process QE goal. Organizations need to develop separate technology and process QE plans for each product if those products are developed using different technology and development processes. It is usually recommended to make process QE improvements before making technology improvements, since your development organization will ultimately benefit from those process improvements when making technology changes. It not recommended to try to conduct both technology and process changes concurrently, or to try to move too rapidly since the shock of these changes could impair your organizations ability to keep the lights on. In the example below, a development organization has decided to revamp the development process for Product A first (Step 1). Once the process changes are completed, the value of the changes will be assessed, tied to business benefit, before deciding to proceed with Step 2, improving the technology platform.

Development organizations stand to derive the greatest benefit from integrated, holistic process to QE. If a big bang approach to Process QE is too daunting, or stakeholders need convincing that there is a business benefit to Process QE, incremental or point adoption can also be leveraged. The following are process QE adoptions you may wish to consider. Process QE Test Regime: Use Cases/Test cases should be developed at inception and applied early in the development cycle. Make sure that use cases/test cases are comprehensively documented. With the aid of use cases, comprehensive unit tests should be created and leveraged to ensure developers understand the business issue being addressed and to catch bugs or misunderstandings early in the development cycle. When possible, peer reviews should be leveraged to identify issues. Automated testing reduces testing overhead and may help uncover more issues. A heavy reliance on testing early and frequently eliminates issues before they become serious and expensive to fix. Requirements Traceability: Trace completed requirements back to business needs to ensure the need is met. Iterative and complex developments sometimes lead to the original product mission getting changed or lost. If requirements evolve, then it is important to update business requirements to reflect this evolution. The completed software product and its subsequent releases must always be in sync with the business requirements document. Use Defect Analysis to Identify Root Causes: Finding clusters of defects may point to a particular root cause such as a design flaw. Developers should investigate issue clusters and not just focus on fixing observable defects. Defect analysis should be conducted to track the number and source of defects throughout the development cycle to identify trends and take early action if an endemic root cause is uncovered. Advanced techniques such as Taguchi

methodology identify applications areas that are prone to defects and helps focus testing activities to those areas. This results in fewer test cases to achieve full application coverage. Adopt Agile Processes: Adopting an Agile development approach can help you test drive a QE approach. Agile by its nature uses smaller teams and requires close collaboration across all disciplines. Each Agile team almost by definition results in a QE approach although its restricted to that team and whats being accomplished in a given iteration. Applying any Agile approach (be it SCRUM, XP, FDD) to development helps build upon early successes and validates design assumptions. Instead of trying to release multi-faceted complex functionality, start simply and leverage market and stakeholder feedback to continually refine and enhance functionality. By building it simply the first time, you eliminate unneeded complex functionality that can detract from the user experience and introduce more issues. Only add in extra capabilities after you feel you have a strong justification to do so. Increase Focus on PSR Testing: Comprehensive performance, scalability, reliability (PSR) testing uncovers weak links in the product. PSR should cover all possible uses of the product, using different configurations, hardware/software environments, and data sets. PSR testing should help drive coding standards to ensure that as new functionality is added, PSR does not degrade. Document Processes to Enhance Repeatability and Consistency: Great software products are not created unless development processes are well documented and followed. Whether you follow a classic waterfall or an agile methodology (or something in between), it is important to have total process clarity from the requirements stage to end of life for a software product. Good process understanding is especially important if you have distributed development teams. Always look for opportunities to improve the process, and be sure to update process documents with those improvements. Leverage metrics to understand how well the team is tracking to expectations around productivity, quality and on time delivery. Report on those metrics using a real time dashboard that provides total transparency on the state of a release.

Deciding on QEML process goals depends on a variety of factors: Are you taking an integrated on incremental approach? What are the cultural, educational, and business challenges you are facing that could impede adoption? What are the perceived benefits (business case) for QE process adoption? What is your expected ROI? How will you measure success? Technology QE Development organizations are often faced with supporting and/or extending several generations of applications from Cobol to Java or .NET. Disparate technologies, design and skill requirements all

present challenges to a QE centric development organization. One of the biggest questions many organizations face today is: How can I afford to rewrite legacy applications to leverage the latest thinking in architecture/technology, while continuing to meet the needs of the marketplace? Any technology QE strategy must be balanced against market and business realities. Like process QE, many organizations take an incremental approach to technology QE, this can sometimes feel like changing the engine in your car while driving down the freeway. Like process QE, technology QE must be substantiated by business benefits. For example, a development organization doesnt adopt SOA because it seems like the technology savvy thing to do, or because developers like to have SOA expertise on their resume. SOA adoption should be justified by a strong business case. In some cases, SOA may not make sense for your organization. The following are technology QE adoptions you may wish to consider. Reduce Design Complexity: Eliminating complexity and encouraging reusability can help simplify large system design, thus reducing quality challenges. Designs incorporating SOA , object orientation, and partitioning, are more extensible and able to adapt as the product evolves. Avoiding a hard design commitment is preferable if an abstraction serves the same need. Re-factoring code early in the development process will reduce complexity and help uncover flaws. Designing for ease of integration to other products/systems by encapsulating variation reduces 3rd party software integration/testing challenges. Keep external interfaces as stable as possible when changing internal application functionality. Improve Usability/Interactivity: Poor usability/interactivity should be considered a design defect. A rushed, poorly researched approach to UI design will result in users logging issues. These issues cannot be ignored just because the product was built to spec and is supposed to have a user interface that is unintuitive. Be cognizant of how the product is used, leveraging use cases and story boards to validate the final design. Take into consideration the different needs of casual users and power users when analyzing usability/ interactivity. Make sure UI designs are intuitive, self-explanatory, and self-documenting. Design for Maintainability: If fixing parts of the application causes an undue amount of collateral damage to other parts of the application, there may be issues with the design. Engineers need to look at memory usage, designs and data access to understand if there are fundamental flaws causing the issues. As well, if porting the application to different platforms/environments, causes an undue amount of functionality to break, there may be issues with the design portability that needs to be addressed. Poor maintainability is sometimes caused by developers who feel they can walk away from a software project after it is released. Developers should realize they their team (or a subset of the team) are responsible for bug fixes and improvements in future releases. Design for Extensibility: If adding functionality causes an undue amount of existing functionality to break, there may be issues with the design. Revisit the product architecture as a product evolves to ensue it meets the needs of product. The adding of new functionality can be constrained by an architecture design that does not anticipate the product evolution. Poor extensibility is sometimes caused by a project view towards software development, vs. a long term multi-release view. Take Security Seriously: Proactively identify security holes and look for vulnerabilities before users stumble on them. Conducting a thorough security audit as part of the release process not only uncovers security flaws, but also helps refine coding standards so designs are more secure. Make sure your security goals are aligned with the type of application you are developing/supporting, and how exposed the application is to hackers.

Deciding on QEML technology goals depends on a variety of factors: Are you rewriting all your applications or taking an incremental approach? What are the training and business challenges to making technology changes? What are the perceived benefits (business case)? Do your customers really care if your application is written in Java or C++? What is your expected ROI? How will you measure success? The QEML planning process should be sequential and iterative. Checkpoints and metrics are used to ensure value is being generated. The chart below describes the steps your organization should use to set QEML technology and process goals.

Finally, instituting a culture of quality is critical across the entire company, not just the R&D organization. Develop training programs and make sure stakeholders in the rest of the organization understand the QE mandate of the development organization. For example, sales or management should not over commit development, resulting in death march releases and the resultant quality degradation. Professional Services teams should create customizations or configuration changes leveraging the same QE principles that development uses. Creating a culture of quality that permeates

throughout the entire organization will return dividends in the form of higher customer satisfaction, greater ability to meet market needs, greater customer retention, and ultimately, a healthier company.

Roger Nessier

Author

Roger provides product development services to software companies and companies that use software in their business. He has delivered dozens of projects to a range of clients around the world, using global teams, from gathering initial requirements, to design/build, to long term support. He is currently VP of Client Management at Estuate, and has held services management and product development executive positions at i2 Technologies, Symphony Services, and Steelwedge Software. Roger lives in the Bay area. In his spare time, Roger enjoys running or cycling the hills near his home, or taking a yoga class with his wife.