A traditional software development model defines software quality assurance as the “set of systematic activities providing evidence of the ability of the software process to produce a software product that is fit to use” [G Gordon Schulmeyer and James I McManus, Handbook of Software Quality Assurance]. Such systematic activities have been extensively researched, fine-tuned and implemented through various models. The classic statement when it comes to Open Source projects is that “Given enough eyeballs, all bugs are shallow“. The accepted implication in that statement is that the software will not be free of defects at General Availability and mandates community participation, collaboration and feedback in improving the software.

From “A Survey on Quality Related Activities in Open Source” (which is available from the ACM SIGSOFT as Software Engineering Notes vol 25 no 3) by Luyin Zhao and Sebastian Elbaum, the conclusions or lessons learned are interesting:

* the average application spends approximately 40% of its lifecycle in the testing stage. However, 80% do not have a test plan.
* testing efforts are limited before release because open source people tend to rely on other people to look for the defects in their products. However, not all the code developed under the open source model has been revised by a large number of qualified reviewers
* 28% of defects are on interfaces. The dynamic nature of open source and the close interaction between developers and users might be a factor in this defect distribution
* application attributes have a greate impact on quality assurance activities. They must be taken into account when performing quality assurance tasks
* inspections are more popular within larger systems
* there are not enough testing tools available for web applications

Jeffrey Jaffe writes about bringing more applications to linux and earlier about the ISV ecosystem but his thoughts seem to be more in terms of a lowering of barriers for the ISVs through judicious usage of standards. That among other things would be some of the relevant points raised at the Software Testing BoF at foss.in.

An objective of testing software is to ensure that the GA product is reasonably defect free. That said, the objective of a “tested and certified” software should be to provide an assurance to the end user who is planning to deploy the application on an operating system (and perhaps middleware) for production operations. Such assurance includes assurances of stability of the “stack” and quality. Not surprisingly, the notion of quality would be rooted in the economics of operations of the end user of the application. “Quality is a complex and multifaceted concept” (D A Garvin, What does “product quality” really mean? MIT Sloan Management Review, 26(1):25-43,1984) and hence it can only be perceived as a combination of the various properties of the product on the economics of the product (and hence the company doing the software project that produces the product). For the end user, one of the concern areas around the economics of “certified” applications is the “cost of maintenance effort” ie the cost aligned with maintaining the deployment of the application as related to costs incurred due to downtime of the system due the application. The IEEE Standard Glossary of Software Engineering Terminology defines maintainability as “the ease with which a software system or a component can be modified to correct faults, improve performance, or other attributes, or adapt to a changed environment“.

Certification of a software is aligned with the stated quality goals of the company that produces the software and the company that tests and certifies it (sometimes they are the same company) and thus it has an economic currency attached to it. A simplistic notion of “producing certifiable software” would be to “develop software according to standards”. While standards by themselves are reference points around which development can be undertaken, not all standards can actually be converted into implementation ready tools/widgets/applications. For example, while it might be reasonably easy to test an application against something like LSB, integrating LSB framework during the software development process to ensure that the developed software is “certification ready” would be a bit difficult. Current models of certification and testing of applications are reactive – they test once the software is ready and GA-ed. Software development (irrespective of traditional or open source) models are on the other hand iterative through phases. The need would be to put in place guidance tools based around the standards which can be integrated by the software development companies into their software development processes so as to ensure that each and every build can actually be “certified” aka quality assured. This notion of quality is, in a way, an echo of Philip Crosby’s definition of quality (P B Crosby, Quality is Free: The Art of Making Quality Certain) as “conformance to requirements”. Such requirements include elements like the code quality, flexibility, maintainability, portability, re-usability, readability, scalability, reliability.

Thus, the approach to Software Testing in the light of Software Certification has a few concern areas that needs to be looked into:

* creating of tools to test various kinds of software (and not only web applications)
* creating of tools that would provide guidance to software developers in terms of compliance with known standards
* collaborating across a broad spectrum of software technologists towards ABI stability across operating system releases

the price of failing on any or all of the above would result in “software application hell” on operating systems whereby the effectiveness or an efficiency of the product can be compromised. A small component of the way forward is to look at it from the viewpoint of an independent software developer. A larger component of the roadmap is to ensure that there are measurable quantities which can be used by software developers to put metrics to quality and hence to cost during the development phase. Software quality might be quantified through software error data. The factors (as suggested by Kenneth S Mendis, Quantifying Software Quality) that allow putting metrics to software quality could be:

* error by category
* error density
* error by type
* error by severity
* error arrival rate
* error reporting

The generally accepted principles of software testing are:

* test process, test cases and test plan
* techniques, methodologies, tools and standards
* the culture of the organization

The above principles should enable a software developer (or an ISV) to be looking into the feasibility of “software certification” and increase in software quality. What would be good to have at this stage is an attempt at formalization of the technology of certification which would allow the development of methodologies around it. The Certification Service would then be meaningful to have in terms of providing an index of stability and reliability in the software industry aka ecosystem.