ONLINE TESTS EDITING BOARD
RO
EN
×
▼ BROWSE ISSUES ▼
Issue 57

How to use Squish Coco to determine code coverage for automated tests

Ioana Luțaș
QA Engineer @ Bissoft
TESTING

At some point, one may be interested in finding out how well-developed the automated test cases which validate the Web Services functionality actually are. In other words, it would be good to know how much the test cases hit the code. One way to test the Web Services functionality is by using SOAP UI.

This article describes how we can use the Squish Coco application to determine how much the SOAP UI test cases impact the C# code.

Short introduction of Squish Coco

Squish Coco is a coverage tool for C, C#, C++ and Tcl code. Using the instrumented source code, Squish Coco analyses the source code of the application under test.

Executing a test suite against an instrumented application produces data that can be analyzed later. This analysis can be used to understand how much of the source code has been hit by tests, which additional tests need to be written, or how the test coverage changed over time.

Squish Coco may give a measure of how much of the code is tested.

Squish Coco consists of two main tools:

A. CoverageScanner - This is a compiler wrapper which feeds instrumented source code to the native compiler. When using SquishCoco, it will tell the build tool to use CoverageScanner as the compiler. CoverageScanner is a C++ program that—in effect—replaces the usual compiler.

B. CoverageBrowser - This is a GUI tool for visualizing and analyzing code coverage based on the reports generated by running the test suite on an instrumented version of the application.

Squish Coco preprocesses the source code using the native preprocessor program, during which it inserts instrumentation code and, at the end, it compiles and links the project file, in a similar way to a normal build procedure. After the code is instrumented, CSMES files are generated - these files contain the instrumented code.

CoverageScanner's principle of code coverage analysis is not to highlight executed source lines, but rather to place marks on execution paths.

CoverageScanner parses all C++ / C language constructs and detects: executed functions, executed source code lines, execution paths, Boolean expressions which produce different execution paths (if, for, while, switch . . .).

After the detection phase, CoverageScanner inserts the instrumentation statements into the code. This modified version of the application is called instrumentation.

When the application is tested, coverage results are generated for each execution (.CSEXE file). These results may be analyzed using CoverageBrowser.

Instrumentation types:

  1. Line coverage - Instrumenting the execution of every executable source code line

  2. Branch coverage - Instrumenting the execution of each branch block (e.g., the body of any if statement).

  3. Decision coverage - Instrumenting each Boolean decision for loop and selection statements (e.g., record both the Boolean expression itself—true or false—and the body of the while, for or if statement).

  4. Condition coverage - Instrumenting of each sub-expression of Boolean expressions

Squish Coco is a multi-platform tool:

Coverage reports may be generated in the following formats: HTML, XML, EMMA -XML, text, Junit, Cobertura.

Squish Coco stores information per individual test, with an optional name, with FAIL vs. PASS status assigned and with additional comments. This way various types of analysis can be performed in the post-processing phase:

Safety standards and regulatory bodies mandate the use of code coverage analysis to ensure a proper degree of testing:

In most cases coverage is achieved through the use of a unit testing framework. Squish Coco will be able to gather coverage information from whatever framework is being used. It has integrated versions available for frameworks:

In addition, tests can also be driven through automation tools.

Coco can easily be integrated with various build and test frameworks to serve as a component of Continuous Integration (CI) system, maybe integrated with Jenkins, SonarQube, Bamboo or an in- house build system.

How to get code coverage statistics when testing Web Services using SOAP UI

SOAP UI is one of the SmartBear Software applications that can be used to test Web Services.

Since we are instrumenting services that do not terminate, we need to write our own handler which triggers the saving of the .csexe file.

1.Install Squish Coco 2.Set the Environment variable PATH to have the .Net framework location: C:WindowsMicrosoft.NETFramework64v4.0.30319 3.Copy the WebApplication folder. From C:Program Files (x86)MSBuildMicrosoftVisualStudiov11.0 To C:Program Files (x86)MSBuildMicrosoftVisualStudiov12.0

4.In the project to be tested, let's call it the NAS Project, in the Common Service for example, add the following two methods, for starting the Coverage Scanner and saving the Coverage report.

/// Service added just to begin the coverage process
public void StartCoverage()
{
  //var a = "asdasdsdasda";
  #if __COVERAGESCANNER__
  CoverageScanner.__coveragescanner_init();
  #endif
}

/// Service added just to stop the coverage process
public void StopCoverage()
{
  //var b = "asdasddsadassadsad";
  #if __COVERAGESCANNER__
  // set the filename of the .csexe file. 
  // NOTE: the location must be writable. 

CoverageScanner.__coveragescanner_filename(@ 
"C:\SCoco_res\NAS_SCoco_CodeCoverage_results.csexe");
CoverageScanner.__coveragescanner_save();
  #endif
}

5.When Squish Coco is installed on a particular machine, the Add-on for the Visual Studio is also installed. In Visual Studio we may find the option for Code Coverage in the Tool menu.

We select Tools ->Cove Coverage, select the coverage options, select a build configuration, select all projects, and set Enable Code Coverage.

Rebuild the NAS solution, so that project gets configured to use the Squish Coco Coverage Scanner. In the Build options for NAS Services, we may notice the following options:

6.Modify the SOAP UI test suites so that:

7.Follow the steps from the following schema in order to obtain the code coverage report for the executed SOAP UI Tests.

8.First and second steps, from the above schema, may be put in a .bat file

CodeMain NAS_SCoco_CodeCoverage_instrumented.bat

Some parts may be excluded from the instrumentation, as they don't have to be analyzed for code coverage, like:

/Code/Main/NAS.Vendors.Data.Repository/Mappers
/Code/Main/NAS.Vendors.Domain.Manager/Mappers

The content of the .bat file - NAS\SCoco\CodeCoverage\instrumented.bat :

REM build the NAS project with SQUISH COCO Enabled; REM exclude specific folders and files
REM build the NAS project with SQUISH COCO Enabled; REM exclude specific folders and files
cd C:\Workspacenew\Code\Main

set PATH=%SQUISHCOCO%\visualstudio;%PATH%

set COVERAGESCANNER_ARGS=--cs-on 
--cs-exclude-path=C:/Workspacenew/Code/Main/NAS.Vendors.Domain.Manager/Mappers 
--cs-exclude-path=C:/Workspacenew/Code/Main/NAS.Vendors.Data.Repository/Mappers
msbuild /p:UseEnv=true NAS.sln /t:ReBuild

REM copy all .csmes generated files into NAS.Services.REM Host\bin

for /f %%G in 
  ('dir C:\Workspacenew\Code\Main\*.csmes /s /b') do copy %%G "C:\Workspacenew\Code\Main\NAS.Services.Host\bin"

9.From the above schema, step 7 - Generate HTML code coverage report - may be executed from a bat file: NAS\Generate\SCoco\CodeCoverage\Report.bat

REM create Squish Coco html report

REM create Squish Coco html report

cd C:\SCoco_res

copy "C:\Workspacenew\Code\Main\NAS.Services.Host\bin\NAS.Services.Host.dll.csmes" "C:\SCoco_res"

cmcsexeimport --csmes=NAS.Services.Host.dll.
csmes --csexe=NAS_SCoco_CodeCoverage_results.csexe --title=NAS_SCoco_CodeCoverage_Result --policy=import_duplicates_and_empty

cmreport 
  --title="NAS_SCoco_CodeCoverage_Report" -m NAS.Services.Host.dll.csmes 
  --html=NAS_SCoco_CodeCoverage_Report.html

10.Here are some examples of code coverage results when some SOAP UI test cases pass, and some fail. Because the test cases fail, some parts of the source code are not executed anymore.

11.Code Coverage HTML Report

Sponsors

  • ntt data
  • 3PillarGlobal
  • Betfair
  • Telenav
  • Accenture
  • Siemens
  • Bosch
  • FlowTraders
  • MHP
  • BCR
  • Itiviti
  • Connatix
  • UIPatj
  • MicroFocus
  • Colors in projects