SPM Testing Template – Part 5

Incentive Compensation Management test results can be recorded in several ways. One of the approach I like to use groups all plan names, rule names, conditions, expected results and testing status on the same spreadsheet. I find that by keeping all this information together, it is easier to quickly get the picture of the overall testing progress. It also allows to keep all the information on the same spreadsheet instead of having to maintain 2 spreadsheets with identical information.

Here is an example to get started:

[download the spreadsheet]

Other benefits of using an Excel spreadsheet to record test results instead of a Word document include:

• Ability to quickly highlight failing tests in red
• Ability to filter information displayed (e.g. display only failing tests)
• Test results can be printed on fewer pages
• Ability to create macros to perform more “fancy” features such as displaying the number of days a certain issue has been opened.

Other columns could be added to add additional information such as the date at which the test was performed, the name of the tester, how critical the test is, the actual result when different from the expected result, and comments.

ICM Testing – Reviewing Results – Part 4

When test scenarios for an SPM application are defined properly, reviewing the results should be straightforward. However because of the volume of results typically generated, finding the correct result can be a very time consuming process without the proper skills (finding output of rule A in the 1 million results generated can be like looking for a needle in a hay stack, except that during testing, there are several needles that need to be found!).

Every ICM application I have implemented offer an interface to browse through the results. This is usually a GREAT tool to troubleshoot and to resolve issues, but it’s rarely the best and quickest tool to analyze results in bulk. Fortunately, every ICM application I have seen also have a functionality to export the results. Exporting the results will create a file locally which can then be opened by a program such as Excel or Access for further analysis. I will discuss how to use Excel to review the results, and later I will create another quick tutorial on using Access.

Step 1) Export the Results from your ICM System
Every system has its own way of doing this, but generally this should be doable from the interface and not requiring a database administrator to pull the data from the database (although this can also be done if required).

Step 2) Import the Data in Excel
First, open Excel and click on File->Open. Click on the dropdown menu beside “Files of type” and click on “All Files”.
Find the location where you saved the exported file of step 1 and click on “Open”.

If the file is a .CSV file, Excel will automatically place the data in columns. If the file is a .TXT file, more steps are required (see step 3).

Step 3: Opening a Delimited Text File in Excel
In the first Wizard step, choose “Delimited” and click on “Next”. In Step 2, select the character used to delimit each column (Xactly uses semi-colons). Click “Next” twice and the data will be imported in columns.

Step 4: Filtering the Result Data
At this point we have a worksheet with all of our result set displayed. Each of the column can be sorted and the search feature can be used to quickly find a specific order. However there is another trick which allows to quickly display data for a specific rule, result name, person, date, etc.

Click on Date->Filter->AutoFilter

This will make the first row (usually column names) dropdown menus which will contain all the values contained within the column. In the example below, the drop-down for column “Currency” allows to choose “USD” and “YEN”. Choosing USD will filter out all the other currency types.

Filtering can be done in this manner for multiple columns at the same time. For example you could filter a certain position (Julien Dionne), and further filter to display only the USD currency.

Quick Summation
Summing credit and commission amounts is one of the most frequent activity I perform when testing.

Excel has a built in feature which allows to quickly find out the sum of selected cells. By simply highlighting the cells containing a number or a dollar amount, the total value of these cells will be displayed in the bottom-right corner of the window.

Additional Notes

Excel offers more advanced functionalities such as Pivot Tables. They can be useful when reviewing a large volume of results. I will discuss this in more details in the future.

Excel Limitations

Excel cannot have more than 65536 rows. If the result set exceeds this, it will not get imported. It is possible to “workaround” this issue by only importing a subset of data (maybe by restricting the incentive period). MS Access could also be used instead of Excel and would not be subject to this limit. I will discuss using Access to review results in the near future.

ICM Test Planning, Scenarios and Templates – Part 3

In my last post about testing I said that ICM / SPM systems should be tested in phases; the reason for this is that discovering issues late in the development life-cycle could add unexpected delays and ultimately make the budget run over-budget.

It is surprising how often I have seen the business users and stakeholders not agree with the results and the development team exclaim, “Oh, that’s how it’s supposed to work!” or “That’s what you meant!”. Without proper planning, it is possible that during UAT the users will try to perform various activities and “break” the system – to which the development team usually answers “You’re not supposed to do that”, or “That’s not how it’s supposed to work”. Often, shortly before the payroll date, business users often ask questions such as “What about the draws?” or “What if orders have negative amounts” (good requirement documentation is also important!). Proper planning should eliminate all those nasty surprises.

There is an excellent article on Wikipedia regarding creating a test plan based on the IEEE 829 format. The test plan describes what will be tested, how it will be tested, what will be the deliverables, who will be responsible for what, etc.

Creating Test Scenarios for a Sales Performance Management System

I believe the most important aspect of testing is the test scenario preparation. I briefly mentioned how creating good test scenarios was particularly difficult with an ICM application because of the volume of test cases it will typically generate. This is unavoidable, but proper planning is required to ensure that tests are not testing the same conditions twice (wasting time) and that all conditions are being tested (not cutting corners).

A test scenario should have a name, a scenario ID and a description. This will help quickly refer to them during meetings. The test scenario should include the initial conditions, input (such as the order type), and the outcome or expected results. Finally, it is a good practice to list the business requirement ID that the scenario is testing. It is important for the test scenarios cover each plan, each rule, and each formula used within the rules. To test an ICM system, I like to group the scenarios, by Plan and by Rule:

1. Plan A

1.1 Rule A

1.1.1 Scenario 1

1.1.2 Scenario 2

1.1.3 Scenario 3

1.2 Rule B

…

Creating Test Data
The test data is the data that will be “staged” to test the scenarios. Typically, an order or a combination of orders will be required to test different scenarios. These orders should be created in the appropriate format to be staged and it should be documented which orders test which scenarios. Because of the different testing phases and because test data is often altered or corrupted during testing, it is important for the test data files to be kept together and be readily available to be re-staged when required.

After processing all the test data, all the test scenarios should be tested. That should cover the entire system, and in theory, after completing these steps, the system should have no outstanding defect.

ICM Testing – Part 2: Test Phases and Traps

ICM systems, as every other software system, should be tested in phases. Here are the major test phases (there can be more or less depending on the complexity and size of the project)

Unit Test: Performed by the developer to ensure a specific component performs as per the requirements. Unit tests should be performed as the system is being built.

System Test (ST): While unit testing was performed with only a single rule, system testing tests entire plans. It ensures the entire system performs as expected and as defined.

System Integration Test (SIT): An ICM system often relies on receiving data from other existing systems. For example, it may expect a file received via FTP, from an ETL process or fetch data directly from a CRM system such as SAP or Peoplesoft. SIT will ensure the systems interact the way they should.

User Acceptance Testing (UAT): Once the ICM system is fully tested by the implementers, UAT is when the business user or owner of the system test and review the results of the system. UAT should be performed on the production system, with production-like data.

Other very important tests which should be performed during ST, SIT and UAT include:

Performance Testing: This test confirms if the processing of commissions can be performed within the required window.

Stress Testing: This test verifies how the system behaves if the data load is higher than expected. For example, the compensation data may increase year over year, or may be higher during quarter end, year end.

Regression Testing: Every time a system is altered, there is a risk for a defect to be introduced. Relevant tests should be performed again to ensure the system still works as expected.

Each of those test phases are usually done in parellel to major development life cycle activities – the “V-model” above illustrates this. I will have to keep the development phases discussion for the future but yes, the left part of the “V” is also necessary.

When working with an implementation partner or vendor, the client is usually mostly responsible for the User Acceptance Testing which I will discuss in more details later.

What I must stress as being extremely important is that it cannot be taken for granted that the implementer will necessarily have a rigorous testing methodology and ethics. To avoid having big surprises when reaching UAT, the client should keep an eye on other testing activities and ask for regular status updates.

Another common trap is that getting regular status updates may not be enough. It is important for the project team to design and agree on the test cases and test plan early on. In larger implementations this is often a step which should get a sign-off. This will ensure the implementers know what the expected results should be, which is very different than the implementers assuming they know what these results should be.

YAWN… ICM Testing – Part 1: Introduction

Testing is a quality assurance process required to ensure the system works as expected and as defined. I never met anyone who found testing “sexy”, fun, or even interesting. However, it is essential!

No matter how well designed the compensation plans are, if the commission and bonus results are not correct, the implementation will be a failure. We discussed that it is important for payees to be able to correlate their rewards to their performance, but mistakes could prevent them from doing this efficiently. Another result of “bugs” or system defects will be an increased number of complaints from the payees which will consume the compensation team’s time. These errors will also add confusion, making sales people unhappy and ultimatly lead to a lost of confidence in the system.

Traditionally, testing was a task done only once the entire implementation was completed; you built the system, you tested it and you rolled it out. This is one of the best recipes for failure. Test planning should be done at the very beginning of the project and testing performed throughout the development of the system, until it is rolled out.

Testing an ICM system is particularly challenging because of the volume of different cases. A “standard” software system testing may have a test that if a button is clicked, an action will be performed. With an ICM system, every plan, every rule, and every formula should be tested. For example, if there are four quarters, each quarter using its own rate table, each rate table having 8 tiers, and each tier corresponding to a formula, there will be 32 individual formulas to test – 32 test cases.

In my next posts I will discuss many testing related topics including the different test phases, how to create proper test cases and test plans, and how to review results in Excel and in Access.