An information technology (IT) services company was engaged by a European telecom service provider to provide end-to-end testing for its five lines of business. End-to-end testing is a methodology used to test whether the flow of a software application is functioning as expected from start to finish – for example, from receipt of customer order all the way through to fulfillment, accounting and shipping. This type of testing helps identify system dependencies and ensure that the right information is passed between various system components.
The end-to-end (E2E) test team soon discovered that each line of business within the telecom company had its own way of operating, including how it managed delivery, handled fluctuations in workload, reported on performance, and so on. The complexity of the situation was compounded by the number of internal and external teams the business lines interacted with, such as the design community, compotent teams and the project management office (PMO). The table below summarizes some of the challenges faced by the E2E team as they began their engagement with the client.
| Key Challenges in the Five Business Lines | |
| Issue | Implications |
| Different delivery structure |
|
| Different processes for different lines of business | No standard operating procedures followed by the business lines. |
| Uneven demands |
|
| Resource utilization | Demand fluctuations and lack of cross-skilled resources led to resource underutilization. |
| Handoffs between teams | Multiple handoffs between multiple teams led to complicated and time consuming process. |
| Different structures for business reports | Different sets of reports produced by each of the lines of business led to rework. |
Creation of a Global Test Factory
To best manage the challenges noted above, the E2E test team decided to create in collaboration with the client company a “global test factory” – a dedicated space in which all E2E testing for the five business lines would take place. The factory would be built on Lean principles and shared services to enhance the client’s capabilities and thus exceed their performance expectations. (See Figure 1.) Its mission: to lead the way to the future of IT testing services by creating an adaptive organization through intelligent design.
The factory’s process owners (senior E2E team managers) identified the following key enablers needed to for the factory to fulfill its mission:
- Lean practices
- Shared services
- Knowledge management
- Right talent
- Standardization
- Management by metrics
Lean Implementation in Global Test Factory
The company chose to apply Lean in order to reduce waste in the processes related to E2E testing. The elimination of waste from the processes would help reduce cycle times, reduce the cost of operations and improve performance. The IT company had experience with Lean, having already implemented the practice in the operational areas of fulfillment, assurance and billing.
The concerns noted in Figure 1 provided additional motivation to apply Lean principles to the global test factory to enhance E2E testing capabilities. The factory contained five “layers” (see Figure 2):
- Factory command center
- Test scenario workshop
- Test execution plant
- Service introduction bays
- Shared services
The factory team prepared a high-level project plan (Figure 3) and project charter (Figure 4). Deliverables were mapped after each milestone. The company approached the application of Lean to E2E testing by performing due diligence and identifying the non-value-added steps from the processes. This approach (as shown in Figure 3) was followed for each of the five layers of the test factory.
Improvement to the Release Management Process
Figure 5 shows an example of one of the release management processes used in the factory. Release management is the process of managing new versions of the software, from the requirements-gathering stage to when software goes live. Releases are likely to include sprints (agreed-upon phases to design, develop and test) so that the particular software reaches the final stage in a controlled manner.
In the current-state VSM, several problem areas were identified:
- Design process does not involve E2E testing involvement
- Difficulty in release scope alignment
- Scope in workflow system and demand catalog needs to be aligned
- Design input per release scope is missing
- Designers do not make proper status updates in demand catalog, leading to wasted time
- Issued version of design document missing some functionality
- Follow-up with designer required to get all necessary documentation
- Late sprint commitments from components due to late designs closure
- Need to keep close eye on status of functionality
- Sprint misalignment with component functionalities and sprint commitments are delayed
- More time spent on sprint alignment calls
- Post-RPM churn functionalities are taken up by release manager on a case-by-case basis
- Perform follow up on these plans
Lean was used to develop the future-state value stream map (VSM). In the course of that exercise, the Lean implementation team eliminated 10 non-value-added steps from the release management process and made the following recommendations:
- Customer to prepare and circulate a design calendar for future releases to make the design planning more efficient.
- Release manager to send the workflow system extract (problematic designs and status) to factory leads so that they can update status in demand catalog.
- Dashboard to be maintained by front door team and published to customer with problems highlighted.
- Customer to ensure problems identified on dashboard are resolved.
- Customer and business line owner to determine cutoff time for receiving design input and finishing design plan.
- Release manager not to accept more than 10 percent of new churn functionalities.
- Priority of these functionalities to be determined by customer delivery plans.
Similar to its work on the release management process, the factory’s Lean team also completed due diligence with other areas, such as demand management, front door (team that looks at the low-level details of the customer’s demands; a team within the E2E team), and test execution. Figure 6 shows some of the qualitative recommendations of the factory team which clearly tied recommendations for actions to solution owners.
The following qualitative benefits were achieved at the global test factory:
- Process standardization wherever possible
- Elimination of redundancy and duplication in processes
- Prioritized KPIs based on business impacts
Quantitative benefits were calculated for all functions after NVA activities were removed, as shown in Figure 7. In total, the changes led to a savings of more than 4,400 hours over a nine-month period. Those 4,400 hours equated to 25 full-time employees; each person was reassigned to a different position within the company – no jobs were lost.
