Annex 15 paragraph 11.5 says: 'After implementation, and where appropriate, an evaluation of the effectiveness of change should be carried out to confirm the change has been successful.' ICH Q10 3.2.3(e) asks for 'an evaluation of the change undertaken after implementation to confirm the change objectives were achieved and that there was no deleterious impact on product quality'. Two questions: did it do what it was meant to, and did it do anything else. A change record closed without answering both is closed on completion, not on effectiveness, and inspectors know the difference.
Define the check when the change is raised
The effectiveness check cannot be designed after the fact, because after the fact the temptation is to choose whatever measure happens to look good. It is defined during the impact assessment, and it consists of four things: the objective of the change stated so that it can be tested, the measure that will show whether the objective was met, the acceptance criterion, and the window over which the measure is taken. The window must be long enough for the change to have had a chance to fail.
- Objective: what the change was meant to achieve. 'Reduce filling line stoppages due to stopper jams' rather than 'improve line performance'.
- Measure: something that is already recorded or can be recorded without heroics. Stoppages per shift from the line log; deviations by category from the quality system; yield; cycle time; test results; complaint rate.
- Criterion: the value that means success. Fewer than two jams per shift over the window; no deviations attributable to the change; yield within the validated range.
- Window: enough batches or enough time. For a process change, typically a defined number of batches. For a system or procedural change, a period long enough for every shift and every trained person to have used it.
- Unintended effects: what else will be watched. A change that fixes one thing and starts another is not effective.
Running the check
At the end of the window, the owner gathers the data defined in the plan, compares it to the criterion, and writes a short conclusion with the evidence attached or referenced. QA reviews it. If the criterion is met and nothing unexpected has appeared, the check passes and the record can move to closure. If not, the change is not effective, and the record should say so and say what happens next: a further change, a CAPA, a return to the previous state. A failed effectiveness check is not embarrassing; a check that was never designed to fail is.
Closure
A change control is closed when all actions on the plan are complete with evidence, verification is documented, the effectiveness check has passed or been formally waived with a reason, all affected documents are at their approved new revision, regulatory submissions have reached the status the classification required, quality agreement notifications are acknowledged, and any linked records (CAPAs, deviations, validation reports) are cross-referenced. QA closes it. The closure date should be after all of these, and if the record has been open a long time because the effectiveness window was long, that is fine; it is the correct state and the reason is on the record.
Sites that set a fixed closure target, such as 90 days, and measure performance on it, create pressure to close before the effectiveness window has elapsed. The better metric is closure within the target date set for that change, with extensions documented and approved. Late with an approved extension is controlled. Late without one, or closed early to hit a metric, is not.
Trending the change system
Chapter 1 paragraph 1.10 requires the product quality review to cover 'all changes carried out to the processes or analytical methods', and ICH Q10 section 4 expects management review of the PQS to include the performance of the change management system. Trending the system means looking at it as a whole: the number of changes by type and area, the proportion classified minor versus major, overdue actions, overdue effectiveness checks, changes implemented before approval, changes raised retrospectively, changes that led to deviations, and effectiveness checks that failed. A site where 95 percent of changes are minor and none has ever failed an effectiveness check is not running a good system; it is running one that does not look.
Change control in the PQR and management review
The PQR should list the changes to each product in the period with their classification, regulatory status and effectiveness outcome, and should ask whether any trend in the product's data coincides with a change. Management review should see the system metrics and act on them: resourcing where actions are chronically overdue, procedure revision where retrospective changes are common, training where classifications are inconsistent. Module 5 of the course works through defining checks for five change types and building the trending report that a QA head can take to management review.