PRC approves new USPS internal service performance measurement system

SteveSlideshow, Story

The Postal Regulatory Commission has approved the Postal Service’s new internal Service Performance Measurement (SPM) system for market dominant products.  The new system replaces the External First-Class (EXFC) measurement system that has been used for many years to track how successfully the Postal Service meets its goals for on-time service standards.  (The SPM system is described in detail in this USPS report.)

One of the main issues the Commission had to address was whether an internal system would be “capable of reporting accurate, reliable, and representative service performance data.”  After a review that’s gone on for over three years, the Commission decided that the new system, while not perfect, does meet these criteria.  Yesterday the Commission issued Order 4697 granting the Postal Service permission to implement the new system, starting with the first quarter of FY 2019.

The Postal Service proposed this change in the way it measures on-time service performance back in January 2015. Instead of contracting out to a third-party to measure performance, the Postal Service proposed to take advantage of the increased use of barcoding and do the measuring itself.  The Commission established Docket PI2015-1, and for the past three years, it has heard comments from various stakeholders as well as postal watchdogs Doug Carlson and David Popkin.  During this time, the Postal Service submitted quarterly audit reports showing side-by-side comparisons of data generated by the two systems.

The “legacy” External First-Class measurement system — it’s “external” because it was conducted by an independent third-party contractor (IBM) — measures the end-to-end (mail deposit to delivery) length of time it takes for a representative sample of mailpieces to be delivered.  Under this system, “droppers” deposit letters and flats addressed to “reporters,” and the participants record when mail was sent and received. Based on the transit times for these mailpieces, the system generates service performance data showing the percent of mail that was delivered on time, i.e., within the service standard.

For the new internal measurement system, the Postal Service contracted a third-party vendor to develop an audit approach for reviewing, verifying, and ensuring that the proposed service performance measurement system and its processes produce accurate, reliable, and representative results.   This system takes advantage of the fact that much of the mail is now barcoded and can be tracked on automation equipment by the Postal Service itself.

Unlike the legacy system, the new system doesn’t track particular pieces of mail, end to end.  Instead, it uses a statistical composite that averages, weights, and combines live mailpieces on three different stages of the delivery process — First Mile (collection), Processing Operation, and Last Mile (delivery).  The data are combined to measure overall transit time for what amounts to “virtual” mailpieces, as opposed to the actual mailpieces tracked by EXFC.  Here’s a graphic illustrating the process (from the SPM plan).

Internal Measurement Process Flow

Since the new system tracks such a large portion of the mail flow, the Postal Service considers it a “census” rather than “sampling” methodology (just as the U.S. Census tries to include every household, not just a representative sample).

The Postal Service began testing the new internal measurement system during the first quarter of fiscal year 2016.  In many instances, the on-time performance data produced by the two measurement systems seemed significantly different.  The Postal Service continued to modify the new system, the margins of error kept improving, and eventually the new system produced results that appeared to be at least as accurate, reliable, and representative as the legacy system had produced, if not more so.

The following chart shows one set of comparisons between the EXFC and SPM.  This one is for 3-to-5 day First Class Single Piece mail.  As the chart shows, the differences were relatively small, and as time passed, those differences tended to get smaller.  Also, while there had been concerns that the new system might give the Postal Service an opportunity to fix the data to show better results, the SPM generally produced lower performance results, not higher.  (For 3-5-Day First-Class Mail Single-Piece Flats, the SPM did sometimes produce better results than the legacy system.)

The Postal Service has stated that it “has not justified its proposed service performance measurement system on the basis of cost savings,” but it did indicate that there would be some savings.  The Postal Service estimated one-time costs for implementation of $10.75 million and recurring annual costs of $12.25 million, as compared to the $41 million it was spending each year on the legacy system.  The Commission found that the information provided by the Postal Service was “not sufficient to perform a cost analysis” to verify such savings.

Now that a new system is going into place, one problem will be trying to compare future results with previous years.  How will the Commission be able to evaluate service performance trends?  Will the differences indicate a change in service performance or will they be attributable to methodological differences between the systems?

In its Order yesterday, the Commission directs the Postal Service to explain, in the first annual compliance report, any significant service performance discrepancies between the two systems and to propose a method for comparing the data they produce so that the Commission can continue to track trends.

For more about the change in measuring systems, see these previous posts:

(Image: USPS OIG report)