According to a notice posted on USPS Blue, the Postal Service began conducting the second round of the Postal Pulse survey yesterday.
Postal Pulse is the Postal Service’s name for the Gallup Q12 survey, one of the most widely used employee engagement surveys in the world. (A copy of the Postal Pulse survey is here; its twelve questions are identical to those on the Q12, with one additional question.)
Last April, the Postal Service replaced the Voice of the Employee (VOE) survey, which had been used for many years, with the new Pulse. In October 2015, a USPS press release indicated that the results of the survey would soon be made available to postal employees on USPS Blue.
The Postal Service has not shared the results of the survey with the public, and it’s not certain that they were ever published on Blue for postal employees either. In early December, we got a note from a postal worker saying that the results weren’t on Blue yet. If they’re on Blue now, perhaps someone can share them with the rest of us.
The morale of employees in the workplace is no small matter. As noted by Dead Tree Edition, the Postal Service hired 117,000 new workers in FY 2015 (nearly all of them in non-career positions), and almost half of them (52,000) are already gone.
Senator Tom Carper’s iPOST bill, which is now working its way through Congress, has an entire section (108) that addresses the morale of Postal Service employees. The bill requires the Office of Inspector General to examine morale and develop a list of recommendations, and it would also appoint an individual to work with the postal unions and other employee organizations on morale issues.
The Postal Pulse survey already contains information that the OIG and the new morale official would probably want to gather. There’s no need to wait for Congress to act on iPOST. Now that round two of the Pulse is beginning, the Postal Service should publish the results of round one.
Measuring employee engagement
In its 2015 Annual Report to Congress, the Postal Service explains that it switched from the VOE to the Gallup Q12 because it is “far superior to the VOE survey as an instrument for measuring employee engagement. It’s shorter, simpler to complete and includes actionable items.” The Annual Report also provides a few numbers about the survey conducted last year.
The Postal Service reports that it delivered the survey to 595,000 career and non-career employees. It then offers this summary of the results:
Forty-seven percent of employees completed and returned the survey. The grand mean engagement score result was 3.16 on a scale of 1 to 5. These results equate closely to the overall response rate and favorability ratings of our FY2014 end-of-year VOE survey results.
That’s all the Annual Report has to say about the Pulse results. (The “grand mean” is the overall average score for all the questions on the survey.)
Then in January of this year, the Postal Regulatory Commission asked about the Postal Pulse as part of its Annual Compliance Determination review. The Commission specifically asked the Postal Service to “provide FY 2015 result(s) and FY 2016 target(s).”
In response, the Postal Service provided basically the same information that was in the Annual Report. As stated in the Annual Report, the grand mean was 3.16, and the response rate was 47 percent (for some reason, slightly higher than quoted in the Annual Report).
The Postal Service told the PRC that it has no target for next year. It also explained that “because The Postal Pulse is purely a measure of Employee Engagement (as compared with the VOE, which measured engagement and satisfaction), the scores cannot be directly compared.”
It’s not clear how the Postal Service was nonetheless able to state in its Annual Report that the Pulse results “equate closely” to the favorability ratings in the VOE survey.
The response rate
While comparing the Pulse results with the VOE results may be problematic, it’s not as if the Postal Pulse exists in a vacuum.
The Gallup Q12 is considered the “gold standard” for employee engagement surveys, and it has been administered to more than 25 million employees in 189 different countries and 69 languages for use by hundreds of organizations. Gallup has a massive database of survey results that permits comparisons between an individual organization and other organizations. This is one of the main benefits of using a standardized survey like the Q12.
Just in terms of the Postal Service’s response rate of 47 percent, it helps to look at the broader context.
For example, in 2005, Gallup reviewed the data from 89 organizations (388,716 employees) in its database and found a response rate of 84 percent.
Gallup gives out a Great Workplace Award “to recognize organizations worldwide for their extraordinary ability to create an engaged workplace,” and one of the criteria is a minimum response rate of 80 percent.
It would seem, then, that the Postal Pulse response rate of 47 percent wasn’t so great. It might be pretty poor.
Perhaps the efforts of the APWU to discourage its members from submitting a response to the Postal Pulse have been effective. The union opposes the Postal Pulse, just as it opposed the VOE, because it’s concerned that results will be used against the union in contract negotiations.
Some employees may have chosen not to do the survey for other reasons. Maybe they were worried about confidentiality, or maybe they were cynical about the Pulse and management’s attitude, or maybe they were not engaged enough even to fill out an engagement survey.
The grand mean
The scores on the Q12 are based on employee responses to the twelve questions on the survey (thirteen in the case of the Pulse). They’re actually not questions per se but statements, like number ten: “I have a best friend at work.” A response of 1 means “strongly disagree,” while 5 means “strongly agree.” The higher the number, the more favorable the response.
When Gallup gives an organization an analysis of the survey results, it provides a lot of information: the number of people who responded to each question, how many respondents gave what score, the average score for each question, the overall average (the grand mean), and so on.
A typical analysis also provides benchmark scores for each question, drawn from Gallup’s database, and the organization’s ranking in the database among comparable businesses and organizations. A percentile score of 60 on a given question or the grand mean, for example, indicates that the average score is higher than 60 percent of all other scores and lower than 40 percent. Gallup says that “percentile rankings of 75 and above are considered world class when compared with the Gallup database.”
In the absence of comparisons to a benchmark or a percentile ranking, it’s almost impossible to know what to make of the Postal Service’s grand mean of 3.16.
Since it’s roughly in the middle of the 1-to-5 scale, it may seem about average, but 3.16 may actually be well below the grand mean for similar organizations. One can get some sense of what the number means by looking at the survey results from other businesses and agencies. It turns out that many organizations — unlike the Postal Service — publish their Q12 survey results online.
Looking at other organizations
In 2013, the Montgomery County School System (MCPS) shared its Q12 results with the public. MCPS had a response rate of 76 percent, and its grand mean was 3.83. That score ranked it in the 34th percentile in Gallup’s Education Services Database (near the bottom third).
The executive summary for the MCPS survey also shows the average scores for each question and comparisons to Gallup’s Education database. Scores that are especially low are highlighted. For example, MCPS scored 3.18 for the fourth question (“In the last seven days, I have received recognition or praise for doing good work”), which put it in the 27th percentile (near the bottom quarter).
The University of Virginia Medical Center also published its 2013 Q12 results online. Its grand mean was 3.84, which ranked it in the 29th percentile in Gallup’s Healthcare database. That wasn’t so good. The average grand mean for the top 25 percent of workgroups in this database was 4.44.
The Commonwealth Health Corporation (CHC) published its 2011 survey results. It had a grand mean of 4.06, which placed it in the 55th percentile of Gallup’s Healthcare database, just about in the middle of the benchmark group. As Gallup explains in the report, a “Best Practice” designation requires a score of 4.30, which puts the organization in the top 25 percent of the Healthcare database, and a “World Class” designation requires a score of 4.58, which puts an organization in the top 10 percent.
The USDA Farm Service published its Q12 scorecard for 2006 online. Its grand mean was 3.55. That’s compared to 3.74 for the top half of Gallup’s Public Administration database and 4.03 for the top 25 percent. A score of 3.55 put the USDA Farm Service in the bottom half of its benchmark group.
GlaxoSmithKline, the British pharmaceutical company, published its 2005 Q12 results online. Its grand mean was 3.97. That was very good. The top 50 percent of the database had a grand mean of 3.72, and the top 25 percent had a grand mean of 4.02. (It’s not clear if those comparisons refer to pharmaceutical companies or another benchmark group or to the entire Gallup database.)
A complete report
Overall, as this randomly selected group of survey results suggests, what makes a good score varies considerably, depending on the type of enterprise. Generally speaking, though, a good score would be around 4, and anything close to 3 puts an organization in the bottom half (or worse) of comparable organizations.
Unfortunately, this means that the Postal Service’s grand mean score of 3.16 may not be very good, and it might be pretty bad.
In order to get a better sense of what last year’s Postal Pulse survey revealed, however, it would be necessary to see a more complete report. It would show the scores for all of the questions, not just the grand mean, as well as comparable numbers for a benchmark group, plus all of the other types of information shown in these online reports.
The Postal Pulse survey is costing the Postal Service serious money. According to the contract between the Postal Service and Gallup, which we obtained by filing a Freedom of Information Request last year, the total amount for the award was $1,790,724. That covers a two-year period, from August 18, 2014 to August 19, 2016. (Most of the contract is redacted, but you can see it here.)
The Postal Service did not spend $1.8 million just to get the grand mean. Gallup has obviously given the Postal Service a lot more data and analysis than the Postal Service has shared with Congress, the PRC, or the public, and perhaps postal employees as well.
It will be interesting to see if the PRC is satisfied with what the Postal Service has shared so far about the Pulse results. It would be unfortunate if the Commission did not demonstrate more concern about employee engagement as part of its compliance review.
The Postal Service responded to the Commission’s request for the survey results on January 21, 2016. Since then, the Commission has not asked for any additional information. Maybe the response rate and the grand mean were enough. (Correction: The Commission asked for all of the results on February 11, a few days before this post was published.)
There’s more on the Postal Pulse in these previous posts:
The Postal Pulse results are out (sort of) (October 7, 2015)
A happy worker is a productive worker: Postal Pulse surveys employee engagement (April 5, 2015)
The doctor will see you now: The Postal Service takes the pulse of employees (March 27, 2015)
(Cartoon credits: The Dilbert cartoons by Scott Adams are from Dilbert.com, where there are more on employee satisfaction here.)