A couple of weeks ago, the Postal Service announced that it was starting the second round of the Postal Pulse.  That’s the Postal Service’s name for the Gallup Q12, the world’s most famous employee engagement survey.  The Postal Service decided to switch to the Q12 after using the Voice of the Employee survey for many years because, as explained in the 2015 Annual Report, the Q12 is “shorter, simpler to complete and includes actionable items.”

As part of its Annual Compliance Determination review, the Postal Regulatory Commission has been asking the Postal Service questions about the results of the Postal Pulse survey conducted last year, and more information has come out since our previous post on the topic.

 

Scores on the 12 questions

When it first reported on the survey to the PRC, the Postal Service shared only the response rate and the grand mean, i.e., the overall average for the twelve questions on the Q12.

To follow up, the PRC requested the “disaggregated” scores for each of the twelve Postal Pulse questions, and last week the Postal Service shared the mean scores for each question.  (See USPS Responses to Chairman’s Information Request No. 13, Feb. 22, 2016.)

Participants respond to each question on a scale of 1 to 5, with 1 meaning “strongly disagree,” and 5 meaning “strongly agree.”  The higher the score, the more favorable the response.  Here are the Postal Service’s average scores.

POSTAL PULSE SURVEY RESULTS FOR 2015	Mean Score
Grand Mean	3.16
Q01. I know what is expected of me at work.	4.20
Q02. I have the materials and equipment I need to do my work right.	3.52
Q03. At work, I have the opportunity to do what I do best every day.	3.63
Q04. In the last seven days, I have received recognition or praise for doing good work.	2.60
Q05. My supervisor, or someone at work, seems to care about me as a person.	3.29
Q06. There is someone at work who encourages my development.	2.86
Q07. At work, my opinions seem to count.	2.71
Q08. The mission or purpose of my company makes me feel my job is important.	3.42
Q09. My fellow employees are committed to doing quality work.	3.40
Q10. I have a best friend at work.	2.84
Q11. In the last six months, someone at work has talked to me about my progress.	2.62
Q12. This last year, I have had opportunities at work to learn and grow.	2.92

The scores seem to indicate that employees feel they know what’s expected of them, but they don’t  get enough recognition or encouragement from their supervisors, they feel that their opinions don’t count for much, and they don’t get much feedback about their progress.

As the next round of Postal Pulse gets started, it appears that management has plenty of room for improvement.

 

Grading the scores

If you think of the numbers 1 to 5 as letter grades, with a 1 being an F and a 5 being an A, a grand mean of 3.16 would be a C, and the scores of 2.6 and 2.7 on questions 4 and 7 would be a C minus.

Those grades might seem kind of average, but interpreted another way, they’re probably worse than that.  Here’s why.

A complete Q12 report from Gallup doesn’t just provide the mean scores for each of the twelve questions.  It also provides comparisons to similar organizations in Gallup’s huge database, as well as a lot of other analysis.  Being able to make such comparisons is one of the main reasons to use the Gallup Q12 in the first place.

Unfortunately, the PRC did not ask for the full report, so the scores for the twelve questions have no context.  There’s no way to know how the Postal Service’s averages measure up to other companies, and the scores can’t be compared to previous USPS surveys either because the Voice of the Employee had different questions.

If one could see the full Gallup report, it would probably show that the Postal Service scores are even worse than at first appears.  For example, a sore of 2.6 on Question #4 might put the Postal Service in, say, the 5th percentile.  That would mean that 95 percent of organizations in the Gallup database ranked higher on this question.  Rather than a C-, this score would look more like an F.

Another problem with the data the Postal Service gave the PRC is that there are actually thirteen questions on the survey, not twelve, and the Postal Service has not provided the score for the additional question (which is numbered Q00): “On a five-point scale, where 5 means extremely satisfied and 1 means extremely dissatisfied, how satisfied are you with the Postal Service as a place to work?”

This is the first question on the survey and it’s one of the most important because it helps evaluate overall job satisfaction.  When a company discusses its Q12 results, this score is frequently mentioned.

Unfortunately, the Commission’s information request asked for the disaggregated survey results “for each of the 12 Postal Pulse questions,” and that’s all that the Postal Service provided.  The Postal Service could have provided the mean for the thirteenth question anyway, but it chose not to do so.

 

Putting the USPS grand mean in context

Given that the Postal Service has not provided any percentile rankings for its scores, we wanted to get a better sense of how the Postal Service’s grand mean of 3.16 compared to other organizations.  We surveyed several dozen reports of Gallup Q12 results that have been posted online and found 55 cases in which the grand mean was cited.

Here’s a table showing how these other agencies and companies did on the Q12.  (You can see it on Google Docs, where it can be sorted, etc., here.)

The average score for these 55 organizations was 3.96.  The lowest grand mean we could find was 3.07, for the UK’s Thames Water Utility.  Its corporate responsibility report called the score “disappointing.”  The next lowest score was 3.26, for the Criminal Investigation Branch of the New Zealand Police department.

The Postal Service’s grand mean of 3.16 was thus the second lowest of all the scores we could find, and it was significantly lower than the overall average for the 55 organizations.

The Criminal Investigation Branch’s grand mean of 3.26 put it in the 11th percentile of the Gallup database.  That means only ten percent of the comparable companies did worse.

The Postal Service’s grand mean of 3.16 could put the Postal Service somewhere in the bottom 10th percentile, maybe lower, depending on what the comparable organizations were.  After our previous post on the Pulse, we got an email from a postal employee who said that Jeff Williamson, the Postal Service’s Chief Human Resources Officer and Executive Vice President, appeared in a video about the survey and said something to the effect that the results placed the postal service in the bottom 1st percentile.  If true, it doesn’t get any worse than that.

Williamson went on to explain that comparing the Postal Service to private sector companies and other government agencies isn’t fair, and he may be right about that.  None of the 55 organizations from our Internet search are as big as the Postal Service, and none face the same challenges.

But comparing an organization’s results with other organizations is what Gallup does when it prepares a Q12 report.  Why bother paying for the Q12 if you’re not going to use Galllup’s database to make these comparisons?  (The Postal Service has a contract for $1.8 million with Gallup for implementing the Pulse.)

It’s also important to remember that the grand mean of 3.16 is the national average.  The Postal Service was probably given data at a more granular level as well.

In response to a question from the PRC, the Postal Service explained that it has approximately 18,000 business units nationwide (a business unit is defined as “a postal manager at or above the EAS-18 grade level e and his/her team of employees”).

The Postal Service probably has Pulse scores for each of these 18,000 units.  The averages must vary considerably, and some must be well below the national average.  One wonders what postal headquarters is telling management in these units.

 

Opt-outs and the response rate

Another issue for the PRC in reviewing the Pulse results has been the response rate.  Companies typically have a participation rate of about 70 or 80 percent.  Gallup gives out a Great Workplace Award “to recognize organizations worldwide for their extraordinary ability to create an engaged workplace,” and one of the criteria is a minimum response rate of 80 percent.  One report says Gallup’s median response rate is 87 percent.

In its 2015 Annual Report to Congress, the Postal Service said that it delivered the Postal Pulse survey to 595,000 career and non-career employees, and 47 percent of employees completed and returned the survey.  That response rate is obviously very low, but it might be even worse than that.

At the top of the Postal Pulse is a check box for “I do not wish to participate in the USPS Employee Survey at this time.”  This opt-out box does not seem to be typical for the Q12 that we found online.  If the Postal Service added it to the survey, it’s not clear why.  What’s the difference between checking the box and not doing the survey at all?

Well, it turns out that the Postal Service may have counted those who checked the opt-out box as part of the participation rate.  That’s the implication of a statement in the USPS FAQ page that went out with the survey.  It states:

Employees that answer at least one question will have their data counted and included in final reports. Employees that opt out but still return the survey will be counted toward overall participation rates but their unanswered questions will not be included in the final reports.

In order to find out the how the opt-out box was affecting the response rate, the PRC has filed another Chairman’s Information Request (CHIR No. 17, question 6).  It asks the Postal Service to provide the participation rate excluding those employees who opted out of the survey (as opposed to not submitting the survey at all).  The responses for this CHIR were due yesterday, but the Postal Service is running late and this question hasn’t been answered yet.

 

Comparing the Pulse to the VOE

In the Annual Report, the Postal Service stated that the grand mean and overall response rate “equate closely to the overall response rate and favorability ratings of our FY2014 end-of-year VOE survey results.”

But elsewhere (Responses to CHIR No. 3), the Postal Service told the PRC that the two surveys were so different that they “cannot be directly compared.”  The Commission therefore asked the Postal Service about the apparent inconsistency.  (CHIR No. 15, question #3.)

Yesterday the Postal Service filed its response to this information request.  It explained that the response rate for career employees was 46 percent; for non-career employees, 49 percent; and overall, 47 percent.

The response rate for the VOE in 2014 was 51 percent, which the Postal Service points out was “only 4 percent more” than the 2015 Pulse response rate.  The two sets of result therefore “equate closely.”

The Postal Service didn’t mention it in yesterday’s CHIR response, but the VOE participation rate was 61 percent in 2008 and “over 60 percent” in 2009 — about 14 percent higher than the Pulse rate.  Plus, in its 2014 Annual Report, the Postal Service noted that the VOE response rate for non-career employees was over 70 percent — almost 21 percent higher than the Pulse rate for non-career employees last year.

As for how the grand mean of 3.16 equated closely to the VOE responses, the Postal Service offered this explanation in yesterday’s CHIR response:

The VOE Index Score in 2014 was 65.01; this is moderately more than the midpoint of its scale of 0 to 100. The Postal Pulse index score was 3.16; this is moderately more than the midpoint of its scale from 1 to 5. These are numerical references only. It should be reemphasized that the VOE and Postal Pulse survey results cannot be directly compared. The two surveys and what they measure are completely different.

That explanation may make sense, but it’s a little misleading.  Characterizing a grand mean of 3.16 as “moderately more than the midpoint” makes it seem like the score was slightly above average or something.  In fact, as we’ve seen by comparing the score to that of other organizations, the Postal Service’s grand mean was way below average and probably ranked in a very low percentile compared to other organizations in the Gallup database.

Overall, given that many of the Postal Pulse questions are about how employees think they’re being treated by supervisors, the survey results do not reflect well on management’s performance.

The extent of employee-engagement problems would be much clearer if the Postal Service released the details of the Gallup report, complete with the percentile rankings for each question and all the other metrics used to analyze the results.  That, however, will not be happening.