Document created: 31 August 04
Air University Review, March-April 1970

The OER—Heaven Help Us

Colonel John T. MCCabe

 Recently a meeting took place in the reception area outside Southgate-One. An older gentleman in flowing white robe greeted a somewhat perplexed Air Force officer. “Hello, Major Jones. Sorry to keep you waiting. We’ve known of your pending arrival for some time, but the computer listing with the details for your processing was a little slow getting to the reception area. You can call me Thaddeus; I’m C-Team Interviewer, working Southgate-One this shift. I report directly to Saint Peter, who has been delegated broad authority on admittances.” “I’m please to meet you, sir,” Major Jones responded politely. “It was all so sudden, I haven’t got used to the idea of being in Heaven. I’m anxious to learn more about the routine up here.” “Well,” said Thaddeus, “let me start by saying you’re not in Heaven yet. Not that there isn’t every probability that you’ll get in. You’ve passed the initial screening done by our computer, but there’s more to it than that. The input data for our computer system comes from our angel-agents on Earth, and in many respects the data just aren’t good enough. That’s really the main reason for these interviews. I suppose you’re familiar with computers and their shortcomings, Major?”

“Oh, yes sir, almost everyone in the Air Force today knows about computers, and about having proper data inputs. We have a saying—”

“Garbage in, garbage out,” Thaddeus interrupted. “I’ve heard it before, Major. Not that it’s very amusing up here. In fact, the data problem is a rather sensitive subject right now. It’s being said that the decision on when we go to the next-generation equipment hinges largely on getting improvements in the accuracy of our Earthly inputs. But with your population explosion, we’ve just got to do something to speed up the system.”

“Gosh, sir, you ought to have some of our personnel specialists up here to help you with this problem,” said the major. “They’ve done wonders automating our Air Force records. Just the other day I got a punch-card telling me it was time for a tetanus booster. What a waste that was?”

“Oh, don’t worry, Major. We make a special effort to get your personnel types. Not that we’re allowed to discriminate, but we can waiver minor irregularities and recommend probational assignments. However, we just don’t seem to be getting our share of the top talent. It’s almost as if there were a conspiracy.”

Continuing on, the fatherly gentleman said, “But let’s get on with the interview, Major Jones. We can discuss computers some more during the coffee break if you’d like.”

“Yes sir,” replied the major in his best military manner. “What would you like to know?”

After leafing through a file of papers, Thaddeus looked up, “Well, I think we have all the factual information that’s needed—family data, education, military training, awards and decorations, and so forth, even your physical condition—Chart 3B of 5BX, but with a little trouble on the push-ups.”

“It was too soon after lunch,” Jones offered quietly.

“Well, it doesn’t really matter; mostly we’re joggers up here now, but it’s probably just a fad. I can remember when everyone was throwing the discus. My, how time flies!”

Then Thaddeus went on, “Now let’s see, your Earth-angel sent along a file of OER’S. That’s ‘Officer Effectiveness Report,’ isn’t it? To tell the truth, Major, I’m a little disappointed.”

This startled Jones. “But, sir, my ratings have been excellent.”

“It’s not that. Or rather, it’s just that. You see, every OER I’ve ever looked at rated its subject top-drawer. I’ve also been told that most of your officers who don’t get past our initial screening also get excellent ratings. It’s a shame, “ continued Thaddeus. “We really need information on Earthly job performance and honest opinions of superiors. Of course, we can still get this information, using techniques like your TV instant replay. But it isn’t very cost-effective.

Major Jones groaned, “Don’t tell me, sir—not up here, too!”

“Ah, yes, ‘Waste not, want not.’ You must have seen the latest campaign posters?”

“I’ll watch for them,” the major muttered, half to himself.

Thaddeus let the remark go unnoticed. “Back to these OER’S. Let’s see if we can salvage anything from them. I might tell you, in the past we’ve tried giving numeric values to the boxes checked and running them through the computers; but the results have so far defied analysis. Too much inflation, and too many inconsistencies among raters.”

“I think our promotion boards have the same problem,” offered Jones.

“I can believe it. I just don’t see how—in Heaven’s name—we can use such data. And the word picture doesn’t really help either. I’m sure it’s designed to support the ratings.”

“It’s supposed to be the other way around,” said the major, “but I have to agree with you, sir.”

Thaddeus lifted one of the blue sheets of paper. “This one says you were ‘responsible for the timely development and implementation of an automated records filing, retrieval, and inventory system.’ Does that mean you’re a programmer?”

“Oh, no,” replied Jones quickly. “I’m more of a systems analyst. You know what that is?”

“Of course,” the older man answered. “We’ve got lots of systems analysts here, but not many of them can program. We’re very short on good programmers, and with talk of going to the next-generation equipment, things will no doubt get worse before they get better. Programmers are a ‘special action’ item on our interview check list. If you’re interested, I can arrange an aptitude test and a training interview?”

“Thanks,” was the unenthusiastic reply from Major Jones. “I’ll think about it. But at my age—Earthly age, I should say—well, it’s tough just remembering things like multiplication tables.”

Thaddeus then read through several more of the major’s OER’S and referred to a report from Jones’s Earth-angel.

“How is it, Major, that these discrepancies—like overstocking supplies and running out of TDY funds—aren’t mentioned in your OER’S?”

“Boy, if they were, I’d be dead—ha—I mean career dead!” And somewhat nervously, Jones continued, “Besides, my rater was responsible for reviewing and approving my recommendations, so he made the same mistakes. Anyway, we managed to correct these problems. They happen all the time.”

Thaddeus wasn’t convinced. “From what I’ve seen, your Air Force could use another rating system. I’m sure the OER is good for some things, but I don’t think it’s much help in separating the wheat from the chaff. Or, how do you say it?”

“The men from the boys?”

“Yes,” continued Thaddeus. “We sure could use an honest Earthly rating system as a part of our evaluation process. If only such a system could be devised and tied to the computer, why, the savings in Heavenly time and Earthly frayed emotions would be terrific!”

“It will never happen,” Jones guaranteed.

“I guess you’re right. But it’s something to think about while we’re waiting for the analysis of your replays,” suggested Thaddeus. “You might also want to look over those FORTRAN manuals—the programmer option I spoke of?”

Jones tried to ignore the last remark. “Isn’t it time for that coffee break?”

Thaddeus agreed, “Indeed it is, Major.” Then as the two men rose to leave, he caught Jones somewhat off guard by adding, “And with regard to another item in your history, while we’re having coffee maybe you can tell me something about that affair in Naples.”

Major Jones and Thaddeus concluded their coffee break and the remainder of the interview routinely. Major Jones’s admittance into Heaven seemed assured but required some additional verification of data. So there was to be a slight delay

A few days after the initial interview, Major Jones requested and was granted another audience with Thaddeus, who greeted him warmly.

“Good to see you again, Major. Sorry I don’t have anything more on your entrance status. These things always take time. What’s on your mind?”

Jones appeared relaxed and confident. “Well, Thaddeus, with all the free time you’ve given me, I’ve been thinking about what you said concerning the need for a better ‘down-to-Earth’ rating system. I’ve now got some ideas that might interest you. Do you have time to discuss them?”

“Certainly,” replied the older man. “It’s a subject dear to my heart, as you no doubt have guessed. What are these ideas?”

“Well, first of all, I think the handwriting on the wall says computers are here to stay, and any new rating system should be compatible with present and future data-automation techniques,” said Jones. “This suggests a numeric rating system.”

“I’ll agree to that, Major. Go on.”

“Of course, the idea of a numeric rating system isn’t new. Schoolteachers have used some such system for years. What I’d like to suggest as maybe being new is a way of achieving a degree of comparability among all the ratings of a group. I think this has been lacking and is a major criticism of our rating systems to date.”

“You know,” Jones continued, “we could go from our present block-checking system to an equivalent numeric system by simply having the rating officials give their evaluations of percentile standings, among contemporaries, of the individuals being rated. The trouble with such a numeric system is the same as with the present one—there are no hard and fast standards that would allow comparisons of ratings made by different individuals or groups. The tendency would still be for raters to inflate their ratings in an attempt to get at least a fair share of the rewards for their own people.”

“You mean,” said Thaddeus, “a so-called percentile evaluation rating system would probably show a lot more people above the 50 percentile score than below? I can believe that.”

“Yes,” the major continued, “and after each rating cycle, those raters whose distributions of ratings were lower than the ratings of the group as a whole would probably attempt to adjust their next ratings upward to achieve parity with the group. Hence the perpetual inflation.”

“Then what do you suggest as an alternative?”

“Well, Thaddeus, I like the idea of a true percentile rating system when it’s applied to a large group. In such cases the relative standing of each individual in the group corresponds automatically to a percentile level indicating, in general, the percent of the group with lower ratings.”

Jones continued, “In large groups, small differences in evaluated standing generally correspond to small differences in percentile values. However, as groups get smaller, the percentile differences between individuals in adjacent standings get larger, thus possibly forcing the rater to spread his percentile ratings more than his honest evaluation warrants.”

“See if I’m following you, Major,” Thaddeus interrupted. “You say if I’m rating only two people on a true percentile scale, I must rate one above the other and give the better man a numeric percentile rating of 100 and the other a zero? That is quite a spread!”

The major was quick to reply, “No, Thaddeus. First of all, you could rate the two people equal. But suppose you did choose to rate one above the other; then their percentile ratings, by my system, would be 25 and 75. I figure it this way: there are two people being rated, so the percentile scale, zero to 100, should be divided in half with the lower rated person representing the range from zero to 50 and the upper rated person representing the range from 50 to 100; then each person is given the mid-point value of his portion of the scale. If the two people were rated equal, they would each represent the whole percentile range, zero to 100, and each be given the mid-point value of 50.”

“I think I see it now. Those are my only options if I have two people to rate under a true percentile system—either 25 and 75 or two 50s?”

“That’s right, Thaddeus,” replied Major Jones, sensing he had really captured the old gentleman’s interest. “Of course, the computer could calculate the numeric scores if you just specified the relative standings, including any ‘equal standing’ individuals.”

“But like you said, Major, for small groups the options are either a wide spread or no spread of percentile scores, or some limited number of combinations, I suppose. How can you get around that?”

“Well, first, you should notice one good thing about the true percentile system. The average of all the ratings, regardless of the size of the group, and for all groups, is always 50. So,” reasoned the major, “let’s design a numeric rating system that forces the rater of a group of people to come up with ratings that have a fixed average. That average could be 50, or it could be some other number, like, say, 80. We’re not very used to thinking of scores according to percentiles, but most of us are familiar with educational grading systems where 80 is a pretty fair average.

“Then,” asked Thaddeus, “you’re suggesting that when I or others rate any group of people, their numeric scores must be made to average 80? How about limits on the amounts of spread in the ratings?”

Major Jones had given this a lot of thought. “I think we should try to keep the spread of grades as uniform as possible without being unduly restrictive on the rater. I think this can be done by placing some kind of a limit on the overall amount that a group of scores may differ from their assigned average. One way could be to require that the difference from average be not more than some set value, say five.

“Or better still,” Jones went on, “using a statistical term, we could limit the root-mean-square difference to something like five. This would allow a little more rater flexibility but still restrict large departures from the assigned average. Do you know what I mean by ‘root-mean-square’ difference?”

Thaddeus answered, “Yes, I’ve been exposed to RMS by some of our computer people. They say it’s the square root of the average of the squared values. But I’m not sure how it would apply to your rating system.”

Jones was ready. “I’ve worked out a few examples that might help explain this. Let’s say the ratings have to average 80, and they have an RMS difference of five or less. If I have only one person to rate, I obviously have to give him 80. If I have two people to rate, I can rate them both 80, or any combination equally above and below 80 so long as I don’t go beyond 75-85. At 75-85 the average is still 80, and the RMS difference from 80 is the maximum allowed, five.

“When I have three people to rate,” Jones went on, “it’s more interesting. I could go as low as 73 or as high as 87, but not both. The greatest range I could have would be 12, a low of 74 and high of 86, with the third score being 80.

“You can see, Thaddeus, as the number of people of a group being rated by one individual increases, so does the maximum permissible departure from the mean, as well as the maximum permissible range. However, these latter increases taper off quite rapidly. For example, if I were rating ten people under the same rules, I couldn’t give a score below 66 or above 94, and not both. Also, my maximum range would be 22, from 68 to 90, 69 to 91, or 70 to 92.”

Thaddeus had some doubts. “That’s all very interesting, Major. But how could it apply to your Air Force OER situation, where you usually rate people singly, not in groups? And if you did group them, it wouldn’t seem fair to have the lower-grade people compete in the same groups with the higher ranks? Yet, if you didn’t do this, your groups would often be too small to get a meaningful distribution of grades.”

Major Jones wasn’t disturbed. “Those are good points, Thaddeus. As I envision it, this system would require Air Force-wide rating of each grade, each at a designated time, maybe every six months. Lieutenants might be rated in February and August, captains in March and September, majors in April and October, and so on. As for the groups being small and thus limiting the distribution of scores, my feeling is that the grading system should include the first-line supervisory rating and probably two levels of what we now call indorsement ratings. While the group size might be a problem at the lowest level, the secondhand and third-level raters probably would have an adequate-sized base to work with. From my own experience, I think an officer’s performance is generally recognized to his third-higher level of supervision. If not, I don’t think it would hurt to encourage that kind of interest.

“Of course,” Jones continued, “there are bound to be instances where an officer’s performance is not known at one or more of the three supervision levels, and then he should receive the average numeric rating. In our example, that was 80.

“I suppose,” interjected Thaddeus “an automatic average score would be given when the rating official was not in the individual’s supervision line at least some minimum length of time, like 90 days?”

“Yes, or for any number of other reasons, like some TDY or training assignments. And don’t forget the average score that is mandatory when there is only one individual in the group being rated.”

Thaddeus offered a suggestion. “You know, Major, where the average score is assigned in a noncompetitive way, it would be nice to attach some kind of footnote or subscript to provide the reason. I’m sure this information would be of some value in analyzing an individual’s history of ratings.”

The major was impressed. “Absolutely, Thaddeus. I can imagine several different analyses of an individual’s rating history being generated by the computer. For example, the system could provide averages of the ratings of the three levels, or it could assign different weights to each level to produce a weighted average. Noncompetitive scores could be eliminated in another analysis.

“One other option that appeals to me,” the major continued, “is to do as is done in judging some of our Earthly sports events: compute average scores after discarding the highest and lowest ratings received over some time period. This could be done while providing something like two-year running means of each individual’s ratings.”

Thaddeus gave this some thought. “Let’s see; if each person were rated every six months and at every three supervisory levels you mentioned, that would mean six ratings each year. It would also mean a lot of bookkeeping.”

“More than that, Thaddeus. Six overall ratings each year is right. But I think the system can be designed to handle all the rating factors that make up the present block-checking portion of the OER.

“Each factor,” the major went on, “like leadership, job capability, writing ability, and so on, would be rated according to the same restrictions of group average and spread of scores. In the long run, hopefully this could provide a means of identifying individuals best qualified in particular talents or combinations of talents.

“As far as the bookkeeping goes,” he continued, “don’t forget the computer. Not only can it keep track of the ratings and provide any of a wide assortment of analyses, but with proper inputs it can also be made to provide the periodic lists of who rates whom at the various levels of supervision, even identifying certain mandatory noncompetitive ratings, like for insufficient time in the line of supervision.”

Major Jones pressed ahead, “Also, don’t forget that the computer can keep track of the ratings of each individual rater. Some useful and interesting analyses could be made from this information, I’m sure. For instance, each rater would probably like to know how his spread of ratings compares with those of others. It would also be nice to know if, in general, there is a tendency for a person’s ratings to increase significantly as he is repeatedly rated by the same supervisor—the familiarity syndrome. If this turned out to be the case, then ratings could probably be adjusted to correct for this, at least in a general way.”

“A lot of what you have said seems to make sense,” Thaddeus said. “But I can see one big objection. How is this rating system going to affect those officers who get recognized for doing outstanding work and are assigned en masse to elite groups like your top staffs or professional schools? Isn’t the competition going to be keener in these groups? Isn’t it unfair to bring in a group of officers who had an average score of around 85 and have the system force their next ratings into a distribution that averages only 80?”

Major Jones wasn’t shaken by this attack. “You phrased the problem so well, Thaddeus, the answer is almost obvious. If, as you suggest, the officers being brought into particular groups have ratings which actually show significant differences from the Air Force-wide average, then their scores should be adjusted to account for this selectivity of assignment. And at the same time, maybe we should also look at the corresponding ratings of the individuals in these same elite groups as they move on to other assignments, to determine whether or not the adjustments were reasonable. In other words, if the average rating of all majors coming into the Air Staff is 85 and their ratings while assigned there are adjusted to keep that average, then we should expect comparable officers leaving the Air Staff to continue to earn ratings which, on the average, justify the increases given for the Air Staff duty. These adjustments should be made by the computers, based on the history-of-rating data that continuously flow into the system. Then, as far as any rater is concerned, his ratings must satisfy the same limitations of average and spread as those of any other rater in the Air Force.”

“Sounds like it might almost work, Major Jones.”

“Oh, I’m sure the system won’t work too well, if at all, in some situations.” The major was almost apologetic. “I imagine that the ratings of officers about to leave the service would have to be watched carefully to see if raters were unjustifiably downgrading them in order to raise the ratings of others in the group who are staying on. Likewise, officers on control rosters shouldn’t be rated with their grade groups for similar reasons. There are bound to be other problems, too. One way to find and deal with them is through some kind of a series of ‘shakedown’ exercises. My feeling is that the benefits derived from the competitive aspects of the system—brutal as they may appear—should outweigh the disadvantages, especially when compared with the present OER system.”

Thaddeus might have been tiring. “Well, then, let’s see if we can summarize these ideas so I can brief the staff and maybe get some action started. Give me the high points, and I’ll jot them down.”

Jones was agreeable. “OK. First of all, the system is computer-oriented. The computer will, at the required times, prepare name lists of those individuals a rater is to rate at each of the three levels of supervision. Each list will also identify, as far as possible, those individuals requiring noncompetitive (average) scores. Next, the rater will fill in the ratings, subject to the restrictions of coming up with the required group average and not exceeding the RMS difference limit in each rating factor. The computer will check these items, of course.

“The point to emphasize here,” the major continued, “is that by forcing each rater to conform to these standards we eliminate inflation, and, over the long run, the ratings made by any one individual are more likely to be comparable with those made by any other individual than under the present system. At the same time, we eliminate much of the possible unfairness that can sometimes arise when one officer rates others of his own grade who indeed may be his competitors for promotion or preferential assignment.”

“That’s a good point, Major. The staff will like that pitch for fairness.”

“Next, Thaddeus, you might mention the idea of adjusting the ratings when there are significant group differences. I’m not at all sure how significant these differences will be, but the system can identify and take care of such problems as they come up.”

“OK,” said Thaddeus. “We’ve now got what we think are reasonably fair and unbiased rating data into the system. What sort of products should we generate from the data?”

“We mentioned a few,” Jones replied,  “like weighted and unweighted averages, one-year or two-year running averages with or without the exclusion of highest and lowest ratings, and, of course, the listing of individuals by scores for various screening and selection actions. You can also be sure that the personnel people will have many more ideas about how a person’s rating could be combined with such factual data as his age, flying status, education level, awards, service time, and so on, to come up with composite information for comparison purposes.

“Would this put the promotion boards out of business?” Thaddeus asked.

“I’m sure it wouldn’t,” the major replied. “But it could certainly lighten their load. Also remember that the system has other applications, like in screening people for writing or speaking ability, with or without college degrees, rated or nonrated, with or without Southeast Asian tours, and lots of other options. The thing is, this system permits what should be reasonably reliable rating information to be a part of an otherwise objective screening process. The human element hasn’t been eliminated, but the computer can and should do the dog work.”

Thaddeus smiled. “You’re talking like a systems analyst, Major Jones. You’ve sold me. Now all we need are some recommendations on how the system should be implemented.”

Jones thought a moment. “Needless to say, this type of rating system won’t be implemented overnight. It will have to undergo a limited test and be modified to fix any problems that turn up. Then, I think it should be run concurrently with the present OER system for a couple of years. During this time a data base would be building, from which group adjustments could be determined. Also, detailed comparisons between the systems would be made with a view to further modifying the new system and integrating those portions of the old system still felt necessary.

Jones paused a moment. “One thing bothers me, Thaddeus. If your staff goes along with the idea, how do we get the word back to Earth?”

It was the older man’s turn to be confident. “That’s not a great problem, Major. Of course, it will depend on the priority we get. When the situation warrants, our communications system can respond very effectively. Remember Moses and the Ten Commandments?”

Major Jones was impressed. “I see what you mean. What a way to sell a new OER system! But I don’t suppose we’ll rate that kind of priority?”

“No, I think we’ll have to settle for something less, like planting the idea in the mind of one of your troops on Earth and letting him advertise it.”

“How about the Chief of Staff?” Jones suggested.

“I doubt it, Major. I think our channels to him are already loaded with more important traffic.”

“Well, then,” Jones said, “it should be through someone who has no personal quarrel with the present system, an officer who already has it made. How about one of the Air War College students? Those guys have it made, and they’re always sounding off about something or other in papers they have to write.”

Thaddeus smiled. “I detect a note of bitterness or jealousy, but I like your suggestion. I’ll recommend it. And I’ll see what can be done to hurry along your entrance approval because I may need you to back me up on the briefings.”

“That would be swell of you, sir,” said the major as the two men rose. “Incidentally, did I understand you to say, the other day, that you get advance word on pending arrivals up here?”

“Yes, Major, normally six months in advance. However, we can go beyond that if there’s a reason. What did you have in mind?”

“Oh, nothing much, I was just wondering about a girl I once knew in Naples.”

 Tan Son Nhut Airfield, Vietnam

Contributor

Colonel John T. McCabe (M.S., Massachusetts Institute of Technology) is Vice Commander, First Weather Group, Tan Son Nhut Air Base, Vietnam. After aviation cadet meteorological training, he was commissioned and during World War II was assigned to the Southwest Pacific area as staff weather officer to troop carrier units. On inactive status (1947-50) he was a U. S. Weather Bureau forecaster at Seattle, Washington. Since being recalled, he has served as a weather officer in French Morocco, Japan, Oregon, and in the Washington, D.C., area. Colonel McCabe is a graduate of Air Command and Staff College and Air War College class of 1969.

Disclaimer

The conclusions and opinions expressed in this document are those of the author cultivated in the freedom of expression, academic environment of Air University. They do not reflect the official position of the U.S. Government, Department of Defense, the United States Air Force or the Air University.

Home Page | Feedback? Email the Editor