Document created: 2 September 03
Air University Review, May-June 1975

In 1907 the Army Signal Corps asked for bids for a flying machine. The one-page bid covered everything from how to make drawings to what the minimum performance should be. On a specified day, the competing craft were to be brought to Fort Myer and demonstrated to the customer. The Wright brothers won the contract to supply an "aeroplane" and signed a simple one-page contract for its delivery. Evaluation of research and development was quite simple: Did the end item work?

Today evaluation of R & D is not as simple. Most projects are so complex that they take years and often millions of dollars to complete. There are more problems than there are people, money and time to solve them. Many R & D programs are focused on systems that will not be operational for years and then may bear little resemblance to the original concept. Other technology may be advancing toward a system that will be canceled along the way. Today the Air Force manager of research and development faces an enormous challenge in attempting to insure that limited resources are used most efficiently to give the greatest payoff. To meet this challenge, the R & D manager must become a master of the art and science of research and development project evaluation.

Out of any research effort come many things--technology, data, information, hardware, and results. These are accompanied by problem--technical, schedule, and financial. It is the job of the R&D manager to appraise this output against some list of criteria that includes the original objectives of the research undertaking and the value of the research results versus the resources invested. He must also determine if the results will be available on time and will not duplicate the work of others. He must know if what is being done has relevance to future military needs and saystems. This "output against criteria" appraisal, then, constitutes research and development evaluation. Now let us look at the whys, whats, whos, and hows of the evaluation process.

The research and development cycle, from an idea to an operational aerospace vehicle, is long, costly, and filled with pitfalls. The job of the military research and development community is to develop aerospace systems for the various using commands, systems needed to maintain and improve the United States' military posture. As part of the R&D establishment, the research laboratories are given the job of translating basic research results and fundamental technical ideas into proven technology that can be used in future systems with minimum risk. Also of importance is the laboratories’ responsibility to assist technically in solving problems with weapon systems under development and even those already in the field. Thus research projects must be reviewed continually to insure they are progressing on course to desired goals and are obtaining benefit from every bit of knowledge available.

It would be nice to be able to solve every technical problem that comes along and pursue every technical breakthrough. But this is impossible because in the Air Force, and for that matter in the entire technical community, there is a resource shortage. Today the shortage is acute, with inflation, reductions in defense money, and people cutbacks all taking their toll. Coupled with this is the fact that today's technology is so complex and costly. Therefore, in miserly fashion, the various levels of management must constantly insure that resources are being spent in an optimum manner. The biggest challenge is not whether a particular technical problem can be solved but whether we can afford to solve it.

Besides watching the financial picture, managers must take a hard look at each project and ask themselves the following questions:

Is it progressing to the objectives set?
Will the results be ready on time?
Have the needs and goals changed? Do the projects reflect the changes?
Are we duplicating someone else's work?
Can someone else be helped by doing a little more--perhaps another test condition or a minor change in approach?
Have all pertinent technology advances been included? Have we looked recently?
Are we reinventing the wheel?

Evaluations must be made by management to get an overall view of laboratory operations. First of all, to get an insight into the future. When long lead-time items, like facilities, are involved, they must be identified and budgeted many years in advance of actual need. Also needed is an insight into where major breakthroughs must be made before a piece of technology can advance to a usable stage. Management must look over its entire program to establish priorities based on a relative comparison between competing needs and programs. The management must also be sure the laboratory is covering risky but critically needed technology with options, alternatives, or backup programs.

While the main function of a military laboratory is to develop particular items of technology to satisfy future system requirements and to help solve today's critical problems, the laboratories must also maintain and improve the level of technology in its general area of responsibility. This will assure that the laboratory will be able to solve new problems when they occur. Such things as developing tools, like computer models and experimental equipment and techniques, are important. Also time and resources to cultivate new ideas must be allotted. Within the entire laboratory's budget, some new ideas with great potential must be pursued, even if now there does not appear to be any established end use. The whole laboratory operation must be evaluated to be sure that in the zeal to solve today’s problems the future has not been forgotten.

Research and development is evaluated at all levels of the military command structure. However, at each level the method and scope are different, since each level's objectives are different, At the higher headquarters level, i.e., non and the service headquarters, the need is for only key information over a broad area. Their evaluation is concerned with the overall scope of applicable research and development; for example, total resources and overall trends. The staff officers have too many projects to monitor to spend a great amount of time on the day-to-day problems or the technical details of any one project. These managers are concerned with achieving the proper balance in the total research program and seeing that all the various individual research efforts are properly integrated. However, they sometimes see the need to evaluate a particular area in greater detail, especially when a technology area becomes vital to a particular mission capability or when a breakthrough occurs that has some major implication or future capability.

Within the laboratory, each laboratory commander or director is responsible for the total research program in his laboratory's area of interest. This responsibility means not only millions of dollars in funds, hundreds of people, and upwards; of billions of dollars in facilities but the responsibility of assuring that research programs are responsive to military technology needs. Whether it be "signing off" a purchase request or the entire technology plan of his laboratory the commander and his staff must base their decision on a sound evaluation. Once projects are under way, evaluations must continue.

At lower levels of laboratory management, evaluation is really another word for good management practice. Because these managers are closer to the individual efforts, they can sift out the "soft" projects before they are proposed to boss. These soft projects might be too far out, too risky, or too expensive. Because lower-level managers have fewer efforts to track, they can keep abreast of day-day progress.

Outside the laboratory's chain of chain of command, many other organizations are constantly evaluating the laboratory's technology programs. These include the operational organizations that want to know what they can expect in the way of available technology so that they can formulate realistic system requirements. The organizations developing military systems review laboratory programs to see what proven technology items can be included in the systems they plan and develop to meet the user's needs. Finally, there are many advisory groups that can make, hopefully, unbiased reviews and critiques of laboratory operations. These groups, if appropriately picked for their expertise, can provide valuable assistance to the laboratory manager or for higher headquarters' evaluations. The important thing is that there be a sufficient amount of evaluation that includes both broad views and critical detailed searches.

Up to this point we have neglected the key person in the evaluation, the project engineer or scientist. This omission has been intentional, since we want to dwell more on his role. The project engineer is so important because it is he who either does the technical work or directs the progress of others. In other words, he is one who produces the results that the others evaluate. Furthermore, he has the important function of communicating results and progress. Finally, he is the expert on a particular technical subject, so he can best evaluate breakthroughs, plan the next step, and identify insurmountable problems. At this point, it is appropriate to mention the project engineer’s role with respect to contracted research. He is a contract manager or technical director, not merely a "contract monitor." The project engineer uses evaluations to find out what is going on and, based on evaluations, takes action by reporting to higher managers, or by directing others to do something, or doing something himself.

Unfortunately, the project engineer is often too close to the problem to put it in its proper perspective. Therefore, periodically, higher levels of management must evaluate all projects critically and weed out worthless, costly, or unneeded efforts and must insure that the approach for each project is the best one to reach the final goal.

The projects for which the laboratory is responsible get the most intensive scrutiny. These are the contracted and in-house efforts for which the laboratory is supplying its funds, people, and facilities. But there are other technical efforts that laboratory personnel must review and evaluate--granted more informally and passively--as part of their job. These are the efforts under the auspices of other organizations that have a bearing on the laboratory's technical areas. They must be reviewed on a continuing basis to benefit from their results and to prevent duplication. These corollary efforts include:

• Independent research and development (IR &D)--research by contractors funded by the government but not directly controlled by it. Nevertheless, our laboratories periodically evaluate the IR&D, providing guidance as to the most relevant areas for the contractors to pursue and assistance in picking the optimum approach.
• Research pursued by other military laboratories.
• Research under way within the sister services and NASA.
• Other research, usually more basic, conducted by universities and research institutes.

The evaluation of research and development has to occur on a continuing basis: (a) before a project starts, to insure that approaches are sound, results are achievable, and there will be a usable payoff at the end; (b) while the project is under way, to make sure it is on track; and (c) finally, at completion, to assure that goals are attained and results get into the hands of the users, and to determine what is the next step the technology should take. However, evaluations should concentrate on the first two periods, since at the end of the effort not much can be done. In reality, technology planners must lay out programs several years in advance and thus must extrapolate results before a program is completed.

All levels of management, from the project engineer overseeing the work at a contractor's facility to the headquarters staff officer, should recognize that evaluations are a management tool and nor an end in themselves. It is very easy to ask for so much information in the form of briefings and reports that the people doing the work have little time to make progress. Therefore, considerable thought has to be given to planning an evaluation scheme that provides timely information with a minimum of interference to technical progress.

Evaluations can he divided into two types: first, those that can be scheduled, or the steady state operation; and second, the unscheduled evaluations, or the perturbations on the steady state mode.

A look at the scheduled evaluations shows that a logical planning of evaluations can and should take place to coincide with the budget planning cycle. Each laboratory must decide annually the programs it will pursue in subsequent fiscal years. This planning is usually firmed up in some type of formal planning document. The plan is usually completed a year before the fiscal year during which the funds will be spent and the work actually done. To complete this plan, the laboratory manager, with assistance from his staff and operating divisions, must concentrate on evaluating the validity of future projects, scrutinizing them for their technical feasibility and their relevance to the overall laboratory mission. However, since these new programs are usually based on previous ones, current and even completed programs also are looked at during the planning cycle.

Through scheduled evaluations, the manager can find out what is going on in his current programs. Therefore, at least once a year, he should look at each program for which he is responsible, emphasizing the technical results and the progress toward planned goals. These evaluations are called, for example, "project reviews." On down the organization hierarchy, the evaluations should become more detailed and occur more frequently. They also become less formal, and if the group is small enough the evaluations can be eliminated if the manager has a day-to-day knowledge of all his project engineers' efforts.

The important thing here is that, except where the supervisor can get intimately involved in all his subordinates’ projects, all projects get a periodic review. While evaluation or management by exception (that is, leaving a project alone until something abnormal occur might work in a more routine situation, it should not he the mode of operation in the research and development environment because:

• The technical workers are close to the project that they might fail see where the potential problems are, the nonoptimum approaches, or the significance of the results.
• Some people will not report on what they are doing unless asked.
• More experienced people, especially those with broad backgrounds, can help relate results to actual needs, prevent duplication, and point out poor approaches.
• Just the requirement to explain his project and results can force the project engineer to think out his ideas in a more detailed fashion.
• Periodically required reviews prevent something from "falling through the crack."

From an efficiency standpoint, it would be nice to be able to operate under the steady state mode, but in the real world this is not possible. Many things can happen that force us to reevaluate a laboratory's technology program. These outside perturbations usually have several things in common: they usually require urgent responses, are not predictable, and could have a major impact on a laboratory's plans and programs.

For example, a weapon system toward which a laboratory's technology efforts are aimed may undergo a major redirections. It may require a change in technology, may be given a higher priority resulting in a speed-up of the supporting technology efforts or a substitution of less risky technology items, or it may be canceled, requiring the laboratory to change the direction of related programs or to delete them also.

A cutback in personnel or laboratory budget may require a review of where budget cuts can best be made. An unexpected technical breakthrough may necessitate a change in emphasis to exploit the breakthrough fully. The converse of this is an unattainable goal requiring reduction of emphasis in the particular technical area and perhaps an added interest in another option. Before any decision can be made, the responsible manager has to make an evaluation. Because of the short time involved, the project engineer and his immediate management must have information at hand so that a timely and complete evaluation can be made.

Now that we have established the whys, whos, and whens of evaluation, we come to the most important and difficult part: how. This part is so difficult because

—end results are usually very difficult to define, if indeed they are even known. Often not enough is known to be able even to establish realistic goals.
—military research and development environment is subject to so many perturbations that cannot be foreseen.
—we may not even know where technology will be used.
—the most important results of an effort may be not what was originally intended but something that was discovered along the way, i.e., spin-offs.
—there is no realistic yardstick to use to measure a technology program.
—comparison of planned and actual schedules and resource expenditures does not tell the entire story; it measures only the input.
—it is difficult to measure brainpower, general knowledge, or expertise, that is, the ability to solve the next problem.

Attempts have been made to mechanize the evaluation process. Mechanization schemes usually involve definition of criteria, preassignment of weights to each of the criteria, scoring each phase of a project against the criteria, and then looking at the resulting score and comparing this against a perfect score. While this looks like an ideal method of making a very subjective topic more objective, mechanization of evaluation is filled with pitfalls. First of all, establishing the criteria-weight relationship is very difficult, if not impossible, since every project is unique. It is quite easy to assign weight criteria and then score such things as timely reporting, quality of reports, meeting of schedules, and number of financial overruns. But how do you score the technical results? With a mechanized system with preassigned scoring factors, a timely, well-presented, and well-reported project that was completed within project costs and that met stated objectives that were ill-defined at the program onset could receive an A-number-l score even though the results were mediocre and had little actual usefulness.

However, the history of technology is filled with engineers and scientists who got a project and really "ran with the ball," inventing new concepts, exceeding planned objectives and goals, and making significant contributions to increasing military capabilities. Because of the expanded scope of the project and the enthusiasm of both the investigator and the users, more resources were spent than planned, schedules were slipped to allow more work to be done, and results spread to the final users as they became available; so any final report was anticlimactic and probably late or poorly presented. Under a predetermined evaluation scheme, this latter effort would probably score poorly in comparison with the project first discussed. Yet in the real world, the benefit from the latter project might be an order of magnitude better than that from the first project.

Finally, in mechanizing the evaluation scheme, we could spend too much time in determining the criteria weight relationships and in trying to put numerical values on the various parts and results of the work--time that could be spent more profitably in judging progress and results against the real world environment and in directing the effort to obtain the maximum payoff.

The message is simply this. Mechanization of the information needed for the evaluation is a worthy objective, allowing timely and readily available visibility to the evaluators. The actual evaluation is a human activity

Evaluation requires experience, depth of technical knowledge, an understanding of what is going on in the military and the R&D community, and sometimes merely a gut feel. These are things you cannot leave to inexperienced people, and you cannot program them on a computer.

Imperfect as they might be, some criteria must be used for measuring technical progress and value received from resource investments. These criteria can be divided into two types: technical goals and objectives, which are hard to measure; and efficiency of resource expenditure, which is somewhat easier to determine.

Objectives and Goals. The results of any of our laboratories must be judged against the long-range objectives and goals of the users of military systems. The goals for each technical area and individual project must be consistent with the overall laboratory goal. In establishing the objectives for a technical effort, we should consider certain things so that measurement of attainment of the goal made easier:

• The objective should say what is to be achieved and not how it will be achieved.
• The objective should be a clear-cut end point, one that is obvious when it has been reached. A go/no-go criterion is ideal.
• Accomplishment is more easily measured for a quantified objective, that is, a numerical value like a rocket engine specific impulse, a unit cost for a guidance system, or the resolution of a remote sensor. The quantified objective allows the evaluator to determine how close he came if the original goal was not achieved.

Resources and Schedule. At the start of any project, the project engineer must forecast how long it will take to reach the goals and how many dollars or man-hours of effort we have to be consumed. These estimates should be realistic, requiring the project engineer to know the objectives and the approach to be taken. Too long a period to do the job can be as bad as trying to rush the job. Research takes time if it is to have depth, but too much time can not only produce results a problem that no longer exists but cause the people performing the work to interest. Inadequate resources can result in just skimming the surface of each subtask to satisfy the requirement that all tasks be completed. Too many people cause inefficiencies, and too much money is wasteful. Most R&D projects are endless and could be researched in infinite depth as long as there were people to work them or money to spend. The objective is to set aside only the resources needed to provide what is wanted by the user. Once time and resource expenditure schedules are set, it becomes a relatively easy job to evaluate these on some type of resource-milestone chart. A glance will show when a schedule is slipped or when money is going to run out, and thus when the manager must take action.

Other Criteria. The other criteria are the real hard ones to be set, let alone evaluate. Evaluation of these is based strictly on experience and knowledge of the environment. A good R&D manager has a feel for this that often he cannot define. These criteria include:

• Relevance to the military mission. This requires the evaluator to have a broad understanding of what is happening outside his laboratory.
• Duplication. Not only must the evaluator know what he is doing but he must keep up with what others are doing.
• Soundness of approach. This comes from experience and technical competence.
• Cost/performance. In today's cost-conscious environment, an increase in performance has to be weighed in light of total systems cost, i.e., development, acquisition, and operation. Often to evaluate this criterion, a substudy must be performed to see what a few seconds of increase in rocket engine specific impulse or a few inches in resolution will cost over the entire lifetime of a military system.
• Resources expended/payoff.

The resources expended in investigating a piece of technology make a relatively easy item for an experienced laboratory manager to estimate. The denominator is the hard one to come up with because it is often intangible, and the payoff is really unknown since an item of technology can have such a wide range of application. For example, the technology developed for valves and plumbing in a liquid rocket engine is now being applied to making railroad tank cars safer.

It is quite clear that the most valuable tools of the evaluator are his awareness, experience, and knowledge. The most important part of the evaluation process is to get all the information needed to evaluate in front of the evaluator, whether he be the project engineer or the service secretary.

The project engineer or scientist is the keystone in the evaluation process, for it is with him that most of the information rests. He must efficiently obtain the information on which to base evaluations, and he is the one who presents it to high-level managers.

The project engineer managing a contractual effort obtains his information through progress reports and visits to contractors. It is his responsibility to develop a rapport with the contractor so that he is constantly aware of what is going on in his contract. The project engineer must cultivate a relationship with the contractor whereby the contractor will honestly report progress and problems, will identify where judgment has been applied in interpreting results, where technical difficulties still exist, and what is the level of confidence in the accuracy of the results. To be aware of related efforts, the project manager must follow corollary projects in his own laboratory and with other organizations working the particular area through visits, reviewing of the technical literature, and attending appropriate symposiums. The project engineer must also obtain an understanding of the system that is the end item where his project will find its application. In short, he must become the expert on his contract, for his knowledge forms the basis for all higher-level evaluations. On in-house projects the responsibilities are essentially the same except that now he is the one doing the research and preparing the reports and documenting the action.

The evaluation of research and development projects is a very vital part of the military systems acquisition cycle, especially in today's environment of increasing technical complexity and dwindling resources. The R&D manager must become a master of the art of R&D evaluation. This is truly an art, since no "cookbook" formula has yet been developed to prescribe how to evaluate. Evaluation is based on experience, technical knowledge, and sometimes pure gut feel. The most important thing is to have proper information available to make the evaluation. Thus the project engineer, the person closest to the work, is the most important link in the evaluation process.

Contributor