Air University Review, November-December 1969

Necessary Elements of An Equipment
Development and Test Program

Andrew S. Carten, Jr.

New Air Force equipment embodying advanced concepts almost invariably owes its existence to the fact that a scientist or engineer had faith in a novel idea. He nurtured the idea to the point where feasibility was demonstrated, and he communicated his success to the ultimate user, who saw in the idea the answer to an operational need. A series of administrative and technical actions then followed, which culminated in the delivery of the fully developed equipment to the field, complete with supporting data, spares, and ancillary items.

One might ask how the scientist or engineer happened to be working on an idea which so nicely satisfied an operational need. This is the ideal situation, of course, from both the developer’s point of view and that of the user. In the real world, it seldom happens that operational personnel can pose a problem to the research and development community and receive a ready-made answer. Although this is an end to be pursued, a more achievable aim is the establishment of a development program in which operational needs are anticipated to the maximum extent and appropriate action is initiated to advance the state of the art, where possible, to accommodate those needs. The stage is thus set for a potential full-scale development effort when the occasion arises, and a solid technical base is established for the overall program. For a program of this type to have any hope of fulfillment, it must be geared to a series of realistic goals. This brings us to the four essential elements of a truly successful development program: the technical objective, feasibility, the requirements statement, and the development plan.

The first essential element is a technical objective (TO), a published statement of specific goals in the exploratory development area. If a scientist is seeking to advance the state of the art in visibility measurements or in antenna design, for example, he is doing so in response to TO’S stating that advances in those fields are required. The TO’S, in effect, becomes the charter under which he conducts his investigation.

TO’S are general in nature and are not particularly adaptable to time phasing. This is in keeping with the independent, free-standing character of the research and development process and with the wide geographic and administrative separation between developer and user which the Air Force structure imposes. Nevertheless, TO’S are expected to relate to operational needs. The exploratory development projects built around them are approved on that basis. The degree of relevance is the key issue. The TO’S could mirror exactly and exclusively the formally expressed needs of the user, but this tends to be a self-defeating process since scientists become preoccupied with day-to-day operational problems in this case and the "ivory tower" atmosphere disappears, taking with it the objectivity and detachment desirable in R&D work. On the other hand, the TO’S could be so pure that they bear little or no relation to the user’s present or future situation, and the program’s output is of little value to anyone. The answer lies somewhere in between.

The catalyst that brings about the adoption of worthwhile TO’S is meaningful dialogue between the developer and the user. This is informal intercommand coordination of the highest order, consisting of almost daily verbal or written communication between the two groups on the problems and findings of each. This dialogue is best carried out by spokesmen for the two groups each knowledgeable in his own area and able to speak for his group.

The developer and user see life from two entirely different viewpoints. The developer takes a detached view of operational problems. He looks at them academically as an endless challenge to his ingenuity and imagination but of no real concern to him on a day-to-day basis. He visualizes new approaches, new instruments, new observational platforms in terms of their functional excellence, without worrying about how they will be paid for and maintained. His mind races ahead five or ten years as a matter of course. His special awareness of advances taking place in related technical fields allows him to project into being techniques of which the user usually has limited knowledge.

The user, on the other hand, is generally overburdened with daily operational crises, sometimes global in nature. He is hard put to maintain his existing capabilities because of personnel turnover, and he is under constant pressure to acquire new capabilities. His operational equipment is usually a collection of old and new devices, many of them plagued with deficiencies which reduce their reliability and usefulness and further aggravate his problem of maintaining his capability at an acceptably high standard. He is painfully aware of the long time required to get newer, improved equipment into the inventory. Thus, his viewpoint tends to favor the present time period and the immediate future. Under these circumstances, his five- to ten-year planning documents can very easily be either mere projection of existing techniques or unrealistic expressions of advances that will never materialize.¹

If a close exchange of information flourishes between the two groups, the user can maintain a realistic awareness of the latest state of the art and can plan intelligently for improved mission capabilities. He is encouraged to widen his horizons to include promising new techniques, of which he is not fully aware. From the developer’s point of view, a continuous window is made available to the operational world which illuminates his investigations and keeps them in touch with--although not limited to--actual problem areas. The developer’s technical objectives, by virtue of this dialogue, are scaled to the heights to which the user may wish or may be persuaded to climb. Priorities can be agreed upon which will discourage the developer from devoting his resources to efforts of no ultimate value to the user.

Relating these conditions to the exploratory development process shows that the essentiality of the technical objectives lies in the degree to which they provide practical guidance to the development effort and shape its output into a form compatible with the interests of the user. The ideal objectives are those which create and sustain the desired solid technical base from which will emanate a steady flow of advanced techniques and equipment capable of upgrading the mission effectiveness of the user. (The phraseology employed here permits considerable flexibility in the coupling of the R&D effort to the user's needs, but this is highly desirable. It allows the scientist to consider all possible solutions to an expressed need and guarantees the introduction of new ideas.)

The dialogue informs the user of the proper moment to begin the administrative actions required to bring specific new equipment into the inventory. This comes about in two ways: the developer may conclude that it is in the interest of the user to adopt a new technique whose feasibility has been established after extensive investigation but for which there is no officially stated requirement; or the user may seek the developer’s opinion as to whether equipment embodying a new technique is feasible and capable of development.

In the following discussion of feasibility, which is the second essential element in the planning process, I shall draw upon my experience in the meteorological equipment development field.

It is an oversimplification to say that a new technique is feasible because it can be shown to work. Mere performance of the specified function is an insufficient basis on which to determine feasibility. There are many additional considerations: Are the conclusions reached with respect to successful performance based on statistically meaningful test results? Where instrumentation is involved, are the measurements obtained by the new technique accurate and repeatable? Do they cover the entire range of interest? Is there any significant lag or hysteresis? How are the new measurements related to measurements taken with standard instrumentation? Is there a diurnal effect? Are the outputs compatible with standard data-handling procedures? Can the technique be embodied in instrumentation that is reasonably priced for the intended application? Can the proposed instrumentation be easily calibrated? Can the calibration be maintained? And so on.

These questions indicate the depth and the importance of the feasibility concept. They are part of the check list that a scientist must apply to new techniques before he can declare them feasible or within the state of the art.

Determination of feasibility is mandatory before serious consideration can be given by the user to requesting any development involving a new technique. Thus, feasibility is more than an essential element in the development process. It is actually the keystone. If the user seeks to employ an unproven technique, the required development effort will be experimental and risky, requiring a state-of-the-art advance. On the other hand, if feasibility is clear, the resultant full-scale engineering development will be straightforward and safe, with predictable milestones.

It follows that the feasibility determination should decide whether exploratory development or engineering development funds will be allocated for a particular effort, the type of managerial controls to be employed, and the long-term or short-term nature of the development. If the feasibility concept is compromised and a technique is erroneously stated to be qualified for full-scale engineering development, despite the absence of demonstrated feasibility, an unfortunate malassignment is the inevitable result. Development milestones and cost figures are established which prove to be meaningless, and the user, in most such instances, receives either no equipment or equipment that is late, underbudgeted, and often less than satisfactory. It cannot be emphasized too strongly that feasibility must be established if the engineering development program is to be successful in terms of end-product quality, reasonable production lead times, and budgetary control.²

The feasibility check list should be applied before any experimentation is done. By this procedure many initially attractive ideas which arise in the course of the developer/user dialogue can be eliminated or declared feasible at virtually no cost. This is made possible by using the fund of knowledge accumulated by the developer. Some ideas, of course, are not classifiable by the check-list type of screening process. If they are agreed upon as desirable technical objectives, they should undergo experimental verification, and the possibility of development failures must be considered in planning the eventual use of the techniques being investigated.

An intermediate step is sometimes required to bridge the gap between exploratory and engineering development. This step is called Advanced Development and is particularly useful for those situations where a proposed system’s feasibility cannot be truly proven in exploratory development because of financial and time constraints. It allows a concerted one-time effort to be undertaken in which all the loose ends of the previous efforts are tied together and the resultant system is subjected to a meaningful test cycle. Thus the decision to proceed or not to proceed to engineering development of the system can be made on relatively safe grounds.

If it has been established that feasibility is essential to a successful equipment development program, it follows that a large government agency attempting to mount such a program must have in its organization a permanent and competent group of scientists and engineers working in the exploratory development area. This group must be continuously supported with adequate funding and physical facilities. In return, the group is expected to serve as a center of technical expertise, fully qualified to provide the feasibility determination and to generate new feasible techniques capable of being developed into operational instruments.

It also follows that there must be a second group of competent engineers knowledgeable in their technical area and dedicated to the latter-stage engineering development task. They, too, must be provided with the resources required for their special task. As an alternative, the original investigators can be allowed to pursue their endeavors into the advanced stages of development, but because they are usually not well versed in the administrative details of those stages the exploratory development effort is diluted.

Normally, a hand-off of development responsibility to the engineering development group occurs with the declaration of feasibility. The original investigators now assume the less active role of technical consultants. The need for close communication between the two groups is obvious, as is the need that each group appreciate the special competences as well as the differing goals and missions of the other. In the absence of such an understanding, there is a real danger that the final development will fail to live up to earlier promises.

The third vital element, the requirements statement, is the document which provides a recognizable shape to a development. Let us assume that the user must add to his operational capability a new measuring technique which has been shown to be feasible. The user must make the first move. He may prepare a formal requirements statement, which in the Air Force is called a Required Operational Capability (ROC), or he may express his need less formally as an immediate operational requirement or a test-range instrumentation requirement. The requirements statement expresses the deficiency that exists in the user’s measurement capability, explains why this deficiency must be corrected, spells out the specific performance characteristics of the equipment needed and the time period by which it must be available, and, finally, discusses the known feasible approaches, based on knowledge acquired from the developer. It is staffed throughout the user’s command and coordinated with the developer to insure consistency with the state of the art. The developer assists the user to keep the requirement within reasonable bounds. Accuracies are often readjusted to what has been obtainable experimentally. Measurement ranges are similarly treated. Time phasing is sometimes introduced, the user being advised to settle for less ambitious goals initially and to wait for further advances in the state of the art before demanding the ultimate in performance. The investigators also estimate the probable development cost and the time required for completion.

This coordination is a healthy process and usually results in a reasonable requirements document. Once finalized, the document goes forward to Headquarters USAF for approval. If the user’s justification is strong enough, a directive is issued calling for a development plan to be prepared by the activity responsible for engineering development.

The development plan, the fourth essential element in the process, is the blueprint for the latter stages of development, covering all aspects—performance, technical approach, management, documentation, test, resources, production quantities, milestones, etc. When it is coordinated with the user and the original investigators and is approved by Headquarters USAF, funds are allocated, and the full-scale engineering development is begun. Specifications are prepared, a contract is let, preproduction models or first articles are designed and fabricated for test. The preproduction models are tested, ideally in conjunction with the original investigators, against the specified criteria. The user is provided with models for his own test. Specification adjustments and design refinements are made on the basis of the test results, and a firm configuration is established. The production "go ahead" is then given. Finally, the resultant items enter the inventory in accordance with the established milestones.

The production release effectively terminates the development process. The new equipment coming off the assembly line should be functionally suitable and highly reliable. This should be the case if the development plan was carefully prepared and coordinated and first article testing and documentation were realistic and thorough.

The latter stages of development are very essential steps, requiring meticulous planning and execution. They are well-documented steps for which instructions and directives abound. They tend to be highly visible, and the coupling with the user is very tight. Therefore, exhaustive treatment of these stages does not appear warranted in this discussion.

The philosophy I have presented here is a distillation of my impressions from many years of effort in the exploratory development phase of meteorological equipment. Since the precepts set forth were not always followed in the developments with which I have been associated, numerous problem areas were encountered which in retrospect underscore the essentiality of the key elements I have identified. A few examples, described in general terms, will illustrate the point.

The engineering development phase is straightforward and predictable if the essential steps leading up to that stage are followed. I have witnessed several instances where the latter stages of development went astray because insufficient attention had been given to the feasibility concept. In these situations, the engineering development activity made the administrative decision that feasibility had been established, despite evidence to the contrary. User eagerness to initiate final development led to the calculated risk that an engineering development or production contractor could complete the work of the original investigators. Lest this article appear to single out a specific group for criticism, I should explain that in some instances the original investigators also had the final engineering responsibility. In other cases, where they did not, they concurred in the feasibility decision. In any event, the problem lay in the fact that in applying the rules for contractor selection inadequate consideration was given to the need for scientific depth on the part of would-be contractors, in addition to engineering and production skills. The bidders, in turn, failed to realize or appreciate the fact that some residual exploratory development was involved.

In almost every case of this type, serious problems were encountered. A time delay of one or two years in delivery of equipment was common. A less frequent but more severe penalty was the delivery of equipment that was of marginal quality and of very little use in the field. On at least one occasion, outright cancellation of the contract was required, and the user received no equipment at all.

Another situation which has occurred is a change of heart on the part of the user towards a development that has proceeded to the demonstration of feasibility, or beyond, with the understanding that the technique was of interest to the user. The developer is shocked to discover that the user has lost interest and the work has no further relevance to the user’s needs. This situation indicates that a meaningful dialogue was not properly established between the user and the developer or that it was not properly maintained. In those fortunately rare cases where the development was allowed to proceed into the final stages prior to the expression of non-interest, the user’s requirements statement must come under scrutiny. The obvious conclusion is that the requirement was at best marginal.

On occasion, exploratory development is initiated on the basis of an investigator’s personal preference, with no real attempt to relate it to user needs or capabilities. It is not surprising that the user would express little or no interest in the technique being investigated—unless, of course, the investigator’s choice happened to fit the user’s needs. In a well-conducted program, in which the user/developer dialogue is well established and the technical objectives are truly relevant, this seldom occurs.

In contrast to the negative aspects of these cases, many new items of equipment in the meteorology field, for example, are being used routinely and satisfactorily today for operational measurements of atmospheric parameters precisely because the essential development steps were carefully followed. The meteorological rocket systems in use at the various test ranges are perhaps an outstanding example. The user/developer dialogue there has been exceptionally close, and the results have shown the value of such careful coordination. The rocket sounding system results from a project in which the original investigators also carried out the final engineering aspects of the development. Similar examples can be cited for meteorological equipments in which the responsibility for development was handed off by the original investigators to an engineering development group.

Developers and users in other technical areas within the Air Force will recognize the general types of situation described here and could easily add their own specific details. They would undoubtedly agree that successful equipment developments do not just happen and that a carefully planned program is mandatory. I trust they share my conviction that a farsighted exploratory development program, based on relevant technical objectives and scrupulously adhering to the feasibility concept, will lead to a successful final development--if the user carefully prepares the requirements document and if the resultant development plan is faithful to the user’s requirements and to the experimental findings which led to the declaration of feasibility.

Notes

1. For an interesting discussion of the foresight which attends the scientist’s thought processes, see Chapter 9 of Science and Government by C. P. Snow, which was separately published in Science and Technology magazine, January 1969.

2. For a strong non-Air Force point of view on this subject, see Aviation Week and Space Technology, February 17, 1969. In the feature editorial, Robert Hotz states that the Department of Defense "total package" concept has foundered or is in deep trouble in several instances because of the assumption that no state-of-the-art advances were called for and thus 10-year costing was possible. I am not in a position to refute or support Mr. Hotz’s claim, but I believe his editorial is relevant to the need for establishing feasibility prior to firming up other parameters.

The photographs of fog-seeding operations referred to on page 47 of Major Thomas A. Studer’s article "Weather Modification in Support of Military Operations" in our September-October issue were deleted intentionally; the same photographs appeared in similar context in the July 1969 issue of Aerospace Safety. Unfortunately, we failed to incorporate the necessary text revisions into the pageproofs.

Andrew S. Carten, Jr. (M.S., Tufts University) is Chief, Equipment Engineering and Evaluation Branch, Aerospace Instrumentation Laboratory, Air Force Cambridge Research Laboratories. After graduation from the USAAF Weather Officer School, Grand Rapids, Michigan, in 1943, he served as staff weather officer, Eighth Air Force. He was an engineer with the Bendix Corporation until joining the Air Force Cambridge Research Center in 1954 as Chief, Design Engineering Branch, Atmospheric Devices Laboratory. In 1960 Mr. Carten assumed his present position. He is author of two articles on meteorological measurements published in the Bulletin of the American Meteorological Society and of numerous papers presented at meteorological conferences.