Document created: 21 April 03
Air University Review, March-April 1976

Can Academic Management 
Research be Profitable?

Many people, over the years, have questioned the value of academic management research conducted by Air Force students in graduate or professional education programs. There has been little rebuttal because, on the whole, it is almost impossible to trace the value of academic research. A stronger challenge, though, has been to question the ultimate value of the research findings to Air Force functioning. Does academic research just carry on traditional educational methodology or can it be both educationally and organizationally valuable? It is my belief that academic research can be both, which is the reason for this article. Unfortunately, academic management research is not as valuable organizationally as it should be. The reason for diminished practical value is that the majority of both academic and field research is incomplete. The "incomplete standard" for management research was caused by inadequate time, inaccessible data, and failure to concentrate effort on long-term priority problems.

I will support these assertions by discussing the ideal research process, the incomplete research process, and the problems with time and data, which tend to pressure both military academic institutions and field agencies into supporting an incomplete research process. The discussion will end with proposed solutions that were found to be useful.

The ideal management research process may be defined through six stages. The stages are represented in Figure 1 with the understanding that there is no clear-cut point of separation between stages. In fact, the process is more correctly conceived as a never-ending cycle through several levels of abstraction and generalization. For example, we might identify our problem as the difficulty in predicting the costs of a weapon system in acquisition. The requirement for cost prediction is nothing new--it has been with us a long time and will probably always be a part of the managerial job. Cost prediction, though, is a symptom. An example of the many problems in the cost of weapons is risk analysis. This risk analysis example will be carried through the six stages.

Stage 1: Abstract Problem. The process begins with the expression of an abstract problem. If we knew more about the risk involved with each step in the acquisition process as related to final cost, we might be able to do a better job of predicting weapon system costs. Someone coins a term and we begin to refer to this concept as "risk analysis." No one knows exactly what it is, and there are likely to be as many concepts as there are people who use the term. Nevertheless, knowledge about risk analysis may provide a predictive tool for the cost of weapons. The term is bandied around, and soon we have a management research problem called risk analysis.

Stage 2: Concept Building. Once risk analysis becomes an identifiable problem, the first research efforts are concerned with descriptive syntheses of belief from any sources of available knowledge merely to define the problem and render tentative and abstract solutions. The conclusions to these studies are nothing more than generalized predictions or hypotheses for the future. Many journal articles report on stage 2 concept-building research.

With the passage of time, and after several attempts, a concept inclusive of most of the present knowledge of risk analysis will emerge. The generally accepted belief about risk analysis, at this stage of the game, is based upon a synthesis of old knowledge from many disciplines applied to hypothetical situations. As such, it is inferred knowledge--a theory.

Stage 3: Testing for Concept Validity. Several concepts of risk analysis that predict relationships gradually emerge. Research on risk analysis should move toward the generation of numerous studies that test parts of these predictive concepts in specific "real world" situations. For example, several studies might concentrate on testing the predicted sources of risk. Other studies might concentrate on predicted risk variables by contractor characteristic variables, and so on. For each of several major subdivisions of risk analysis, we should generate a number of studies that test theoretical predictions in specific real-world instances using real-world data. Over time, the many individual studies would derive and empirically test predictions from the risk analysis concepts in specific instances.

The first two stages consisted of old knowledge put together in new ways to arrive at hypothetical solutions. It is important to note that stage 3 is the first research in which new knowledge is being created with each test for the support of a predictive hypothesis. The creation of new knowledge is the key to real--world progress and whether management is or can become science.

Stage 4: Building a Structured Concept. The testing stage consisted of many individual studies that, when taken together and synthesized, would tend to validate and define the parameters of a risk analysis theory in terms of generalization. In other words, in what situations and under what conditions does a risk analysis concept seem to predict cost relationships accurately? Now, a few studies should identify the research trends established by the new knowledge created by the many tests in stage 3. Here we hypothesize the future in terms of specific policy application to weapon systems acquisition programs. Many studies fall into this category of research. However, unless sufficient time has elapsed between two staff studies on the topic, adequate new knowledge has probably not been generated through stage 3 testing to shed new light on the subject. This phenomenon could well be the basis for the phrase "reinventing the wheel."

Stage 5: Field Testing. If we were ever to codify risk analysis to the point of implementation, such a management program should be field tested in a representative situation before wholesale policy change is directed. Field testing has always been accepted for hardware but often seems to be ignored for new management programs.

Stage 6: Implementation or Practice. The implementation stage, while a part of the ideal research process, is not normally a function of researchers. An agency, aware of the research, would implement findings because the original symptom still exists and the new knowledge created by research would appear to offer a solution. Researchers would probably continue their analysis of associated problems in a continuing research cycle until theory and practice merge in complete knowledge about the problem.

If the ideal research process is a valid construct, then a conclusion related to the role of student academic research follows. The majority of student studies should be focused on the concept testing of stage 3 where a few broad-scoped abstract theories can in a sense be narrowed and tested in many specific situations to create new bits of knowledge about a theory's predictive validity and the limits of generalization. Only after enough testing studies have been conducted can the new knowledge created by the tests be synthesized into more structured predictions in stage 4, which are appropriate for field testing and implementation.

Compare the ideal research process to the actual situation as it often exists today. The comparison is depicted in Figure 2. The testing-of-concepts stage of the ideal research process is often slighted in both academic and field research; and as a result, much of our research has been of the stage 2 concept-building and the stage 4 staff study variety, without adequate generation of new knowledge through testing in stage 3. These theory-building studies of stages 2 and 4 do not create new knowledge, but rather the results are in a sense predictions derived by inference or relationships from a synthesis of old knowledge. Much of that old knowledge is based on theory, authority, personal experience, and expert opinion. Most of today's magazine and journal articles on management report on research that is based upon a review of the existing knowledge about the problem. Upon examination, the existing knowledge on which these articles are based is not new knowledge gained through tested predictions but rather old knowledge from retrospective analysis.

Decisions on real world problems are of necessity based upon staff study predictions rather than on fully tested hypotheses--all the more reason to emphasize theory testing as the standard for student academic research. For research to be useful, it should lead to management concepts that predict relationships. The only way to learn whether a prediction works is to put it to the stage 3 test in a real world environment. Academic research is one situation in which we can afford to test these predictions and in terms of the philosophy of science: create new knowledge.

Basic to the philosophy of scientific research is a belief that new knowledge can be created through testing predicted relationships. Without controlled testing we can only find knowledge through the slow process of retrospective historical synthesis in which we lack control and often data. Science forces the knowledge issue by (1) using historical synthesis to build a logical predictive relationship and then (2) finding or creating situations and data to determine if the prediction holds true. Science is indifferent to whether the prediction or hypothesis is supported, for in either case new knowledge is created by testing predictions.

The issue over hypothesis-testing research can cause friction between researchers and practitioners. The theory building in stage 2 and the more structured best possible solutions of staff study models in stage 4 merely synthesize old knowledge to infer possible solutions. Although these kinds of studies are often well done and useful for decision-makers, they are nevertheless incomplete research that adds little to knowledge needed in the modem management environment.

Figures 3 and 4 illustrate the idea of building knowledge. Figure 3 represents the state of today's incomplete knowledge about risk analysis. Figure 4 represents the ideal of complete knowledge when managers can predict with 100 percent accuracy. Note how structured and therefore more accurate is the predictive theory in Figure 4 as opposed to Figure 3. In Figure 4 managers can "practice what they preach" because there is no difference between theory and practice. Since the predictions are 100 percent accurate, we use the theory. We discard theories when they cease to be accurate predictors. For example, no one has "proved" Einstein's theory of relativity. It will always be a theory and only useful until replaced by a better predictive theory.

Unfortunately today's knowledge about risk analysis is as depicted in Figure 3. The continual cycling of stage 2 and stage 4 research studies depicted in Figures 1 and 2 will do little to create the new knowledge necessary to produce a more structured and hence more accurate predictive theory on risk analysis.

If one seeks solutions to problems, the ultimate answer is the total knowledge that scientific philosophers call law. The best research alternative appears to be the creation of new knowledge rapidly through integrated research projects in which the preponderance of research effort occurs in the tests of stage 3. To accomplish this, sound theory should first be developed on the basis of old knowledge and then tested for predictive validity through a series of situation investigations. Each of these stage 3 studies would provide possible modifications to theory as a result of the new knowledge gained from the test results.

The process is slow, but nevertheless faster than the alternative of recycling stage 2 and 4 studies until we find ourselves in the present stagnated situation. Today, our individual research studies consist mostly of stage 2 abstract management theories and stage 4 staff studies. One's favorite management theory is merely a matter of taste rather than a choice based upon the support of many validation-testing studies. Instead, we should have many stage 3 validation studies that when synthesized in stage 4 could produce a mature research trend.

The ideal research process is being aborted in academic institutions and field agencies alike. An overview of academic dissertations, theses, journal articles, and staff studies on management leads me to believe that the academic institutions and practitioners produce an overabundance of theory-building research and a paucity of theory-testing research. I cannot speak for practitioners in the field, but I can speak of the research conducted by academic institutions. Two major problems cause the low proportion of student-conducted theory-testing research. The problems are time and data. Time and data problems tend to pressure our academic institutions into accepting incomplete theory-building research as the standard for formal thesis and dissertation requirements.

Generally, a lack of time creates pressure for research implementation before the research process is complete. Adequate new knowledge has not been gained to establish research trends. Two dimensions of the time problem may help illustrate the pressure to support theory-building as opposed to theory-testing studies. One of the dimensions is time and the narrow topic; the other is time and the broad-scoped topic.

Time and the Narrow Topic. My experience with agencies in the field has led me to the conclusion that a good number of the narrow and researchable problems submitted to academic institutions are of the "put out the fire" variety. "The boss wants the waiting lines at the hospital cut in hall without spending additional funds." Companies and government agencies are not much interested in theory and seem unwilling to generalize the problem or its solution to a broader context. If the school can't produce a thesis and solve a problem within four months, forget it. A cursory look at the administrative time lag between topic selection, research, and reporting develops the fact of one and one-half to three years for a field agency to obtain research results from the academic community on a problem topic of immediate interest.

Time and the Broad-Scoped Topic. The topic of scope and magnitude is as in the example we have used: predicting costs of weapon systems in acquisition. Usable results through student research on such broad-scoped topics will likely require a number of years. Whether the ideal research cycle is accomplished through contract research, academic research, field research, or some combination thereof, it will literally require years to move through the concept-testing stage. Time is required to move through a mature research cycle and finally into field testing and possible implementation.

The time problem on broad-scoped topics is affected by many practical and political factors. Those faculty who use student research as a basis for consulting contracts may not have the time or desire to see their proposed solutions tested in any sense before implementation. Those military staff officers who complete staff studies may not have the time or desire to see their proposed solutions implemented. Once the research is implemented there is often no follow-up testing because the one shot research contract is over or interested personnel have moved on. Students who conduct theory-building and staff study research suffer. If their study was not part of a continuing project in which all elements of a complete research process were anticipated, students will be released with sheepskin in hand to perpetuate unknowingly the theory-building atrocity as a standard for complete research on future generations of management practitioners.

Given a period of time, the research efforts of our academic institutions can use student research to nibble away at testing some of the management theories. This ignores the problem of securing data for testing predictions. The biggest single reason for the low proportion of studies that create new knowledge through stage 3 concept testing is data--either a lack of data or their practical inaccessibility. Given the constraints on an average student, it is difficult enough for him to synthesize and infer a predictive concept, more so to find an existing situation and data that will provide a reasonable test in a real-world situation.

The majority of management concept-testing studies require military agencies or companies, in the near vicinity of the school, that will support the student research effort. When a cooperative agency is found, the student must conduct extensive a priori analysis merely to determine whether the available data can be converted to information appropriate for the objective testing of his prediction. Sometimes our students find limited support and empty promises when the supporting agency is asked to spend some time in the research effort or when the data begin to suggest results that do not support the sponsor's preconceived solution.

The problems of time and data are critical to the kind of academic research that tests predictions in order to create new bits of knowledge. If my thesis is accepted --that academic research can be made more useful by increasing the proportion of theory-testing studies--then we should focus on some recommendations that may help alleviate the problem.

Research Education. Academic institutions should instill a macro view of research, demonstrating the place of individual student academic research in relation to a dynamic research process, from problem concept to implementation. We should emphasize the desirability of many individual research studies that test predictions so as to create a body of new knowledge. This new knowledge, when taken as a whole, may be synthesized to establish research trends. Students and practitioners in the field may learn that each research study need not, in itself, produce a new Salk vaccine. However, each such study should very likely add new knowledge to the research cycle.

Research Management. Good research management should be directed toward integrating individual studies into a complete research process on a few priority problem areas. The managers of research programs should focus their efforts on a wide variety of techniques designed to negate the time and data problems that render stage 3 testing studies difficult to accomplish. Although the techniques cannot be clearly centered on either of the problems, we will concentrate on time first, then on data.

I think it fair to say that increasing the proportion of concept-testing studies at the expense of concept-building studies will effectively decrease the time required to complete an ideal research cycle on one broad topic. If this belief is sound, then those who manage research programs should stress --or even require--a high proportion of student studies that test predictive concepts.

One successful approach is to take full advantage of available time by integrating the student research program throughout the full interval of a student's academic program. If we create new knowledge through concept-testing research, then with a little thought research can be integrated throughout a curriculum consisting of many individual courses covering a multitude of subjects. An integrated research program should involve certain identifiable elements interwoven throughout the academic calendar. These elements should include faculty screening of topics before student arrival, early topic selection by students, a review of existing knowledge about the topic as term paper requirements in appropriate courses, instruction in the macro view of an ideal research process, and instruction in the tools of research. The tools should include the characteristics of research writing and some attempt to demonstrate how the theories and quantitative techniques developed in other courses are applied in research. A survey course in theoretical research design is necessary. Some kind of a research proposal, rewritten, critiqued, and approved, is a very helpful vehicle. These research elements should be structured and integrated to occur as early in the academic program as is practical.

In my opinion, it is sheer folly to place research in an academic square as you would a traditional course and expect a quality theory-testing study to begin and end within a three- or six-month period of time such as one academic quarter. Concept-testing research is an iterative process in which considerable research must be accomplished merely to define the problem and determine if data are available in a form necessary for testing. In applied concept-testing research there are so many possible delays beyond the control of students that time is wasted and research quality deteriorated if a research program is not designed to integrate the effort from beginning to end into a student's academic program.

One way to overcome problems associated with data is to relate the student research programs to faculty professional development. Several techniques are available to research managers. Partially successful but controversial techniques have included (1) soliciting topics, (2) requiring faculty members to personally screen all research topics from all sources, and (3) requiring each faculty member to submit for student selection some minimum number of topics that he has reviewed and found to be researchable. The faculty topic screening requirement will encourage faculty to initiate contacts with research sponsoring agencies. This contact usually provides some reasonable assurance, before topic selection by students, that data are indeed available for testing. This association of the faculty with sponsoring agencies often leads to ongoing relationships that generate a continuing series of student research studies. These associations also serve to keep the faculty in contact with the real world. Students are provided with ready-made situations where data are available to test and thereby create new knowledge. Individual academic research studies may begin to support the ideal research process and also become the foundation for much of the faculty professional development activities --which in time renders the teaching and the school more responsive to a profession's needs.

Research managers should find situations where data are available and advertise these data sources to the faculty. We have found both faculty and students will tackle almost any problem where data are readily available for stage 3 concept testing.

A potential service available to all management schools for obtaining data was recently initiated with the establishment of the USAF Business Research Management Center. This agency, at Headquarters USAF level, is co-located with the School of Systems and Logistics at Wright-Patterson AFB, Ohio. The center is beginning to influence management research by coordinating between academic facilities that need to do research and agencies in the field that are willing to support research efforts with data. In this manner the center is attempting to influence the thrust of individual studies so that several studies may form a complete research process.

Academic management research is concentrating too much on theory-building studies and not enough on theory-testing studies. An efficient and complete research process requires the right proportion of both kinds of studies. I have suggested techniques to help academic research become more useful and profitable by moving toward a more complete research process. Although practitioners must render decisions based upon the best hypothetical solution of staff studies, perhaps we risk too much when such theory is based on old and incomplete knowledge. Air Force managers have access to the latest analytical techniques, sophisticated computers, and massive amounts of data. However, until we shift the emphasis of analyses toward theory testing to create knowledge, Air Force managers will continue practicing modern gamesmanship rather than mature science. Mark Twain expressed the difference when he said, "It’s not what we don't know that hurts us, it's what we do know that isn't so."

Lieutenant Colonel Ronald R. Calkins (Ph.D., University of Denver) is an Associate Professor of Research Management, an Assistant for Research, and Head, Research and Communicative Studies Department, School of Systems and Logistics; Air Force Institute of Technology. Commissioned from AFROTC in 1956, he served seven years as a navigator with weather reconnaissance squadrons in Alaska, Washington, and California, in WB-50s, and a Southeast Asia tour with tactical electronic warfare squadrons in EC-47s. Colonel Calkins has also taught at Officer Training School.

Disclaimer

Air & Space Power Home Page | Feedback? Email the Editor