Field studies or case studies are an attempt to study practitioners in a realistic environment. The goal is to evaluate actual performance in a precise manner, with minimal interference to standard practice. Not all the variables can be carefully controlled, but unexpected events or insights may reward the careful observer/experimenter.

[..]

Field studies are appropriate for investigating complex issues such as an entire data management system, interaction modes, or a management strategy for team organization. These issues are not amenable to straightforward controlled experimentation but some conclusions can be drawn from a field study. Critics complain about the lack of controls and the possible influence of external factors such as organizational morale, individual motivation, and personality differences. Results are not always generalizable or replicable, but important insights or suggestions for controlled experimentation can be derived from field studies. A final criticism is that field studies tend to measure current practice rather than the improvement obtainable from new strategies.

This research method depends on an individual’s sensitivity to the cognitive skills required while performing one or more of the five tasks: learning, composition, comprehension, debugging, and modification. [..]These judgments are highly influenced by training and often do not correlate welI with performance metrics. Subjects may perceive a task as being easy, but do poorly at the task.

Two problems come to mind with this yardstick in evaluating the human factors aspect of a language: (1) Many queries that can be written with a relationally complete language are extremely difficult to compose or comprehend. Few people claim to have a thorough understanding of first-order predicate calculus. (2) Many common, useful, simple to understand, and potentially easy to express queries are outside the bounds of relational completeness. For example, in a table of distances between adjacent cities, finding the shortest path between two remote cities is not included in relational completeness. Simi- larly, in a table of employees and their managers, finding the names of all the employees that a given individual manages at all levels, is not a relationally complete query.

Controlled experimentation depends on a reductionist approach which minimizes uncontrolled bias. A small number of factors, say one to four, are chosen as critical to the performance of a task. These factors are varied while all other factors are kept constant, if possible. If varying a factor results in a statistically significant difference in performance, then suggestions for practical implementa- tion can be developed. The factors which the experimenter varies are the independent variables and the performance measures are the dependent varia- bles. The advantage of controlled experimentation is that the results are generaliz- able and replicable. Critics argue that controlled experiments too often have a narrow focus and produce trivial results.