Issue
Article
Vol.30 No.2, April 1998
Article
Issue
Issue |
Article |
Vol.30 No.2, April 1998 |
Article |
Issue |
The Empirical Studies of Programmers: Seventh Workshop (ESP 7) was held in Alexandria, VA. October 24-26, 1997. Dr. Susan Wiedenbeck and Dr. Jean Scholtz were the co-chairs of the event. The first ESP was held in 1986, so this conference marks the second decade of ESP workshops. There were just under 60 attendees.
On Friday, prior to the actual start of the workshop, Charles Kann, a graduate student from George Washington University, and Dr. Laura Leventhal from Bowling Green University organized a student conference. Fourteen graduate students doing research in empirical studies of programming met to discuss their work. The students and other attendees jointly participated in the opening reception and poster session for ESP. About 20 posters were presented.
The ESP program ran from Saturday morning until noon on Sunday. There were two invited speakers, two panel presentations and eleven papers. A short summary of these events follows.
Watts Humphrey, Software Engineering Institute, "What do we Know about Programming?"
Watts described his PSP -- Personal Software Process. This process is designed to teach programmers how to program efficiently. He, as well as other instructors, have taught this course and collected data to determine how well the process works. The data collected during these classes shows that programmers using the PSP method find more code defects prior to compiling their code than programmers not following this process.
Dr. Iris Vessey, Indiana University School of Business, "Problems versus Solutions: The Role of the Application Domain in Software"
Dr. Vessey pointed out that although most computer programs arise due to a problem in a particular domain, we have historically treated program development as application independent. She called attention to strong and weak problem solving methods. Strong problem solving methods focus on specific types of problems, while weak methods are generic and can be applied to many types of problems, but not optimally. Dr. Vessey classified the software communities as product, process, problem and others (expert systems and languages) and discussed the trend toward application specific solutions in each. Dr. Vessey concluded saying that the field needs to distinguish between the problem and the solution and identify ways to characterize each in order to determine the "best" solution for a problem type.
J. Kim, J. Hahn and J. Lerch presented "How is the Designer different from the User? Focusing on a software development methodology." The authors reported on a study in which designers of an object-oriented methodology and students (users) in the object-oriented methodology class were asked to develop a solution to a specified problem. The time each group spent in the four different problem spaces (rule, representation, instance, and paradigm) were measured. They found that the designers used the paradigm space as a major supporting space and that they actively searched the representation space and quickly arrived at solutions which were not extremely detailed. The users, on the other hand, used the instance space as a major supporting space, searched the rule and representation space and arrived at rather detailed solutions after considerable time. The authors note that further research needs to be conducted, but they feel that the representation in the object-oriented methodology may not have been rich enough to convey the design model to the users.
M. Petre and A. F. Blackwell presented "A Glimpse of Expert Programmers' Mental Imagery." They conducted studies of expert programmers doing program design and discussing these designs with other programmers. They found many instances of mental imagery being used by software designers, suggesting a similarity to other design activities. This implies that further research should investigate types of representations that could be used to support these images.
"An Experimental Evaluation of Quality Guidelines on the Maintainability of Object-oriented Design Documents" by L. Briand, C. Bunse, and J. Daly was the third paper in this session. Two object-oriented systems were developed; one was developed in accordance with the Coad and Yourdon design principles. Subjects looked at both systems and were asked questions about their overall understanding of the designs. They were then asked to do an impact analysis of changes that would have to be made for 1) a change in customer requirements and 2) enhanced functionality. The authors found that the system designed according to the Coad and Yourdon principles was easy to understand and to do impact analysis on. The authors have made available the experimental materials to anyone who would like to try replicating and improving upon the experiment.
This session consisted of three papers: P. Mulholland, "Using a fine-grained comparative evaluation technique to understand and design software visualization tools", K. Whitley and A. F. Blackweel, "Visual programming: the outlook from academia and industry", and C. Cook, M. Burnett, and D. Boom, "A bug's eye view of immediate visual feedback in direct manipulation programming systems."
P. Mulholland used protocol analysis to identify user behavior using visualization tools and to give us an understanding of the effectiveness of different visualizations. He classified verbalizations of: information access from the display, the use of comprehension strategies, and misunderstandings of the visualization and execution. He used the analysis from the protocols of users using four visualization techniques to develop a new visualization tool. Testing the new tool using the same methodology showed that it was considerably better than the previous tool.
Whitley and Blackwell surveyed three groups of people as to their opinions about visual programming. They surveyed the existing literature to gather opinions that researchers had about visual programming. They conducted surveys of professional programmers at a trade show and they surveyed a group of Labview programmers. Researchers in visual programming were optimistic about the benefits of visual programming. Professional programmers were skeptical about benefits of visual programming. Labview users, however, were confident that Labview was beneficial to their programming efforts.
C. Cook presented the third paper which looked at the effect of adding immediate visual feedback to a direct manipulation programming language. Although immediate visual feedback did not help with all debugging, it did help in certain instances. The authors found that the type of bug and the position of the buggy code on the screen were influencing factors in how long subjects took to locate and correct the bug.
Two papers were presented in this session. One by V. Ramalingam and S. Wiedenbeck and the second by A. von Mayrhauser and A. M Vans. V. Rmalingam presented "An empirical study of novice program comprehension in the imperative and object-oriented styles." In this study novices were given both object-oriented programs and imperative programs written in C++ and asked questions about different information categories. Overall, novices performed better at comprehending the imperative programs. However, they were able to obtain more domain-level knowledge from the object-oriented programs.
A. von Mayrhauser reported on "Program Understanding Behavior during Debugging of Large Scale Software." They conducted protocol analyses of professional maintenance programmers debugging large computer programs. They classified three levels of abstraction that programmers work at: the program level which is a control flow abstraction of the code, a situation model which uses the program model and creates a data-flow or functional abstraction and the top-down or domain model which is an abstraction of the application domain. They looked at the levels of abstraction that the programmers worked at and concluded that programmers with little domain experience or little knowledge of the software worked at lower levels of abstraction until they accumulated enough knowledge to make higher level connections. However, programmers who had some domain experience could more easily make connections from the lower levels. Programmers who knew both the code and the domain could easily make connections between all three levels of abstraction.
M. G. Williams and J. N. Blehler presented "A Study of program entry time predictions for application-specific visual and textual languages." In this study, the authors looked at the differences in time that it took programmers to enter the program code written in a visual language versus the time it took programmers to enter the program code written in a textual language. A keystroke level model for the entry times was created and validated by an empirical study. These models can be used to approximate the code entry and editing tasks in studies comparing programming tasks in different languages.
H. J. Gottried and M. M. Burnett presented "Programming complex objects in spreadsheets: an empirical study comparing textual formula entry with direct manipulation and gestures." An empirical study was conducted to compare techniques such as direct manipulation and gestures to typing for entering textual formula into spreadsheets. They found that subjects were faster and more correct when using the graphical definition techniques. Moreover, subjects preferred using the graphical techniques to define formulae.
C. Traynor and M. Williams presented "A study of end-user programming for geographical information systems." The authors conducted an empirical study of a programming by demonstration system. End users of a GIS system were given program comprehension and program editing tasks. Subjects were able to comprehend and edit simple programs without error, but had difficulties on more complex programs, implying that some of the more complex constructs in the visual language need to be redesigned. However, subjects preference ratings implied that programming by demonstration is a promising technique.
Two panels were also presented at the workshop. Robert Rist organized a panel on past, present and future research issues in ESP. Panel members were Françoise Détienne, Marian Petre and MaryBeth Rosson. Rob presented an overview of research on program design. Françoise discussed the literature in program comprehension. Marian talked about the work that has been done in notations, including visual languages. MaryBeth gave an overview of the research in computer science education in programming languages and the work in programming environments.
Jim Herbslab was the organizer of a panel "Empirical Studies of Software Production: Methods, Substance, and Impact." Panelists were Beki Grinter, Marian Petre, and Adam Porter. Jean Scholtz was the discussant. The panelists were asked to address the question: "how can empirical researchers of software production have the greatest impact on understanding the essential problems and on improving practice?"
Marian made the point that this was the wrong question. A major impact can be made collectively -- through many small impacts. In order for theory to ultimately have impact it has to be accessible to practitioners. Adam discussed how to have leverage on the development process. In his studies of software inspections, he tries to minimize the intrusion but he emphasizes the cost-benefit of being able to improve the process through studies. Beki advocated looking at the collaborative aspects of programming and using theory to explain practice. Qualitative studies open up new research questions but they take time to carry out and finding access to groups to study is difficult. Jean called for the ESP community to study software developers, not just programmers. She noted that new methods will need to be developed to study the activities of software developers, such as programming, communication, collaboration.
The next ESP workshop will be held in June of 1999. Co-Chairs of the event are Laura Leventhal, Irv Katz, and Marian Petre. Proceedings of ESP 7 are available from ACM Press.
Dr. Scholtz is currently a researcher in the Information Technology Group at the National Institute of Standards and Technology (NIST). Her research there centers on evaluation methodologies and developing the infrastructure to facilitate such evaluation. In particular, she is working on usability evaluation tools for Web sites and Web applications and evaluation tools and methodologies for CSCW and collaboratories.
Jean Scholtz, Ph.D.
NIST Building 225, Room 216,
Gaithersburg, MD 20899, USA
Tel: +1-301-975- 2520
Fax: +1-301-975-5287
http://www.nist.gov/itl/div894/vvrg/jean.html
Issue |
Article |
Vol.30 No.2, April 1998 |
Article |
Issue |