Note: In August 2013, Professor S. Keshav of the University of Waterloo wrote a paper titled How to Read a Paper, which is, as the name suggests, about how to read a paper. After its publication, it quickly drew strong attention. The paper introduces a set of steps for reading research papers, with the hope of helping people read the literature more effectively. Even if you don't read academic papers, it offers useful insights for reading all kinds of technical material. At the time, my mailing list was flooded with recommendations for this paper. I thought it was fantastic after reading it, and I planned to translate and share it despite my poor English. Unfortunately, I was being pushed around by someone on various matters at the time, so the task kept getting delayed.

Now half a year has passed, but since I finally have a little time, I'd better finish this thing.

The original author is S. Keshav, from the David R. Cheriton School of Computer Science, University of Waterloo; his email is keshav@uwaterloo.ca. The original paper is available on Google Scholar and easy to find (for example, at this address). The content isn't hard to understand, which is probably why no one felt motivated to translate it. As for my translation, given my poor skills, please go easy on me.

(Postscript: I suspect understanding my translation may be even harder than understanding the original. TAT)

Enough nonsense. Here is the main text:

Abstract

Researchers spend a great deal of time reading research papers. Yet this skill is rarely taught, leading to much wasted effort. This article outlines a practical and efficient three-pass method for reading research papers. I also describe how to use this method to do a literature survey.

1. Introduction

Researchers must read papers for several reasons: to review them for a conference or a class, to keep current in their field, or to survey a new field. A typical researcher will likely spend hundreds of hours each year reading papers.

Learning to read papers efficiently is an important but rarely taught skill. Beginning graduate students, therefore, must learn on their own using trial and error. Students waste a great deal of effort in the process, and suffer frequent frustration.

For many years I have used a simple "three-pass" approach to prevent myself from drowning in the details of a paper before I get a bird's-eye view. It allows me to estimate the amount of time required to review a set of papers. It also lets me adjust the depth of paper evaluation depending on my needs and the amount of time I have. This paper describes the method and its use in doing literature surveys.

2. The Three-Pass Approach

The key idea is that you should read the paper in up to three passes, instead of starting at the beginning and plowing your way through to the end. Each pass accomplishes specific goals and builds upon the previous pass: The first pass gives you a general idea about the paper. The second pass lets you grasp the paper's content, but not its details. The third pass helps you understand the paper in depth.

2.1 The First Pass

The first pass is a quick scan to get a bird's-eye view of the paper. You can also decide whether you need to do any further passes. This pass should take about five to ten minutes and consists of the following steps:

  • Carefully read the title, abstract, and introduction.
  • Read the section and sub-section headings, but ignore everything else.
  • Glance at the mathematical content (if any) to determine the underlying theoretical foundations.
  • Read the conclusions.
  • Glance over the references, mentally ticking off the ones you've already read.

At the end of the first pass, you should be able to answer the five C's:

  • Category: What type of paper is this? A measurement paper? An analysis of an existing system? A description of a research prototype?
  • Context: Which other papers is it related to? Which theoretical bases were used to analyze the problem?
  • Correctness: Do the assumptions appear to be valid?
  • Contributions: What are the paper's main contributions?
  • Clarity: Is the paper well written?

Using this information, you may choose not to read further (and not print the paper, thereby saving trees). This could be because the paper doesn't interest you, or you don't know enough about the area to understand the paper, or that the authors make invalid assumptions. The first pass is adequate for papers that aren't in your research area, but may someday prove relevant.

Incidentally, when you write a paper, you can expect most reviewers (and readers) to give it only one pass. Pay attention to choosing coherent section and sub-section titles and to writing concise and comprehensive abstracts. If a reviewer cannot understand the gist of your paper after one pass, it will likely be rejected; if a reader cannot understand the highlights of your paper after five minutes, it will likely never be read again. For these reasons, a "graphical abstract" that summarizes a paper with a single well-chosen figure is an excellent idea and can be increasingly found in scientific journals.

2.2 The Second Pass

In the second pass, read the paper with greater care, but ignore details such as proofs. Jot down key points or make comments in the margins as you read. Dominik Grusemann from Uni Augsburg suggests that you "note down terms you didn't understand, or questions you may want to ask the author." If you are acting as a referee, these comments will help you when you write your review and back up your review during the program committee meeting.

  • Look carefully at the figures, diagrams and other illustrations in the paper. Pay special attention to graphs. Are the axes properly labeled? Are results shown with error bars, so that conclusions are statistically significant? Common mistakes like these will separate rushed, shoddy work from the truly excellent.
  • Remember to mark relevant unread references for further reading (this is a good way to learn more about the background of the paper).

The second pass should take up to an hour for an experienced reader. After this pass, you should be able to grasp the content of the paper. You should be able to summarize the main thrust of the paper, along with supporting evidence, to someone else.

This level of understanding will suffice if the paper interests you but is not in your research area. Sometimes you won't understand a paper even at the end of the second pass. This may be because the subject matter is new to you, with unfamiliar terminology and acronyms. Or the authors may use a proof or experimental technique that you don't understand, so the bulk of the paper is unintelligible. The paper may be poorly written, with unsubstantiated assertions and numerous forward references. Or it could just be that it's late at night and you're tired. You can now choose to: (a) set the paper aside, hoping you don't need to understand it to be successful in your work, (b) return to the paper later, perhaps after reading background material, or (c) persevere and go on to the third pass.

2.3 The Third Pass

To fully understand a paper, particularly if you are a referee, requires a third pass. The key to the third pass is to attempt to virtually re-implement the paper: that is, making the same assumptions as the authors, re-create the work. By comparing this re-creation with the actual paper, you can easily identify not only a paper's innovations, but also its hidden failings and assumptions.

This step requires great attention to detail. You need to identify and challenge every assumption in each statement. In addition, you should think about how you yourself would present a particular idea.

This comparison of the real and the virtual can bring strong insight into the techniques of proof and presentation in the paper, and you may bring these into your entire toolkit. Through this step, you should also record these ideas for future work.

Beginners may spend quite a few hours on this step; even experienced readers will spend more than an hour, or even two hours, to complete it. By the end of this step, you should be able to reconstruct the entire structure of the paper from memory, while also identifying its strengths and weaknesses. In particular, you should be able to pinpoint implicit assumptions, missing citations to relevant work, and potential issues with experimental or analytical techniques. (In particular, you should be able to pinpoint implicit assumptions, missing citations to relevant work, and potential issues with experimental or analytical techniques.)

3. Do a Literature Review

Doing a literature review is a great test of your paper-reading skills. It requires you to read dozens of papers, possibly in fields you are not very familiar with. Which papers do you need to read? In the three-pass method, you can get some help.

First, use an academic search engine, such as Google Scholar or CiteSeer, to find three to five recent, highly cited papers in this field using convenient keywords. Do the first pass of the three-pass method on each paper, then read their "related work" sections. If you are lucky enough, you may find a recent survey paper. If you can find such a survey, you are done — read the survey and celebrate your good fortune.

Otherwise, second, look for citations shared among these papers' reference lists and author names that appear repeatedly. These are the key papers and researchers in that field. Download these key papers and put them together. Then go to these key researchers' websites and see what papers they have published recently. This will help you identify the top conferences in those fields, because the best researchers usually publish in top conferences.

The third step is to go to the websites of those top conferences and look through their most recent proceedings. A quick scan usually identifies recent high-quality related work. These papers, together with the papers found earlier, form the first version of your survey. Do two passes on these papers. If they all cite an earlier paper you have not yet found, seek it out and read it, and so on as needed.

4. Related Work

If you need to read some papers to do a review, you can also read Timothy Roscoe's paper "Writing Reviews for Systems Conferences"[3]. If you plan to write a technical paper, you should refer to Henning Schulzrinne's comprehensive website[4] and George Whitesides' passionate overview of the related process[5]. Finally, Simon Peyton Jones has a website covering the full range of research skills[2]. Iain H. McLean of Psychology, Inc. used the three-pass method to compile papers related to clinical psychology (experimental psychology) into a downloadable "overview matrix"[1]; with minor modifications, it should be usable for papers in other fields as well.

5. Acknowledgments

The first version of this document was drafted by my students: Hossein Falaki, Earl Oliver, and Sumair Ur Rahman, to whom thanks are due here. At the same time, Christophe Diot's insightful comments and Nicole Keshav's eagle-eyed editing were also very helpful to me.

I also hope to keep this document dynamic — I will update it as I receive comments. If you have any comments or suggestions, please take a moment to email me. Thanks to everyone for the encouraging feedback over the years.

6. References