Research Skills

Videos of the lectures

Most of the October and November lectures in 2012 were recorded, as were two 2009 lectures on presentation style.

The 2012 Lectures

2012 Lecture 1: How to read and how to find papers to read (66 minutes). N.B. the lecture proper starts at 8:02. The first 8 minutes are an introduction to the course itelf.

2012 Lecture 2: How to referee a paper and the reviewing process (25 minutes).

2012 Lecture 2b (Ethics) - no recording exists

2012 Lecture 3a: How NOT to give a presentation (18 minutes) or from Vimeo: How NOT to give a presentation.

2012 Lecture 3b: Presenting a paper (12 minutes). A short presentation of basic concepts and mis-conceptions in presenting a paper. This is an introduction to the longer presentations later in the course.

2012 Lecture 4: Who are you writing for? Writing style. (40 minutes).

2012 Lecture 5a: Continuation of Lecture 4 (20 minutes). We finish the discussion of Orwell, then look at how to avoid over-qualification, over-emphasis, pretension, pomposity, and obfuscation.

2012 Lecture 5b: Two ways to write the first draft of a document. Advice on editing and advice on getting other people to comment on your document. (12 minutes).

2012 Lecture 6: Professor Simon Peyton Jones, Microsoft Research, gives a guest lecture on writing (35 minutes). Seven simple suggestions: don't wait - write, identify your key idea, tell a story, nail your contributions, put related work at the end, put your readers first, listen to your readers.

2012 Lecture 7 (Writing lecture 4) - no recording exists

2012 Lectures 8-10 (Experimental design and analysis) - no recording made

2012 Lecture 11a: Thoughts on research (8 minutes).

2012 Lecture 11b: Conferences (20 minutes). Which conference to submit to. How conferences work. What to do at a conference.

2012 Lecture 11c: Authorship (and acknowledgements) (6 minutes).

2012 Lecture 12a: Good and bad graphs (24 minutes)

2012 Lecture 12b: Two live examples of creating basic graphs (15 minutes)

2012 Lecture 13a: Figures, Tables, Maths, Algorithms (25 minutes)

2012 Lecture 13b: Two live examples of using graphs to explore data (10 minutes)

2012 Lecture 14a: Posters for conferences (18 minutes)

2012 Lecture 14b: The Problem with Powerpoint - see 2009 lecture Good and bad presentations (starts 14 minutes in, ends 23 minutes in)

2012 Lecture 15a: The Gettysburg Address (15 minutes)

2012 Lecture 15b: Laying out presentation slides (13 minutes)

2012 Lecture 15c: An example of how to present a mathematics proof in a talk (9 minutes)

2012 Lecture 16a: How to read (3 minutes)

2012 Lecture 16b: How to prepare a presentation (33 minutes)

The 2009 Lectures

2009 Lecture 8: Good and bad presentations (50 minutes, low resolution) or from Vimeo: Good and bad presentations (higher resolution) 

2009 Lecture 9: How to prepare a presentation (42 minutes) or from Vimeo: How to prepare a presentation

Graphs and other graphical elements

This is the most challenging part of the course to teach effectively. Some students have been taught, since primary school, how to present material well graphically. Some students have not. For later years, we decided to reduce dramatically the amount of material in this part of the course, with pointers to appropriate resources for those who have not learnt this material.

Videos of the lectures

You can view the 2012 versions of the lectures:

Good and bad graphs (24 minutes)

Figures, Tables, Maths, Algorithms (25 minutes)

Two live examples of creating basic graphs (15 minutes)

Two live examples of using graphs to explore data (10 minutes)

Resources

Notes

Good & Bad Graphs: Concepts

Based on Ross Ihaka's lecture (see above).

  • The basics Choose the correct type of graph for the data. Label the graph. Label the axes.
  • Data content Small amounts of data do not require graphs. The human brain can easily grasp one, two or three values.
  • Data relevance You cannot produce a good graph from bad data: graphs are only as good as the data they display.
  • Complexity Graphs should be no more complex than the data they display. Avoid "chart junk": irrelevant decoration, unnecessary colour, 3D effects. Use the ink to display the data, not junk.
  • Distortion Graphs should not give a distorted picture of the values they portray.
  • Story Decide what "story" you want the graph to tell. The same data can tell many stories, what is important for communicating your ideas?

Six Principles for Good Graphics

Based on Steve Simon's notes (see above).

  • The Minimum Ink Principle Avoid gimmicks like pseudo 3-D effects or fancy crosshatching. Use the minimum amount of ink to get your point across.
  • The Small Table Principle A small table is better than a large graph. If your graph contains 20 data points or less, consider a table of numbers instead.
  • The Error of Error Bars Principle Error bars are confusing and ambiguous. Plot all the data if possible, or use a box plot.
  • The Size and Shape Principle. Carefully consider the size and shape of your graph. Rectangular graphs are sometimes better than square graphs. Bigger is not always better.
  • The Reproduction and Reduction Principle Colours, shades of gray, and small symbols may get lost when a journal prints your graph, when a student photocopies your graph, or when a colleague prints your graph on a poor printer. Make sure your graphs can withstand reproduction and reduction.
  • The Fault of Default Principle Graphing is an iterative process. Do nt rely on the default options provided by your graphics package. Try everything. Re-draw your graphs as often as you rewrite your text.

 

Writing

Writing is a skill that improves with practice. But there are many things we can learn from experts to improve our writing. These lectures give advice culled from many practitioners.

Preparatory reading: Chapters 2, 3, and 8 of Writing for Computer Science.

Videos of the 2012 lectures

You can view the 2012 versions of the lectures on the University of Cambridge streaming media service:

Who are you writing for? Writing style. (40 minutes).

Finishing the discussion in the previous video, then looking at how to avoid over-qualification, over-emphasis, pretension, pomposity, and obfuscation (20 minutes).

Two ways to write the first draft of a document. Advice on editing and advice on getting other people to comment on your document. (12 minutes).

Professor Simon Peyton Jones, Microsoft Research, gives a guest lecture on writing (35 minutes). Seven simple suggestions: don't wait - write, identify your key idea, tell a story, nail your contributions, put related work at the end, put your readers first, listen to your readers.

Authorship (and acknowledgements) (6 minutes). Who should be an author on your paper?

Writing the First Draft & Editing

  • 1984: look at the opening paragraph of the first draft of George Orwell's 1984. Consider how many changes Orwell made to that draft. Here is an analysis of the changes Orwell made to the first page of the book, between that first draft and the final, published version.
  • John Wyndham didn't get Day of the Triffids' opening right at first. Compare the original manuscript (right) with the published version. What is different? What has been removed, added, and moved? Why is the published version better?

Image of manuscript page taken at the Out of this World exhibition at the British Library.

  • For the 2009 version of the course, I wrote an example (88kB PDF) that demonstrates Exercise 5. This shows repeated editing, cutting the number of words down by 30 each time. I have used Microsoft Word's reviewing facility to show you exactly which words have been removed and which added on each iteration; you do not need to do this for your submission. 

Resources

These resources give five views on the writing process. They are the preparatory reading for this lecture.

Presenting

Researchers frequently present their work. Some do it well. Some do it badly. Some do it so badly that the audience cannot remember what the talk was about.

These lectures are designed to help you present better and to help audiences to remember the key point of your presentation.

Videos of the lectures

The 2012 lectures

How NOT to give a presentation (18 minutes, University of Cambridge Streaming Media Service) also on Vimeo

Presenting a paper (12 minutes). A short presentation of basic concepts and mis-conceptions in presenting a paper. This is an introduction to the longer presentations later in the course.

How to prepare a presentation (33 minutes)

Laying out presentation slides (13 minutes)

The Gettysburg Address (15 minutes)

An example of how to present a mathematics proof in a talk (9 minutes)

The 2009 lectures

Compare the 2012 lectures with the two lectures on presentations given in 2009.

Good and bad presentations

How to prepare a presentation

Resources

Presentation Zen — Garr Reynolds approach to great presentations
Made to Stick — Dan & Chip Heath's explanation of why some ideas survive and others die

Brief summaries from the lectures

The basic talk

Who? — title, author, venue, date
What? — the key idea
Why? — why it is important
How? — technical details (if there is time - cannot do this in a five minute talk)
Where? — where it leads next
Final slide: — the key idea (leave this up during Q&A)

The right questions to ask

from Presentation Zen, Garr Reynolds

How much time do I have?
What is the venue like?
What time of day?
Who are the audience?
What is their background?
What do they expect of me?
What do I want them to do?
What is the fundamental purpose of my talk?
What is the story?
What is my absolutely central point?

SUCCESS criteria

from Made to Stick, Dan & Chip Heath, interpreted for Computer Science presentations by Neil Dodgson

Simplicity
not simplistic; not dumbed down
What is your key point?
What is your core message?
Why should your audience care?

Unexpectedness
Stimulate the audience's curiosity
Pose questions
Expose a gap in their knowledge - then fill the gap
Take the audience on a journey

Concreteness
Start with an example
Use examples throughout
Speak of concrete things, not of vague generalities
The abstract is hard to grasp, examples are easy to grasp

Credibility
Provide evidence that your idea works
For example: show results, describe the algorithm
Provide enough detail to make them go read your paper, but not so much that you bore them or lose them

Emotions
People are emotional beings: make them feel something
Catch their interest: can they use your work? do they believe you?

Stories
People love stories
Where appropriate, make the whole talk into a story
Use anecdotes in your talk: "When we started this research we assumed X, but we were surprised when we found that..."

Neil Dodgson's hints & tips

A research presentation is not a paper
You are presenting the idea that is in your paper
You are presenting an advertisement for your paper
You are convicing people to go read your paper

Beware the curse of knowledge
...where you cannot imagine what it is like not to have your level of background knowledge on the topic. [Chip & Dan Heath]
Imagine the typical audience member (e.g., a member of your research group who does not know your work) and plan the talk to be accessible to them.
Try applying the Heath brother's SUCCESS criteria to your talk.

Start well
Start with the key idea
Start by catching the audience's interest - how did they do that? is that possible?
You have only two minutes before many of the audience will drift off - do not waste those two minutes
Ensure that you know exactly what you are going to say to start and have rehearsed it

Stop within the time limit
Using more than your time is rude and shows poor planning
If you run over time, everyone will remember that fact and not your talk
Do not try to cover everything that is in your paper - leave people wanting to know more - they'll go read your paper

Rehearse
Practice, practice, practice
A presentation is a performance - you would not go watch a stage show where the actors had not rehearsed - your audience have given up their own time to listen to you - you owe it to your audience to rehearse beforehand
Only by rehearsal can you ensure (a) that you start well and (b) that you will end on time

Check the technology beforehand
All lecture theatres are different.
Arrive at least ten minutes early to ensure that the technology works.
Ensure that you have a back-up plan if your laptop doesn't connect or if the software is unavailable or incompatible. For example:
If you take your own laptop, have a backup presentation on a memory stick.
If you use Keynote or Powerpoint, have a backup presentation in PDF.
Your luggage may go missing: put a copy of your presentation in The Cloud.
If you need an internet connection for your presentation, have a backup version that doesn't depend on an internet conection.
If you use video in your presentation, have a backup version that doesn't depend on video.
If you use sound in your presentation, have a backup version that doesn't depend on sound.

Lessons from The Gettysburg Address

I use the Gettysburg address as an example presentation. The course is about how to give good research presentations, so it may seem curious that I use a political speech from the 19th century as an example. However, I find that this speech has much to teach us.

Peter Norvig
My original reason for choosing the Gettysburg address was that Peter Norvig had produced a Powerpoint presentation of it. His set of slides demonstrates how a presentation tool can turn a good speech into a poor presentation; this was the key teaching point in the first year I used it. The Gettysburg address is sufficiently short (two minutes) that it is easy to give twice in a lecture: once without visual aids and once with the Powerpoint slides. A video of the first time I lectured this (2009) is available online.

Reflections on the Gettysburg address
My subsequent investigation into the history of Abraham Lincoln's address provided a range of other lessons for students preparing for a life of giving technical presentations. The Gettysburg address was commissioned as a short speech dedicating a military cemetery in Gettysburg, Pennsylvania. It was almost the final act in a three hour-long ceremony of dedication, which included music, prayers, and a two-hour long speech by Edward Everett, one of the great orators of the day. President Lincoln was invited to give the dedication a mere two or three weeks beforehand and his brief was simply to dedicate the cemetery "by a few appropriate remarks". He did rather more than this. What follows are the lessons that I use in my Research Skills class.

You don't need Powerpoint
or Keynote or LaTeX or any other slide software
It is quite possible to give a riveting talk without any visual aids. Norvig's slides clearly detract from the message. If you do choose to use visual aids then ensure that they add to your talk rather than detract from it.

You need to plan
Lincoln did not speak off the cuff. He planned the speech carefully. There are five or six drafts of the speech, most prepared in Washington before Lincoln travelled to Pennsylvania. He worked hard on this speech, ensuring that it said exactly what he wanted to say. You also need to plan when you are invited to speak.

You need to rehearse
You cannot deliver a talk well without rehearsal. In order to get the phrasing and timing right for delivering the Gettysburg address, I have to rehearse it three or four times beforehand each time I give this lecture. Likewise, when I am preparing research talks, I will generally run through my talk a couple of times the evening beforehand, to ensure that I am clear in my own mind what I am going to say.

You should grab opportunities to speak
Lincoln was invited only to give a two-minute dedication speech. He could, conceivably, have refused. But he grabbed the opportunity to address a crowd of 15,000 people, which included six state governors and many others who had influence in American society. We researchers should, similarly, be ready to grab any opportunity to present our work to our colleagues.

You should work within the constraints given you
Lincoln was asked to give "a few appropriate remarks". He knew that this meant he had only two or three minutes. He did not rail against this and write to the organisers asking that he be allowed to speak for longer. He worked within the constraints he was given to do the best he could. Likewise, if you are asked to prepare a fifteen minute presentation, do not prepare a thirty minute presentation and hope that the organisers will accommodate you. They won't. Work within the constraints you are given to do the best you

You should say what you need to say
Lincoln certainly did not restrict himself to "a few appropriate remarks". He made a political speech that inspired his audience. The construction of his speech is clever. The first five sentences are exactly the sorts of things you would expect a president to say when dedicating a military cemetery. His sixth sentence must have come as a shock to his audience. Lincoln says "But, in a larger sense, we can not dedicate, we can not consecrate, we can not hallow this ground." One can imagine members of the audience thinking "What does he mean by that? What is he talking about? Why can we not dedicate this ground?" Lincoln then goes on, in only four sentences, to inspire his audience to re-dedicate themselves to the cause of freedom. He does what he is asked to do, to dedicate the cemetery, and then goes on to say so much more: to say what needs to be said.

You can interpret your brief imaginatively
Lincoln did not restrict himself to dedicating the cemetery, he interpreted his brief imaginatively and gave a short speech that is remembered 150 years later. Likewise, when asked to give a research presentation, be imaginative in how you interpret your instructions. For example, when asked to give a talk about a research paper you may think that your job is to present your research paper. It is not. It is to present the key idea in your research paper. You may find that the best way to do this is to talk about some things that do not appear in your paper and to avoid talking about some other things that are in your paper. Be imaginative in getting your message across.

You must get your message across
Lincoln concentrated on getting his key message across to his audience. It is challenging to prepare a two minute presentation, but at least Lincoln knew that, in two minutes, he had time to convey only one key idea. In a research talk, it is all too easy to try to convey many different ideas and thereby confuse your audience. When preparing a talk decide what your key message is and ensure that that message is communicated clearly to the audience.

Reading

We have all been able to read since childhood. There are still things to learn as we start to do research. When thousands of academic papers are published each day, how do you find the right things to read? How do you effectively skim read? How are papers refereed for publication? How do you referee a paper?

Videos of the 2012 lectures

You can view the 2012 versions of the lectures on the University of Cambridge streaming media service:
How to read and how to find papers to read (66 minutes) N.B. the lecture proper starts at 8:02. The first 8 minutes are an introduction to the course itelf.
How to referee a paper and the reviewing process (25 minutes)
Thoughts on research (8 minutes)
How to read (3 minutes)

Resources

How to do a technical reading program (1 page) by Prof. Fred Brooks

How to review a technical paper (5 pages, 90kB PDF) by Alan Meier, with comments by Neil Dodgson

How to Run a Paper Mill (103 pages, 865kB PDF) by John Woodwark — a lighthearted view of the research process

Google Scholar — not every paper is on the web and not every paper that is on the web can be found by Google Scholar, but it is a good place to start

The nature of papers

The starting research student assumes that all research papers are well-written and describe good research that is both correct and important. This is not true.

Research papers can report poor research. The review process is supposed to prevent this, but it is not a perfect process and there is a fair amount of published work out there that is not good. The publication venue is a guide to likely quality. A top-rank journal or conference is unlikely to publish poor research, because the review process is fierce. A lower-rank journal or conference is more likely to let poor research through.

Published research is supposed to be correct. However, there are cases where the authors and reviewers have honestly failed to notice that their data, assumptions, proofs, or conclusions are wrong.

A big mistake made by many new researchers is to think that every single paper is important. Of course, it was important to the authors, but it may well be reporting on a dead end, a trivial result, a system that does not work well, or an idea that has been superceded. So you will find that many papers are unimportant, at least with respect to your particular research area.

Even if a paper is good research, correct, and important, it may not be well written. Sadly, there are some papers that are incomprehensible. It is wrong to assume that the authors are trying to hide something in their poor writing, though this may be the case. Often it is just poor writing skills. Part of this course is to help you to learn how to ensure that your papers are well written.

A way to read

CEM Joad wrote Teach Yourself Philosophy in 1944. His advice can be applied to reading scientific papers. He says:

Make judicious selection. In philosophy Joad says "Never try to read a whole book at once." In our case, do not try to read all the papers on a subject at once. Joad says "pick out certain chapters, two or three perhaps, which deal with matters that seem to you to be of particular importance or which relate to topics on which you have already read something." We need to pick out those papers that are of most importance to the problem in hand. Of philosophy books, Joad says "Use the introduction and preface to help you identify these." In our case, we need to use something else to determine which papers we should attend to. See Fred Brooks' note (above) for more on that.

Indulge in intelligent skipping. You do not have to read everything. You do not need to read every word of every paper. Do not be embarrassed to skip. You cannot possibly read every paper in the world; you need to ensure that the ones you spend your time on are the important ones. For some papers you need only look at the introduction and conclusion, with a glance elsewhere. For some, you will need to read the whole thing, but maybe not today. Joad says "omit the unintelligible; skip the repetitions; jump over the boring bits". You are allowed to do this!

Ensure you understand. Joad says stop reading when you stop understanding, but he also says to interpret "understand" generously. If you do not understand, there is sometimes point in pressing on a little further to see if it becomes clear, but if you really do not get it, there is no point in reading on for the sake of reading on. You need to stop and say "why do I not understand?"; "is this important enough that I need to learn something else so that I can understand?"

Make notes. The process of writing notes helps enormously with remembering. The process of having to summarise someone else's work helps enormously with understanding. Exercise 1 asks you to precis the content of a four page paper in just one page. This is a great skill to have. Also you need to ensure that you have made good enough notes that you will be able to find any paper again in two years time when it turns out to be important. Too many times have I heard the cry "I know that I have read a paper on exactly this problem, but I cannot remember where!"

More Articles ...

  1. Aims and Syllabus