A thesis defense has two parts: a thesis and a defense. The second mistake many students make is not knowing what their thesis is. The third mistake is not knowing how to defend it. (The first mistake is described later.)
Your thesis is not your dissertation. Neither is it a one-liner about what you are doing. Your thesis is "a position or proposition that a person (as a candidate for scholastic honors) advances and offers to maintain by argument." [Webster's 7th New Collegiate Dictionary] "I looked at how people play chess" is not a thesis; "people adapt memories of old games to play new games" is. A thesis has to claim something.
There are many kinds of claims. Most of the work around here is either design (you built an AI program or learning tool) or modelling (you have a theory of how something works). Such work usually supports one of the following kinds of claims:
where "X" is your model of memory, learning algorithm, authoring tool, learning environment, etc., and "Y" is a task or goal, such as understanding text, learning algebra, writing programs to teach algebra, etc.
Besides being a proposition, a thesis has to have another property: it must say something new. "Understanding natural language requires context" is not a thesis (except maybe in a linguistics department). "Algorithm X is a feasible mechanism for understanding anaphoric references in newspaper text" is. So is "context is not required for visual understanding."
A defense presents evidence for a thesis. What kind of evidence is appropriate depends on what kind of thesis is being defended. There are very different defenses for each of the kinds of claims given above.
One defense for this kind of claim is an analysis of the complexity, or completeness, or whatever, of the theoretical algorithm. In AI, the more common defense is based on empirical results from running a program. A good defense here means more than one example, and answers to questions such as the following. What are the capabilities and limits of your program? How often do the things that your program does come up in the real world? What's involved in extending it? If it's easy to extend, why haven't you? If your program is a piece of a larger system, how realistic are your assumptions about input and output?
Analogous questions arise in the design of learning environments. You should be able to argue that your environment can support learning in more than one specific arena, and what would be involved in implementing it elsewhere. You should be clear and specific about when it would not be a good approach.
The same kind of defense applies here as in the previous case, but now serious comparisons with previous systems are required. In AI, can your program do the same examples the previous programs did, or can you make them do yours? Can you prove they couldn't do your examples? If you claim to be more efficient, what are you measuring?
In the learning sciences, what do you mean by "better" and how are you measuring it? Are you sure the other approaches wouldn't work just as well if they had you spending all that time on them?
This is usually defended by a logical argument. It is usually very tough to do, even if the argument doesn't have to be formalized.
Many students in AI make the mistake of picking a cognitive modelling thesis to defend, thinking that something that looks cognitively plausible is therefore OK. Defending a cognitive model requires serious experimental evidence. Selected excerpts from protocols and surveys of your officemates are not psychological evidence, no matter how much they might have inspired your work.
Collecting enough evidence to really defend a thesis is hard. If you think you have a lot of theses, you probably just have a bunch of undefended claims. One good thesis, or two so-so theses, with adequate description and defense, is more than enough to fill up a dissertation.
Highly unlikely. If you're bright, educated, and have worked hard on a topic for more than a year, you must have learned something no one else knew before.
The first mistake that AI students make is to think that a thesis has to be grander than the theory of relativity. A thesis should be new and interesting, but it doesn't have to change the foundations of all we believe and hold dear.
There's a mistaken view of thesis research that it starts with a thesis and then an investigation to prove or disprove the thesis. That only happens in fields that have matured to "filling in the blanks," and even then it only happens with advisors who like to stick to tried and true questions.
In young fields such as AI and the learning sciences, you'll never start with a claim. Very few of you will even get to start with a question! You start by exploring one or more problems in some task domain. You'll start with some initial ideas, naive or clever, and push them hard for a year or so. Then, you need to stop and think about what you've done and what you've learned. Among your accomplishments and experience, there will be several good candidate theses. Pick one. Test it out on your advisor and other faculty members. Test it out on other students. Watch out for the following flaws:
Once you refined your claim, now you can determine what kind of defense is appropriate for it and what more you need to do. This is where the psychologically hard part comes, because to create a defense for your thesis, you're going to have to attack it harder than anyone else. What happens if the thesis fails? Negate it and defend that! In a year or so of focussed research, you should be ready for a real thesis defense.
See how easy it is, once you know how?