Unroll: three ways of reasoning while debugging

A thread about debugging skills; there are 3 main different ways of reasoning to apply when debugging, each of them useful in complementary ways. It’s necessary to tell them apart in order to debug effectively. They are deduction, induction and abduction.

In this thread, the goal of debugging is to learn the cause of failure by studying the behavior of the system. The method is empirical, testing hypotheses which are inspired by evidence: source code, error messages and runtime traces.

“Deduction” is the safest, most boring and surprisingly least effective method of reasoning to apply. You conclude, based on axiomatic knowledge, the result of a premise. E.g. “the condition was true, so the then-branch was executed”, because you understand the if statement. The threats to the validity of your debugging deductions are the validity of your axiomatic knowledge (know your language!) and the assumptions you based the premise on (is your printf/conditional breakpoint/logfile right?)

If you make a deduction mistake unknowingly, debugging can be very disorienting and ineffective experience. if you think left hand sides are always executed before right with assignments in C, you’re in for a surprise. Or static blocks are done before constructors in Java, etc.

One real problem with using deduction exclusively when debugging is that it does not scale if done by your brain. Your CPU runs millions of instructions filling gigabytes of memory while you are pondering a single deductive step.

But, deduction is done very well by computers and when you run a program under test that is exactly what it is doing for you, deducing computational effects from causes (inputs) using the program as theory. So do use the computer to test debugging hypotheses. Now thát scales.

The main shortcoming of deduction wrt debugging is that it goes in the wrong direction. It reasons from cause to effect, while the goal of debugging is to reason from effect to cause. We need something else to complement deduction: induction and abduction.

“Induction” is when your brain concludes something more general from specific observations. This can be very effective, especially to cut off huge branches in your search space for a cause. It’s theory forming based on evidence, but it’s only true until you find a counter example

For example, you might/would induce from a single stack trace in an issue report that the same bug always exhibits the same stack trace. It’s an assumption with huge benefits. If this is true then you may already be very close to finding a cause by inspecting the trace.

However, the threat to validity is your induction based on a single observation. If you are wrong you may now be going on a long wild goose chase. Better test such inductions first! For example by reproducing the stack trace on your own machine (good for other things as well).

Inductions happen often and often unknowingly in our brains. Implicit inductions are the nemesis of effective debugging. You should always cover your tracks by reflecting on your hidden assumptions and testing them quickly. Only then they can be a very effective debugging tool.

Inductions work well in both directions, cause to effect and effect to cause, simply because they allow you to ignore a lot of uninteresting detail without hiding the path between cause and failure.

The shortcoming of induction with respect to debugging is that it can not help you isolate a cause much, other then exclude a lot of unrelated nonsense. There usually is too much to think about to find the cause by elimination only. The only method of reasoning that can isolate is abduction.

Abduction is when you imagine a cause which would explain (some of) the observations you’ve made. It is informed hypothesis formulation, also known as guestimating, having a hunch or simply intuition.

How useful your imagination is, is evident when you guess right the first time. You’ve found the cause, fixed it and the bug disappeared. It’s effective. It’s genius! Abduction is the first and foremost power tool of the debugging person.

However, if you accidentally thought you were deducing the cause, and got it wrong, you’re now in for some serious soul searching… are you too dumb to understand this code? To avoid this existential crisis, you must always abduct knowingly. Only then it is worth the effort.

I learned from @AndreasZeller’s book on debugging this insightful thought: “the” cause can be defined as a minimal difference between a run that has the failure and one that does not. So to arrive at a plausible abduction, a good hypothesis, you can always “diff” program traces.

You can do this manually, by comparing issue reports, or reading source code or stack traces, or you can diff log files, or the output of print statements, whatever comes to mind. Do differential diagnosis like Dr House does: and the diff would be your first guess at a cause.

So abduction is your debugging power tool, if wielded explicitly and based on evidence. On the other hand it is nothing more than guessing, and so you might be in for a long debugging session if you keep guessing wrong.

When we confuse the three modes of reasoning we are bound to get into trouble. Say you think any null reference must always be caused by a method in the stack trace, and so you deduce that one of the methods is causing it… What will happen?

As an abduction this made sense; why not search for a cause near to the failure? It’s Occam’s razor at play. Ok, but if you thought you were deducing instead and you don’t find the actual cause, you “fix” any method on the trace, hide the failure and more deeply bury the cause.

To wrap this up, the message is to reflect, while debugging, what your brain is doing when reasoning about the problem: deducting, inducting or abducting, to mitigate their threats to validity and waste less time chasing your own tail and to not complicate future debugging tasks.