Destroying the evidence of a Crime Scene: Debugging the Codebase.

Debugging is like a murdered hunting the crime scene for the evidence he/she has left. It is a discipline which develops critical and analytical thinking. This thought process can be applied to many aspects of life.

Here are few Debugging Rules

1. Never approach with preconceived ideas.
   Don’t trust what others say, no one knows the codebase thoroughly.
   Only trust the logs.
   Don’t assume this can’t go wrong. Everything has a breaking point.
2. It might be a serious monster.
   If the program is doing something unexpected, then either the “program is wrong somewhere or the input is wrong somewhere or some external factors are affecting the program”. There is no other possibility.
3. An important part of debugging is reproducing the problem.
   This will help to verify the fix with a greater level of confidence.
4. Walkthrough and understand the code.
   Understanding the code in the first place is must to be able to find the bug. Understanding code broadly and deeply is essential to be able to find the elusive bugs.
5. Come up with the theory based on the code walkthrough and available evidence.
   This will help a lot in attacking the problem with a defined scope so that we don’t miss anything. Validate the theories by creating test cases based on the theories.
6. Start by testing for the simplest use case.
7. Isolate the problem at the root and start bug fixing from the top
8. Change one thing at a time. Not randomly though.
9. Search for Clues.
   Based on the evidence sometimes we might get a clue on what might have gone wrong. Hard code such things and see if the problem is getting hit while debugging.
10. Examine assumptions.
    Trust but verify. Verify everything. Verify even the test cases. Trust only the logs.
11. Use logs to check what is happening at each stage.
12. Use debugger.
    Step In, Step Over, Jump to the cursor, Jump Out, Watchpoint and Breakpoint are all essential features.
13. Beware of Corruption Issues.
    Sometimes you might have hit on a corruption issue, which are the comparatively hard ones to debug than normal bugs.

• First, we need to identify the source of corruption and the corrupted entity.
• Memory corruptions are usually because of illegal access or the race in the critical section.
• Data corruption could be because of :
  1. External entity
  2. Software bug
  3. Hardware issue
  4. Lost write etc.

14. Look out for Performance Issues.
If you are very unlucky you might have hit on a Performance issue. Performance issues are very hard to reproduce and might occur intermittently.

• For debugging performance issue we need to first identify the baseline test. This is a sample unit test which could be easily run and verify the slowness.
• Eliminate the doubt of external factors, make sure the issue is not caused by some external factor.
• In the case of comparison involved with other benchmark programs, make sure the comparison is done correctly.
• Scan through each layer and see if there is anything going unexpected and causing delays.
• Collect evidence at different layers/modules.
• Try to Identify the module/layer having bottleneck or delays.
• Once we identify the module/layer causing the delays, add time checks on entry and exit points to narrow down the exact problem.
• Fixing is also a bit tricky with performance issues. Fix has to be thoroughly tested and checked against any regressions. “Performance fixes are really tricky and prone to cause regressions”.

15. There is always one more bug.
Finally, always assume there is at least one more bug. Never think, the bug you found is the final one.

Thanks for reading. Cheers!!! Feel free to leave your comments below.