Using DeepWiki as a Tester

DeepWiki reads code, and makes documentation to match. It's a Codebase Intelligence Tool.

DeepWiki uses LLMs on top of algorithmic analysis, so while it does contextualise terse output into something a human might take in, it's not definitive. Unlike some of its rivals, it only really reads code/docs/config/imports/repo structure and not issues/commits/PRs/ADRs/meeting notes/minds, so it has no context about history nor intention. Which is probably to the good – inference about human action is best done by someone, not something.

It is, of course, a sales funnel as well as a handy tool. It's easily and freely accessible, but only for open-source projects on GitHub. It's proprietary, so can be made not-free any time (or degraded, taken away, made dangerous or even improved).

It won't use local models, so DeepWiki / Devin / Cognition Labs are subsidising and encouraging your use of a farty / thirsty / nosy / just awful token monster.

It looks as though it could be linked to the projects it has processed. It is not. It looks authoritative. It is not. It is tempting to think of it as a great way to document an undocumented project. It's not, or at least, not really. Its un-checked output is eminently indexable and so contributes to the vast wave of slop that is turning search to shite. It feels not good to the makers of the code. And, dammit, it's not at all a wiki.

However, as code explainers go, it is as astonishing as you might imagine from a deterministic tool gussied up by a well-prompted large language model. Especially when that deterministic tool can stand on the shoulders of well-established and comprehensive open-source precursors. It's great for a tester wanting to better-understand an unfamiliar codebase, from broad strokes to fine details. Just so long as you stay a tester, sceptical of unsupported answers.

Here's a short tour, roughly following me as I pottered around curl, vaguely wondering about some details of that project's testing.

• Take a github URL. Change github to deepwiki and you'll get the documentation. Bosh.
  • https://github.com/curl/curl ? https://deepwiki.com/curl/curl !
  • Do check the Last indexed:  date, top-left...
  • The docs are typically immediate – DeepWiki are clearly cacheing, so there's a compromise between up-to-date and not-needing-to-do-it-again
  • You'll see a sensible and clickable table of contents, which seem to relate to the project, not just to the files. I popped into Test Suite Architecture; you might go somewhere else.
• Documentation includes readable diagrams, clickable links to highlighted core code, and you can ask questions.
  • Try the different ways of exploring the codebase
  • See how many ways to explore you can find
  • Consider what you might do to check that what you're absorbing is right.
• When you've identified something you want to share, you can share the url with your team or with future you, and the analysis comes along with it. Here are some I might want to share:
  • What patterns of tests actually get executed by valgrind
  • Tell me more about the helper methods in ExecResult. Where else appears to include helper methods?
  • Again, these seem to be cached. For me, the first query takes ~1 minute, re-entering is immediate. So the results may be out of date, or they might be up-to-date but different from what you expected to share. You don't get to pick. You do need to know.

Is what DeepWiki says true? Nope. Not that I can identify a specific untruth, because I don't know the truth well enough (and I've not spent the time to look for inconsistencies). But as any fule kno, not being able to spot a lie doesn't make something true. Here and here, DeepWiki said that something was true, when it was (somehow) in the code but not in the product.

Generated text is all imagined, every last word. And I'm using 'imagined' to mean something we don't properly have a word for, using it in the sense of some inhuman and uncanny truth-adjacent noise.

To my eye, DeepWiki's imagination seems close enough to give you good clues, seems guarded from offering plausible illusions about intent and reasons, and gives enough direct access to code that you can at least think about it yourself. You can be just as sceptical of its output as you are of your own conclusions: The map may be weird, but it can still help you to form your own educated opinion.

It's still a world away from dibbling about, as I did publicly in 2023, with the tools of that time. And a world away again from thrashing around with grep , a whiteboard and some departed dev's scribbled handover notes.

Go on. Go play, ask better questions than I did, and share them here.