In my previous post I talked about one common frustration with heavily templated libraries: the verbose stack traces, and how that experience can be mitigated through gdb’s Python API.

Single-stepping through execution is often another trouble spot. Ideally we would stop only in our own code, but library functions - especially those that employ tag dispatching - can perform a seemingly interminable sequence of trivial function calls. Library functions that never call user code can be skipped with the next command, but for the rest, stepping can be quite tedious. A typical solution is to:

  1. Make an educated guess as to which functions and methods are likely to be called
  2. Set breakpoints on all of them
  3. Use continue to run forward until one is encountered

This can work but is error prone and time consuming. You might forget an important function without realizing it, and afterwards you have a bunch of stray breakpoints. What we need is a way to automatically locate downstream functions, insert the breakpoints, and clean up afterwards.

Implementation

Adding breakpoints is straightforward in Python:

bp1 = gdb.Breakpoint('main.cpp:25')   # file and line
bp2 = gdb.Breakpoint('Foo::bar')      # function name

But how can we get the necessary file and line information? gdb can tell us the current file and line number, but lacks the semantic information to know what functions may be subsequently called. Fortunately, with Python we have access to powerful help in the form of libClang’s Python bindings.

libClang

We can take gdb’s current frame information and use it to figure out where we are in the source:

frame = gdb.selected_frame()
line = frame.find_sal().line
fname = frame.find_sal().symtab.filename

Assuming your build flow creates a compilation database - a very useful thing for tools - you can recreate the command lines used to build this translation unit:

from os import path
from clang import cindex
compilation_database_path = path.dirname('./compile_commands.json')
compdb = cindex.CompilationDatabase.fromDirectory(compilation_database_path)
cmds = compdb.getCompileCommands(tu_fname)

It is often the case that the file the debugger has stopped in is not the one that was compiled, i.e., it was an included header. We can search through the current backtrace to find the translation unit:

tu_fname = None
while frame is not None:
    fn = frame.find_sal().symtab.filename
    cmds = compdb.getCompileCommands(fn)   # None if not found
    if cmds is not None:
        tu_fname = frame.find_sal().symtab.filename
        break
    frame = frame.older()
# extract args from cmds...

Now we have Clang (via libClang) parse our source and create an AST, using the flags we collected:

index = cindex.Index.create()
tu = index.parse(tu_fname, args)

We can get a cursor representing the AST node at our current location:

f = tu.get_file(fname)
loc = cindex.SourceLocation.from_position(tu, f, line, 1)
cur = cindex.Cursor.from_location(tu, loc)

We can further interrogate libClang:

  • are there any function or method calls in this node?
  • for each such call:
    • does the function name match our “library” regex?
    • if not, are any arguments non-POD objects?
      • what are the methods of each such object?
      • where does each method begin?
    • are there any lambda arguments?
      • where is the first line of each lambda body?

And so on. We can generate new breakpoints from these locations via the gdb API:

breakpoints.add(gdb.Breakpoint('%s:%d'%(fn, line), internal=True))

Run continue until we reach one:

gdb.execute("continue")

Then clean up the temporary breakpoints:

for bp in breakpoints:
    bp.delete()

We have now implemented a variant of the step command that skips uninteresting library code and lets us concentrate on our own stuff. You can access the full version on Github.

Sample Session

Here’s how the debugging experience looks now, in the example from my previous post:

$ PYTHONPATH=.. LD_LIBRARY_PATH=/usr/lib/llvm-5.0/lib gdb ./swl
GNU gdb (Ubuntu 8.0.1-0ubuntu1) 8.0.1
...
(gdb) python import gdb_util.stepping
(gdb) b main
Breakpoint 1 at 0xf51: file /.../gdb_python_api/stl_with_lambda.cpp, line 21.
(gdb) run
Starting program: /.../gdb_python_api/build/swl 

Breakpoint 1, main () at /.../gdb_python_api/stl_with_lambda.cpp:21
21      int main() {
(gdb) n
27              {"Monoid", "Endofunctor", "Monad"}};
(gdb) 
30          std::sort(data.begin(), data.end(),
(gdb) stepu

Breakpoint -24, <lambda(const Strings&, const Strings&)>::operator()(const Strings &, const Strings &) const (__closure=0x7fffffffd9d0, a=std::vector of length 3, capacity 3 = {...}, b=std::vector of length 3, capacity 3 = {...})
    at /.../gdb_python_api/stl_with_lambda.cpp:32
32                        if (a[0] < b[0]) {

Our custom stepu command has dropped us directly into the sort predicate lambda, without having to manually add breakpoints or single-step through std::sort.

Wrapping Up

The gdb Python API opens up a world of possibilities for tool development. Here we’ve taken just a single package (libClang) and integrated it to build some useful functionality.

The Python ecosystem is very rich; using tools available from it to inspect and modify running C++ code is tremendously exciting. Here are just a few ideas:

  • Graphical display of C++ data structures (pretty-printing on steroids)
  • Serving a REST endpoint for high-level debugging information
  • Finding shared_ptr reference loops with graph algorithms

I believe organizations with large codebases can easily justify investing in this kind of custom tooling.