UDB: The undo.io Time Travel Debugger

Introduction

UDB is a Linux interactive debugger sold by undo [undo-1]. It uses the same command interface as GDB. Like GDB you can use it either in a text terminal or integrated with an IDE such as VS Code or CLion. Integration means that you will get extra buttons/keyboard shortcuts to access some of the additional UDB features. I’m not going to describe using UDB with a kind of GUI. There is plenty of value in using GUIs (in particular, having automatically refreshed views of local variables and the call stack). That said, if you know the UDB command line then using a GUI will be easy. Personally, I use UDB and GDB mainly in TUI mode¹ (text user interface) which presents a split text screen with panes for source code, assembler and debugger commands.

I’ll start with a quick overview of how the GDB command line works and then I’ll go on to describe some of the things that you can go with UDB.

GDB basics

GDB is the GNU debugger [GDB-1]. With it you can debug executables (the ‘inferior’ in GDB lingo). GDB controls the inferior using the ptrace system call [ptrace]. Effectively this means that GDB is going via the kernel to start/stop/step/read registers and memory in the inferior process (Figure 1).

Figure 1

Normally you will have prepared your executable by building it with DWARF debug information [DWARF] (adding the -g or -g3 flag to GCC or LLVM compiler). You can still debug without debug information, but you won’t have the source code so you will have to work at the assembly code level.

Once your executable is prepared you can debug it

  gdb --args your_executable
  [options for your_executable]

This will give you the gdb prompt

  (gdb)

The inferior isn’t running at this point. Unless you need to debug the startup code that runs before main(), you can use the start command which does the equivalent of setting a temporary breakpoint on main(), running and then stopping at the start of main(). If you are using a modern Linux and GDB, it will then ask you if you want to download any missing debuginfo for libraries like libc and libstdc++. This is a great feature, saving you from having to install endless debuginfo packages.

GDB then prints the first line of source and the prompt. This isn’t a tutorial on GDB, so I’ll just add a table of frequently used commands related to executing code. Commands can be abbreviated as long as the abbreviation is not ambiguous.

There is one thing that is missing. Those commands are c[ontinue] forwards, n[ext] statement forwards, s[tep] statement forwards and fin[ish] function forwards. What if you could do all those backwards as well? That is where UDB comes in.

Before I launch into some details, why would you want to do that? Quite often, when you are debugging, the problem is something ‘nice’ (from a debugging perspective) like a segmentation fault. You run your executable, it crashes and the debugger stops. You look at the line of code where it crashed, check the variables and see that something is NULL and you can see the problem. Life isn’t always so easy.

The kind of problem for which UDB excels is when you have a problem where:

1. It is not obvious where things are going wrong (the NULL pointer comes from far, far away, in extreme cases via JITed code that your IDE can’t follow).
2. It takes a long time – maybe hours – to reach the code and condition where the problem is.
3. You don’t know enough about the problem to set a breakpoint just before it happens. An example of that could be when you know which function to debug but the function gets called thousands of times and it’s only when the arguments and class state have a particular combination of values that the error gets triggered. Even worse, that state condition could be non-deterministic so you can’t even methodically note the steps to reach the error and redo them every time you need to restart a debug session.

We’ve all had that sinking feeling when you have just spent all afternoon debugging to reach an error and you accidentally hit ‘next’ one time too many. That is very easy to do when you have been stepping for a long time and stepping has become an automatic reflex. If you are using UDB, that is no longer a problem.

UDB prompt and recording

The first thing that you need to know about UDB is that it needs to record events in order to be able to replay them. Recording is turned on by default. When you start the debugging session the prompt will be not running>. Then, when you run and stop somehow, it will show the recording status. With recording on that might be something like recording 128,774>. The number after recording is the number of events recorded. The events correspond to basic blocks (blocks of linear machine code ending with a branch statement). If you time travel backwards, the prompt will show you a percentage of the record buffer and the event number, such as 99% 128,773> That can be useful when you want to pinpoint exactly where a change occurs. Just keep a note of the event count before and after and then you can just keep going backwards and forwards until you find the change. In addition to the event count, UDB has commands that let you set a bookmark in the event history that you can return to later. UDB can also go back to a given event or wallclock time.

Since recording does have an overhead, you can turn it off with the –defer-recording command line option. You could then run to some breakpoint and avoid needlessly recording events that aren’t of interest. When you know that you are getting close to the code that you want to analyse you can turn recording on with urecord.

Additional UDB commands

UDB adds a set of reverse and undo commands to GDB. I’ll get to the undo commands in a moment. The reverse commands just do what the forward commands do but in the opposite direction.

Let’s take a small example:

   1  int a, b, c, d; 
   2 
   3  void g() 
   4  { 
   5     c = 3; 
   6  } 
   7 
   8  void f(void) 
   9  { 
  10      b = 2; 
  11      c = 3; 
  12      g(); 
  13  } 
  14 
  15  int main() 
  16  { 
  17      a = 1; 
  18      f(); 
  19      d = 4; 
  20  }

If you debug this with UDB and do the following

• start – advances to line 17
• n (for next) – advances to line 18
• <enter> (repeat next) – steps over f() and advances to line 19

At this point if you enter rn (reverse-next) it will take you back to line 18 before f() is called. At the same point, if you enter rs (reverse-step), it will take you back into function f() to line 12.

Now if you do the following after starting UDB:

• b 6 – set a breakpoint on line 12
• r – runs to the breakpoint at line 12

At this point the debugger is stopped near the end of function f(). This time, if you enter rfin (reverse-finish) then it will take you back to line 18 where f() is called.

In addition to the reverse commands, there are also more generic undo commands. The main two commands are uu (ugo undo), which is a generic ‘undo last command that changed your debug time’. That has a mirror command ur (ugo redo) which redoes the last command that changed the debug time. An example of when you might want to use uu is when you have just just done a continue to a breakpoint and you then realise that the breakpoint is too late. You you just use uu to go back to the point where you did the continue.

I have only covered the basics of using UDB. A couple of commands that I’ve never used yet are usave and uload, which allow you to save and reload the execution history. I imagine that can be a a great time saver when you are debugging large executables that take a long time to reach the problem code. There are numerous commands for getting information about the recorded history. I’m a bit fascinated by the ublame command, for debugging POSIX shared memory. That seems to me like a very niche thing to debug. There must have been a quite specific user request to develop such a feature.

How does it all work? UDB is a commercial product so the internals aren’t documented. As I understand it, the key things are to only record the strict minimum of things to keep the size of the recording history down. When going backwards and forwards PTRACE syscalls can be used to restore the registers and memory. In some cases, where UDB needs to make something that is non-deterministic repeatable, it has to fake the non-deterministic call. For instance, if you call clock_gettime then reverse and redo the same call to clock_gettime then the second call will get the same time value even though the physical clock will have changed. UDB does some magic in the background.

UDB in practice

A few months ago I was working on a problem in a large executable which was giving different behaviour when running in two different modes. The expectation was that they should both be very similar. In the second mode, there was an error which was occurring a long time (in terms of statements) after the real cause of the problem. The code involved was developed by several teams and I only had moderate familiarity with some parts. I did try ‘printf debugging’ using our log files. That was intractable as the log files were hundreds of megabytes in size and any kind of diff tool either took forever to work out the diffs or got completely lost synchronising the two logs. It took me several hours of parallel debugging to narrow down the place in the code where the two exes were diverging in behaviour. I could have done that with just GDB, setting breakpoints that slowly advance through the execution, restarting every time that I overshot the problem zone. With each restart of the debug session taking a few minutes that would have added a few more hours. With UDB I would just go backwards whenever I overshot. The final step in that debugging marathon did require extra work to compare some large structs containing many option flags. The source of the problem seemed obvious when I had isolated the correct flag.

More information

Undo.io has a YouTube channel [undo-2] I’ve watched a good few of the videos and they are of a good level technical quality. Greg Law, a co-founder of undo.io, is active on the conference circuit (Meeting C++, CppCon, ACCU, C++ on Sea). Mostly these talks are about GDB and debugging in general . There is one particular ACCU conference where he presented UDB that I recommend that you watch if you’d like to find out a bit more of what UDB does and how it works [Law24]. I did enter the raffle and got a 1 year licence which is part of the reason that I’m writing this (we also use it at work). undo.io is the main contributor to a Reddit community, r/cpp_debugging [Reddit], which is mainly links to the YouTube videos.

Last but not least, there is the undo.io documentation [undo-3]. I found the Quick Reference [undo-4] the best way to get familiar with the commands that UDB adds to the GDB interface.

UDB isn’t the only debugger with time travel ability. Wikipedia [Wikipedia-1] lists several. If you don’t have access to a licence for UDB then you could look at rr [Wikipedia-2] (developed by Mozilla) or the reverse debugging capabilities of GDB itself [GDB-2] (also a tutorial [Redhat]). I’ve never used these myself but I have heard that UDB has better performance and is more reliable.

Conclusion

UDB is great. I’ve had a few niggles with it like reverse debugging multi-threaded applications (this has been improved according to the release notes for more recent versions). I sometimes had issues with our licence server – it didn’t seem to like UDB going back in time. Otherwise, it just works. I do have a vague idea of the gymnastics that it must be doing under the hood like JITting little bits of code so that it can patch up what you see executing with the recorded history. Getting all that to work is an amazing technical achievement. And to use it, most of the time all that is needed is type an extra ‘r’.

References

[DWARF] https://dwarfstd.org/

[GDB-1] https://sourceware.org/gdb/

[GDB-2] Reverse debugging capabilities: https://sourceware.org/gdb/wiki/ReverseDebug

[Law] Greg Law, YouTube channel: https://www.youtube.com/results?search_query=%22Greg+Law%22

[Law15] Greg Law ‘Give me 15 minutes and I’ll change your view of GDB’, presented at CppCon 2025 and available at: https://www.youtube.com/watch?v=PorfLSr3DDI

[Law24] Greg Law, ‘Time Travel Debugging’, presented at ACCU 2024 and available at https://www.youtube.com/watch?v=n3OCQ35Xhco

[ptrace] https://www.man7.org/linux/man-pages/man2/ptrace.2.html

[Reddit] Debugging community: https://www.reddit.com/r/cpp_debugging/

[Redhat] ‘Using GDB to time travel’, published 8 August 2024 at https://developers.redhat.com/articles/2024/08/08/using-gdb-time-travel#

[undo-1] https://undo.io/

[undo-2] YouTube channel: https://www.youtube.com/@Undo-io

[undo-3] Documentation: https://docs.undo.io/

[undo-4] Quick reference: https://docs.undo.io/UDB-quickref.pdf

[Wikipedia-1] Time travel debugging: https://en.wikipedia.org/wiki/Time_travel_debugging

[Wikipedia-2] rr (debugging): https://en.wikipedia.org/wiki/Rr_(debugging)

Footnote

1. In GDB, there are often 3 or 4 ways of doing the same thing (command line options, keyboard shortcuts, GDB commands and python code). Personally, I use keyboard shortcuts like Ctrl+x then a. There is a 15-minute video introduction to TUI [Law15].

Paul Floyd has been writing software, mostly in C++ and C, for about 35 years. He lives near Grenoble, on the edge of the French Alps, and works for Siemens EDA developing tools for analogue electronic circuit simulation. In his spare time, he maintains Valgrind. He can be contacted at pjfloyd@wanadoo.fr