Problems Getting The Debugger To Work With Nintendo 64

[flagrama]

I do a fair bit of for-fun software development and reverse engineering on the Nintendo 64 hardware and software. Debugging is always an issue though because only a couple of Windows-only emulators have fleshed out working debuggers. On the other hand, the ares emulator has a gdbstub which works pretty well for debugging your own code, but isn't quite as useful if your code is a modification to an existing binary.

I already use Ghidra for documenting these existing binaries, so I figured using it's debugger with ares' gdbstub would be pretty easy and would let me reference my documentation easier. So far it has been difficult to get the debugger to work properly in the first place.

First, I don't know if this is relevant, but I use the N64LoaderWV extension to import the ROMs and give a decent starting point for documentation. Maybe it is missing information that it should be providing to make this work easier.

Second, when I run the debugger normally I don't get any display in the dynamic view, memory view, regions, etc. Instead in the terminal I get overflow exceptions about converting a negative signed integer to a unsigned integer. This seems pretty obvious to me, it's a sign-extension issue. The PC and general purpose registers of the N64 CPU are all 64-bit sign extended, so for example the beginning of RDRAM at 0x80000000 will be stored as 0xFFFFFFFF80000000 when in the PC or a GPR.

Furthermore, if I manually edit commands.py and mask these addresses with 0xFFFFFFFF the debugger progresses much further. I'm able to actually see values in the memory view, the regions tab has a region at least. However this does not yet solve all my issues. gdb is still quite unhappy being unable to find the beginning of functions. Trying to step through code seems to increment randomly rather than 4 bytes at a time. Specifically it seems to run until a branch or delay slot, I didn't pay super close attention to which. And trying to set breakpoints fails. In the dynamic view they fail saying that the value 0xFFFFFFFF80xxxxxx is out of range. Trying to set them elsewhere, like in my listing with all the documented code it generally says something about not being in the tracepath or the same snap or something.

So I'm wondering what exactly I have to do to improve the situation from here.

Does the N64 loader need to provide NEC VR4300 specific data files? Do data files need to be added to Ghidra's source somehow? Is there something missing in existing data files? Something entirely different?

I'd appreciate any help I can get with this as I believe it would help my workflow immensely.

[d-millar]

@flagrama Just wanted to let you know that your request has not been ignored - there already may be a fix in-play, but I need to verify that. Will let you know when I know.

[d-millar]

@flagrama Apologies for the delayed response - am trying to verify some of our fixes for related issues but having a difficult time getting gdb to work with the Ares emulator at all. Any tips regarding setup? (Am testing running “Sonic the Hedgehog”, but have no registers, no disassembly, and no ability to break.)

[flagrama]

The GDB server is intended for all systems to use eventually, but only the N64 core actually has implemented it currently. (ares-emulator/ares#1223)

[d-millar]

I have yet to find an N64 rom that works "correctly" in gdb proper, so am considering closing this issue until the emulator is in a more stable state. Objections?

[flagrama]

I do know retail ones don't seem to work in any way with gdb in my experience. However, I've definitely worked with the Zelda: Ocarina of Time decompilation with added debug symbols which seemed to have worked fine. Perhaps I could create or find some homebrew that works well with gdb? It's not something I can do right away, but once I do I could link it here.

[mracsys]

I have been tracking this on and off. Sorry for missing the conversation last week.

Example video of gdb working with ares on the Ocarina of Time randomizer romhack: https://www.youtube.com/watch?v=QdERiG-p0Kw&t=187s

The file I passed to gdb is at https://github.com/mracsys/OoT-Randomizer/blob/gdb_debugging/ASM/build/gdb_bundle.o. This contains only the the randomizer's added code, no Nintendo copyrighted-assets. Ares needs a full rom patched with that branch of the randomizer.

It would be helpful to use ghidra for lower level debugging without resorting to raw gdb commands. I have the same problems as @flagrama

[flagrama]

@d-millar there seems to have been a bug in attaching gdb to ares' n64 core in versions 140 and 141 which should now be fixed in nightly builds. Which 140 was likely the version you tested.

I confirmed with @mracsys that they were using 139 for that video.

Perhaps one of you could give the ares nightly build a try? I don't have anything set up to try it myself at the moment to be honest.

[d-millar]

@flagrama that works much better! we still have a bug in our code (pc is being interpreted as negative somewhere). will let you know when I have a fix!

[d-millar]

hmmmm, ok - well, our bug aside, it still seems to me that the registers returned by the g packet are sketchy. When I connect from gdb (no Ghidra in the picture), "info registers" always return 0xFFFFFFFF for the PC, which seems questionable. Any chance I'm doing something clearly wrong? Am basically, enabling the gdbstub, loading a ROM, waiting for it to be running, and then connectiing. "Show arch" returns "mips:4000", which seems close enough, but the register values make no sense to me.

[flagrama]

I built the nightly because I'm on Linux and usually use the Flatpak. I also use a mips64 specific gdb because it's just part of the toolkit we build for the OoT randomizer and setting up multiarch gdb in Fedora seemed pretty complicated the last time I looked.

I've launched the normal 1.0 version of Zelda: Ocarina of Time here and am not seeing a PC of 0xFFFFFFFF, but 0x800007CC (the infinite loop of the main thread which is just an empty do ... while that runs while other threads do the actual work) and I can break on different parts of code depending on timing, though I don't seem to be able to see different threads to switch between them, so I'm basically stuck wherever it is putting me. Which ROM are you trying in particular?

[d-millar]

Well, if you can point me to the code for building the custom version of gdb, I can try that, assuming it supports MI2, or at least see if it works with the ROMs I have, which are random old games (Asteroids, Frogger, etc.).

[flagrama]

We use https://github.com/glankk/n64 which builds binutils, gcc, newlib, and gdb from freshly downloaded source. You should only have to go as far as step 5 in the Building From Source section I think, the rest should only be needed for software dev. If features are missing from the builds that aren't just because of missing dependencies, they should be fairly easy to add to the build-{app} sections in the makefile.in and we can PR or fork the changes needed later.

[d-millar]

OK, debugger is working (more or less) using mips64-gdb and a fix for the negative address problem. If you want to try the fix yourself, edit the eval_address method in Debugger-agent-gdb/src/main/py/src/ghidragdb/commands.py to test the return and fix negative values, along the lines of:

try:
    val = int(gdb.parse_and_eval(address))
    if val >= 0:
        return val
    language, compiler = arch.compute_ghidra_lcsp()
    if ":64" in language:
        return (1<<64) + val
    else:
        return (1<<32) + val
except gdb.error as e:
    raise gdb.GdbError("Cannot convert '{}' to address".format(address))

It's a pretty hacky fix and with luck we can do better, but it addresses the problem for the moment. Because this code gets called in the event handler, it has a cascading set of unfortunate consequences. That said, I'm not sure it fixes all of your problems. In particular, the execution I'm looking at appears to be single-threaded, so I don't know whether the thread-switching issue will persist. Are you actually seeing multiple threads and, if so, are you changing threads by double-clicking in the Model View, the Threads View, or by issuing raw gdb commands in the Terminal? If not, is the emulator spawning subprocesses? (This just occurred to me, so will check myself tomorrow.) If it is, we may have to modify flags to allow gdb to follow child processes. This is exposed in one of the launchers, but not, I think, in remote-gdb.

[flagrama]

Are you actually seeing multiple threads and, if so, are you changing threads by double-clicking in the Model View, the Threads View, or by issuing raw gdb commands in the Terminal? If not, is the emulator spawning subprocesses?

I don't think the emulator can actually report the threads the game is spawning, so that's probably just something we have to workaround that Ghidra won't be able to fix. When a breakpoint is hit, we'll be on the thread that hit it, and there are only a handful of threads the game (and hopefully any others) spawns. No big deal there.

The biggest issue is I'm still not able to have the dynamic window to update to be able to set breakpoints with ghidra's interface, and there still seem to be some details missing which I'm not sure which are Ghidra's fault and which are the emulator's fault (stack, memory).

Manually setting a breakpoint in the terminal in Ghidra gives Exception in batch operation: TraceRmiError('ghidra.program.model.address.AddressOutOfBoundsException: Range [ram:ffffffff8009cde8+0] entirely exceeds space max') and similar when the breakpoint is hit. That part at least seems like something your python script is having issues with. There doesn't seem to be anything in the UI that updates for any of this except for the list of breakpoints in the Model window.

I'm also going to take a look at some other emulators with gdbstubs and compare them to ares' and see if the support in the n64 core needs to be extended to get some of the other stuff working.

[d-millar]

OK, the fix for the breakpoints is basically the opposite of the one above. If you look in commands.py around line 1126 (may be slightly off of that, I'm playing around in master), you see a line that reads:

  base, addr = mapper.map(inf, l.address)

replace it with something like:

  laddr = l.address & 0xFFFFFFFF
  base, addr = mapper.map(inf, laddr)

Obviously, this is a bit of a hack designed for your system only, but it'll do the trick for now. The address in the location is sign-extended (for reasons that completely elude me). Passing it into mapper.map creates issues.

Re the dynamic window, go to the Regions View and use the upper right pull-down to "Force Full View". The entire address space will then be visible. Alternatively, you can use the template provided in Ghidra to fake out "info proc mappings", which the Ares gdbstub doesn't appear to support.

This should (fingers-still-crossed) get you to the point of having a useable (albeit perhaps not friendly) debugger.

[flagrama]

The address in the location is sign-extended (for reasons that completely elude me).

This is just a quirk of how most software was developed on the N64 and how this particular MIPS processor handles it as I understand. 64-bit CPU running in 64-bit mode with 32-bit binary compatibility running (primarily) 32-bit software. All 32-bit operands get sign extended into 64-bit registers.

Anyway, this helped a lot. I can now set breakpoints via the terminal and the dynamic view and do single stepping. I did run into an issue where it was unhappy with breakpoints at first, but disconnecting and reconnecting I didn't run into it again. Thank you, this is definitely better than what we had before. I'll leave it to @mracsys to bring up any issues they run into.

Is it possible a less hacky method to handle these quirks will be included in the python scripts in the future?

[d-millar]

Yep, I just pushed the branch with the (slightly) less hacky implementation. Probably be a few days before it makes "master" and won't make the next release, but....