RomHacking with Ghidra, Part 2: Viewing Game Text Using Table Files

On my third day at EPITECH, as I was learning the best programming language in the world, and making good progress towards the day’s exercises, which were mostly about reading input, characters, or various calculations, the last exercise became a wall to me. It was something a bit like this:

Write a program taking one argument and converting it from base “esoa” to base “qux”.

Before applying to this school, I knew that some day, my lack of knowledge in maths¹ COULD hinder my knowledge of programming, but I wasn’t expecting my progress to be stumped this early! What were those supposed to be? I had known about base 10 and base 16 ever since I first learned how to hex-edit my game saves. I even knew about more obscure numeral systems, like base 8² and Base64, so I was expecting that I could find something, somewhere, about these two bases, but alas, no amount of Google-fu could help me, and our “benevolent” TAs weren’t helping a bit. Until someone just gave me the answer:

Actually, that’s not a base’s name, but the description of the characters it uses. Like, “base 10” is “base 0123456789”. So in that case, a “base esoa” just means “base 0123”, and “base qux” is “base 012”. You just need to replace the characters.

I was enlightened. And pissed off, because I knew about this already, but under another name: this was just like tables in romhacking!

What’s a table in romhacking?

Codes, and Tables

Thinking back on this, half a lifetime later, it occurs to me this was closer to a character set than an actual “base”, but I guess for students with mathematical backgrounds, this was a better term to think with.

We barely use the term “table” nowadays, because it’s largely been replaced by Character encoding, and with UTF-8 being widely adopted, most users don’t really need to think in terms of character codes anymore³, and given the number of characters, there’s no real point in remembering them. So the only other table we still see as a standard is the ASCII table, which received its last major revision around 1968.

Let’s step back a little: what are “tables”, and what are they used for?

Well, it’s all about the computers and the CPUs and their evolution, or in this case, their lack of evolution. See, even today’s best MacBook Pro money can buy is STILL a pimped calculator. Sure, it can calculate a TON of stuff at once, it can generate videos, it displays your apps on two screens at once, it can play music,… but the processor is STILL a calculator: it only knows about NUMBERS.

And what are you reading here? Text. Words. Characters.

Of course, this is a problem older than computers. You ever heard of Morse code? That’s where the issue began⁴: the electrical telegraph could only transmit electric impulses, and humans needed a way to transmit letters using those signals. Therefore, the Morse system reduced messages to dots and dashes. Later on, machines got involved and the Baudot code was invented⁵, which became one of the first widely adopted standards to transmit letters to machines over electrical wires, one of the major milestones that led to the creation of ASCII.

Which worked just like it does for today’s computers: your computer does NOT know there is an “A” there: it knows there’s a 65, and will use the ASCII table to display an “A”.

I just wanna play (Japanese) video games

So where does romhacking fit into all of this?

Well, in the 80’s, consoles didn’t really have the luxury of carrying thousands of characters. Plus the console, like your calculator or your laptop, had no notion of what “text” was in the first place⁶, it only lets you draw tiles and sprites. Developers can store the characters they need as tiles and sprites, and display them on the screen. Yes, the characters they need: since they control the script, it’s easy to decide which glyphs should be included. And really, when the game is built, who cares what value a character is?

Which seems to be exactly what happened at Babel: every game published had its very own character encoding, something romhackers call a table.

As the gods laugh, the romhackers lament: before extracting and translating the text from a game, they must first find that game’s table. There are multiple tricks to do this, depending on the game type, or the tools you have access to. One I particularly like is the “RPG method”: you start a game that allows you to name your character, input “AAAAAA” as a name, then save. You open up your save in a hex editor, and look for a value repeated six times, and you’ve found the first entry of that game’s table encoding.

Hydras are stronger than towers

Which brings us back to Ghidra: as I recently began looking into romhacking a game, my first step after finding the bytes responsible for the start screen credits was to load the ROM in Ghidra to delimit that string. But since it wasn’t in a standard charset, there was no way I could get a proper preview in the navigation:

I assure you, these bytes say "© BIRD".

And of course, as every good engineer faced with a new problem, my attention shifted from romhacking to figuring out how to display the proper string in Ghidra, because while Ghidra’s “String” type allows to choose among a lot of encodings, there’s no way to define your own. Or is there?

Ghidra Has Head(ers) Everywhere

The best thing about Ghidra is that it’s open-source. The second best thing is its architecture: everything works as a plugin or class that registers itself in the program. Therefore, 99% of the time, when you need a new feature, it’s actually easy to look into one that was implemented and adapt it.

So let’s rack our brains for a second: what’s stopping us from building on top of the base StringDataType, and using that to make our own TblStringDataType?

public class StringDataType extends AbstractStringDataType A fixed-length string DataType with a user setable charset (default ASCII).

So that’s the catch: we cannot extend from StringDataType because it needs a charset, but we can easily extend from AbstractStringDataType and build up our own data type with its own representation. We can even make it modular: just like all data types come with a “Settings” member that drives the UI to create a selector (StringDataType uses it to select among the list of charsets), it becomes trivial to add settings to our TblStringDataType allowing users to choose the table they want to use to decode a string.

Because yes, every game had its own table, and many had more than one, so we need to save the table along with our program⁷. To solve this, I introduced a registry that stores every loaded table in the current program, while each TblStringDataType simply keeps a reference to the selected one. That part felt really uneasy for me, and for Apple developers, it reminded me of Apple’s UserDefaults, which is a pattern I’ve never really been fond of, but I’m (mostly) a web developer used to querying user preferences from a database, so maybe I’m biased there.

I still got that <FF> in the middle!

Seeing the magic come to life

As I was working on the features I used the Romhacking.net wiki page on Text Tables as a reference, and most notably the original format description. But I still wasn’t happy enough about the results: I was too high off building this plugin to go into searching the rest of my ROM for text⁸, yet I wanted more text to check the results. The Romhacking.net wiki came to the rescue again with its archive of .tbl files. I scrolled, until I stumbled on a name I liked: Terranigma. And would you believe that, it had a table specific to the French version!

Even better was the fact that this table included another space-saving trick that developers of that time often used: if you know a word is gonna come A LOT of times in your game’s text, why not register it in the table, and turn it into a dictionary in your dialogue box display function? Take this entry: E51C=Bienvenue, French for “Welcome”. Clearly it’s a word you’ll see coming up in towns, in shops,… It takes nine bytes and is bound to come up a lot, so if you turn it into a two-byte token, you save SEVEN bytes every time you want to write it. Of course, the Text table specification handles that kind of dictionary mapping and I had implemented it, but would it work?

I loaded a French ROM of Terranigma into Ghidra, imported the .tbl file, then looked around. By an EXTREME stroke of luck, the values for the lowercase letters “a” to “z” aligned with the ASCII table’s values for the uppercase letters “A” to “Z”, which resulted in a rather strange preview, as Ghidra likes to display untyped bytes as their ASCII value:

This one is (obviously) "Appelez-nous".

I selected the area, and a bit more, and applied the tblString data type to it. The default table was the only registered one, and the strings revealed themselves to me:

The game contains a thinly veiled ad for the Nintendo Hotline.

Now that I knew the plugin worked, I wanted to try it on a real script instead of isolated strings. Since the Romhacking.net wiki also had a page with the ROM map, I looked up the offsets for the text and dug into the ROM: I could make out multiple strings and give each of them a readable representation. Which is when I noticed the best part of adapting my data type out of AbstractStringDataType: Ghidra was cataloguing it as a string, and all the strings I was defining in the ROM were picked up by the “Defined Strings” tool: just by correctly using the available types, I had gained a new feature I had no idea about!

When I saw this, the 1812 Overture was playing in my head.

For one last feature before publishing, I thought it would be useful to be able to search for text. While it’s a bit complicated since a dictionary means the same word can be written in different ways, I actually found a way to make it work. I searched Google Images for a random game screenshot, so I could look up the text it contained: success.

"You’re not in danger, Colombus, I am the danger!"

Unfortunately, having been curious about romhacking from a very young age, I know this plugin, while useful for reverse engineering, is far from being a silver bullet: while I was lucky to find a game it worked on, even with a table, in a lot of big RPGs from this era, the dialogues are often stored in compressed form, mostly based on LZSS or Huffman since they offer a good compromise between size and performance. Who knows? Maybe that’s for a future release, when I find out how to (safely) do user-provided code evaluation in Java?

---

1. Obviously, I graduated in Literary Studies, which is why I speak both C, and Japanese. ↩︎

2. My favorite joke on the matter: “Why do programmers get Halloween and Christmas mixed up? Because Oct. 31 equals Dec. 25”. ↩︎

3. Though we still have some funny mathematical tricks we can perform, like: “🇫 + 🇷 = 🇫🇷”. ↩︎

4. Please don’t get pedantic about hieroglyphs, or any other system… ↩︎

5. Fun fact one: he’s French. Fun fact two: the baud unit comes from him. France’s reach is limitless. ↩︎

6. None of the classic cartridge-based/OS-less consoles ever did. The best I know is the 3DS which has specific font format and functions to handle text generation, but it’s mostly rendering and…quite frankly, it’s all just the same, but the system OS handles it. ↩︎

7. In Ghidra parlance, a “program” is the file you’re currently analyzing. ↩︎

8. And I know for a fact it’s not the same because that one is limited to the characters appearing in the credits. ↩︎