Text encoding research

[efonte]

Some information.

it may not be all right

'\xcb' = '-'
'\xde' = '*' (star symbol)
'\xff\x80' = new line
'\xff\x86' = new page
'\xfd\x03' = forces name
'\xff\x93' = character name
'\xff\x01' = number of GOTH
'\xff\x9d\x10' = bold text start
'\xff\x9e' = bold text end
'\xff\x8a\x0c' = button START
'\xff\x8a\x03' = button CROSS
'\xff\x8a\x02' = button CIRCLE
'\xff\x8a\t' = button UNK1
'\xff\x8a\n' = button UNK2
'\xff\x8a\x04' = button TRIANGLE
'\xff\x8a\x01' = button SQUARE
'\xff\x8a\x13' = button LEFT_AND_RIGHT
'\xff\x8a\x12' = button UP_AND_DOWN???

[gibbed]

FF 01 apparently being valid is very interesting to me, given what I know about the renderer for text.

FF 9D 10 bolding text is super interesting too, probably a bitset of flags.

I'm working on a visual table for the available insertable glyphs.

[efonte]

I think \xff\x84\x01\xff\x01 contains variable name because the next text:

That will be {VAR1} goth. Agreed?
Begging your pardons, but the oath-wage is\n{VAR1} goth, young master.

[gibbed]

Ah in this case that entire set of bytes is one "operation". FF 84 has three argument bytes. See text formatting doc.

So that FF 01 is not actually its own text formatting operation.

[efonte]

UNK = \xff\x83\x01\x01\xff\xf1\xf0

Learned {UNK}.
{UNK} recruited!
You obtained the {UNK}.\nWould you like to learn it?

[efonte]

FF 89 ?? It can be the sleep time between text messages.

[efonte]

FC ff ww ww ll uu .. I think that uu has the following possible values: \x08 or \x10 or \x18

[Cryosaur]

From my tinkering, the 0xFC header is something like this:

[StructLayout(LayoutKind.Sequential, Pack = 1, Size = 0x05)]
public struct TextMetadata
{
	public const byte Signature = 0xFC;

	public byte Chunks;
	public byte Width;
	public byte Lines;
	public byte Length;
	public byte Flags;

	public readonly bool Flag101 { get => (Chunks & 0x01) != 0; }
	public readonly bool Flag340 { get => (Lines & 0x40) != 0; }
	public readonly int VisualWidth { get => Flags == 0 ? 0 : ((((Chunks ^ 1) / 2) * 254) + Width) / 2; }
	public readonly int NonInterpolatedLength { get => Flags == 0 ? 0 : ((Lines & ~0x40) - 1) * 63 + Length; }
}

For text strings that have interpolated values (embedded format codes), I have only seen both Flag101 and Flag340 active. But I have only looked at a few text arrays so far.

NonInterpolatedLength is the number of characters in the string + the null terminator - any interpolated values.

VisualWidth is a pixel width for screen display. Not sure where else it is used, but I know it is used in the info bar--top region with (i) glyph--to control scrolling long strings into view when they are visually larger than screen width.

No idea what the Flags byte is actually for, or if there are any more flags inside the other fields. I used to think the Flags byte was for character bit width; that may still be true, but it seems unlikely now that @gibbed found the font "files". But I have not yet really examined any of the text strings with the 0x10 and 0x18 values.

[Cryosaur]

Actually, thinking about it again, it seems even more likely that the "Flags" field should be CharSize. I originally thought it was from the JP version of the game likely using the same internal formats, but now that we know the fonts do include "characters" that have values > 0xFF, they must be represented by least two bytes.

I have not yet looked into it, but maybe FD or FE is the first value in these two-byte characters?

[efonte]

@Cryosaur This file D:\PSPISO\TOLUCT\PSP_GAME\USRDIR\FILETABLE_unpacked\1000\1\40.xlc has 0x10 value \xfc \x03 \xf2 A ) \x10

b'\xfc\x03\xf2A)\x10\x03\xf20001\x02\xdap001Experience earned: \xff\x84\x01\xff\x01\xff\x80Skill points earned: \xff\x84\x02\xff\x01\x00'

[efonte]

D:\PSPISO\TOLUCT\PSP_GAME\USRDIR\FILETABLE_unpacked\500\10241\9.xlc
0x10 values examples:

b'\xfc\x06\x8aB\x03\x10Play Time\xff\x80Amount of time that has elapsed since starting the game.\x00'
b'\xfc\x0e0C\x1a\x10Enemies Slain\xff\x80Total number of units you\'ve defeated. A breakdown by clan follows below, with monsters and any unallied persons categorized as "Unknown."\x00'

0x18 values examples:

b'\xfc\x04\xf6B\x1e\x18Guest\xff\x80A unit that has temporarily joined your party.\xff\x80You cannot issue commands to guest units.\x00'
b'\xfc\x06\x16C\x08\x18L: Lawful units respect the rules of society.\xff\x80N: Neutral units strive to attain balance.\xff\x80C: Chaotic units value freedom above all else.\x00'    

0x28 values examples:

b'\xfc\x06<D\t(Your progress in the story has changed significantly\nsince your last save. You may want to return to the\nworld map to save your progress before proceeding.\n\nAre you sure you want to jump to this point?'

[efonte]

In FC ff ww ww ll uu .. the uu is the number of lines * 8.

[gibbed]

Hmm, I'd say that's unlikely since the main fonts are 16x16, not 8x8. Could be related though.

[efonte]

Original texts inside the following EMES files.
D:\PSPISO\TOLUCT\PSP_GAME\USRDIR\FILETABLE_unpacked\902\240\3.unknown
D:\PSPISO\TOLUCT\PSP_GAME\USRDIR\FILETABLE_unpacked\908\650\2.unknown
D:\PSPISO\TOLUCT\PSP_GAME\USRDIR\FILETABLE_unpacked\909\90\37.unknown

b'\x00\x00\xff\xa9\xff\x9c\x01\x01\xff\x8d\x01\x02\xff\x85\x01\x01Coritanae \xff\x81\x01\x01Keep\xff\x99\x01\xc9\x00\xff\x80\x00'
b'\x00\x00\xff\xa9\xff\x9c\x01\x01\xff\x8d\x01\x02\xff\x85\x01=For \xff\x81\x01\x01nearly \xff\x81\x01\x01a \xff\x81\x01\x01thousand \xff\x81\x01\x01years,\xff\x893 \xff\x81\x01\x01the \xff\x81\x01\x01nation \xff\x81\x01\x01of\xff\x80\xff\x8d\x01\x02\xff\x85\x01=Valeria \xff\x81\x01\x01endured,\xff\x893 \xff\x81\x01\x01leaving \xff\x81\x01\x01an \xff\x81\x01\x01indelible \xff\x81\x01\x01mark\xff\x80\xff\x8d\x01\x02\xff\x85\x01=on \xff\x81\x01\x01the \xff\x81\x01\x01pages \xff\x81\x01\x01of \xff\x81\x01\x01history,\xff\x893 \xff\x81\x01\x01before \xff\x81\x01\x01at \xff\x81\x01\x01last\xff\x80\xff\x8d\x01\x02\xff\x85\x01=uniting \xff\x81\x01\x01with \xff\x81\x01\x01the \xff\x81\x01\x01people \xff\x81\x01\x01of \xff\x81\x01\x01Heth.\xff\x99\x01\xa1\x00\xff\x80\x00\xff\xff\x00\x00\x00\x00'
b'\x00\x00\xff\xa9\xff\x9c\x01\x01\xff\x8d\x01\x02\xff\x85\x01\x01Golyat \xff\x81\x01\x01Quayside \xff\x81\x01\x01\xcb \xff\x81\x01\x01Quarter \xff\x81\x01\x01to \xff\x81\x01\x01Four\xff\x99\x01\xf1\x00\xff\x80\x00'

Formatted text (\xFF\x81\x01\x01 removed for readability):

b'\xff\xa9 \xff\x9c\x01\x01 \xff\x8d\x01\x02 \xff\x85\x01\x01 Coritanae Keep \xff\x99\x01\xc9 \n'
b'\xff\xa9 \xff\x9c\x01\x01 \xff\x8d\x01\x02 \xff\x85\x01= For nearly a thousand years, \xff\x893 the nation of\n \xff\x8d\x01\x02 \xff\x85\x01= Valeria endured, \xff\x893 leaving an indelible mark\n \xff\x8d\x01\x02 \xff\x85\x01= on the pages of history,\xff\x893 before at last\n \xff\x8d\x01\x02 \xff\x85\x01= uniting with the people of Heth. \xff\x99\x01\xa1 \n'
b'\xff\xa9 \xff\x9c\x01\x01 \xff\x8d\x01\x02 \xff\x85\x01\x01 Golyat Quayside - Quarter to Four \xff\x99\x01\xf1 \n'

FF A9: Always at the beginning of the text
FF 9C ?? ??: Black message on screen (voice) ???
FF 99 ?? ??: Always at the end of the text
FF 8D ?? ??: Always at the beginning of the line
FF 85 ?? ??: Always at the beginning of the line
FF 89 ??: Waiting time between text blocks ???

Screenshots

[Faloppa]

Hi:

I'm sorry for the inconvenience but I would be interested in translating this game into my language, Spanish.

Checking everything, I see that it does not exist yet (or at least I was not able to find it) an .exe or app to do extract and insert text in order to translate it.

Is it within your plans to release an application similar to what I'm telling you about?

Thanks a lot