Target other toolchains/platforms

[madebr]

#1 added very basic project files that only supports a Windows MSVC workflow, using PDB files.
It would be useful to extend/rework this such that we can target other platforms and/or toolchains.

I propose a format in which we can add various data sources for targets.
It would still consist of 3 files (reccmp-project.yml, reccmp-user.yml and reccmp-build.yml).

• reccmp-project.yml is supposed to be pushed to a git repo, and should not contain absolute paths
• reccmp-user.yml should still not be part of a git repo, and is supposed to be modified by the user (although the tools can initialize it with defaults). This file can contain absolute paths.
• reccmp-build.yml should also not be commited to the repo, and is supposed to be modified by the build systems.

I created #7 to discuss file hierarchies.
The proposal in this issue can be developed independent of #7.

The search order is reccmp-build.yml -> reccmp-user.yml -> reccmp-project.yml
The priority order is cli-arguments > reccmp-build.yml > reccmp-user.yml > reccmp-project.yml

Different from #1, is that in the new proposal the yaml files work together more, are fully equivalent and stack on top of each other.
Values of reccmp-build.yml can override values in reccmp-user.yml, which in turn can override values of reccmp-project.yml.
The basic structure is as follows:

targets: # REQUIRED, value = map of targets
  <id-target1>:  # Only required when the current yml overrides properties
    <datasource1>: # Should be present for every data source property this yml overrides
      type: win32-dll|ghidra|ida|...  # REQUIRED: must be present (should not be overriden, but let's leave this unspecified)
      <various properties that depend on the type>
    <datasource2>: ...
  <id-target2>:  # Only required when the current yml overrides properties
    <datasource1>: # Should be present for every data source property this yml overrides
      type: win32-dll|ghidra|ida|...  # REQUIRED: must be present (should not be overriden, but let's leave this unspecified)
      <various properties that depend on the type>
    <datasource2>: ...

Example project file structure:

# reccmp-project.yml
targets:
  LEGO1:
    original:  # "original" can be whatever. It's a sub-id
      type: win32-dll  # can also be win32-exe (and also elf32, dylib32, ...)
                       # Perhaps also include arch here or somewhere?
      filename: LEGO1.DLL
      hash:
        sha256: 14645225bbe81212e9bc1919cd8a692b81b8622abb6561280d99b0fc4151ce17
    recompiled:
      type: win32-dll
      source-root: LEGO1
    ghidra:
      type: ghidra
      server: @server.mattkc.com:13100  # no username means anonymous
      repository: isle
      filename: LEGO1.DLL
    csv-symbols:
      type: csv
      path: symbols/lego1.csv

# reccmp-user.yml
targets:
  LEGO1:
    original:
      path: /games/isle/LEGO1.dll
    ghidra:
      path: /home/maarten/ghidra_projects/isle/isle.gpr
    ida:
      type: ida
      path: /home/maarten/ida_projects/isle/lego1.idb
      
# reccmp-build.yml
targets:
  LEGO1:
    recompiled:
      path: /home/maarten/projects/isle/build/LEGO1.DLL
      pdb: /home/maarten/projects/isle/build/LEGO1.PDB

This structure has 5 data sources which can be compared.

• LEGO1.original: the original binary
  • type = win32-dll
  • filename = LEGO1.dll
  • sha256 = 14645225bbe81212e9bc1919cd8a692b81b8622abb6561280d99b0fc4151ce17
  • path = /games/isle/LEGO1.dll
• LEGO1.recompiled: the recompiled binary
  • type = win32-dll
  • source-root = /LEGO1
  • path = /home/maarten/projects/isle/build/LEGO1.DLL
  • pdb = /home/maarten/projects/isle/build/LEGO1.PDB
• LEGO1.ghidra: a ghidra project
  • type = ghidra
  • server = @server.mattkc.com:13100 # (anonymous access)
  • repository = isle
  • filename = LEGO1
  • path = /home/maarten/ghidra_projects/isle/isle.gpr
• LEGO1.csv-symbols: csv file with symbols
  • path = /symbols/lego1.csv
• LEGO1.ida: an ida project
  • path = /home/maarten/ida_projects/isle/lego1.idb

We can then compare 2 data sources by running e.g.:

reccmp-datacmp --target LEGO1.original --target LEGO1.recompiled

We can use this hierarchy to still allow running reccmp's scripts without project files.
e.g.

reccmp-datacmp --custom --type win32-dll -f path/to/original.dll --custom --type win32-dll --path path/to/recompiled.dll  --pdb path/to/recompiled.pdb --source-root path/to/recompiled/source

What do you think?