Compare commits
1 Commits
main
...
agent/bitt
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f6f77bdbeb |
103
tools/decomp-assist/call_target_diff.md
Normal file
103
tools/decomp-assist/call_target_diff.md
Normal file
@@ -0,0 +1,103 @@
|
||||
# Direct-call destination advisory
|
||||
|
||||
`call_target_diff.py` checks the callees of functions reported as objdiff100.
|
||||
A 100% instruction score can still conceal a different relocation target. This
|
||||
tool compares resolved destination addresses with branches decoded independently
|
||||
from the original 43U DOL. It does not change source, build outputs, configuration,
|
||||
CI, or the completion policy.
|
||||
|
||||
Requires Python and pyelftools, as do the neighboring ELF tools. Run from the
|
||||
repository root after preparing the normal 43U objects and report:
|
||||
|
||||
```sh
|
||||
python tools/decomp-assist/call_target_diff.py --output /tmp/call-targets.json
|
||||
python tools/decomp-assist/call_target_diff.py src/scene/setting/iplSetting
|
||||
python tools/decomp-assist/call_target_diff.py UNIT --function EXACT_SYMBOL
|
||||
python -m unittest discover -s tools/decomp-assist -p test_call_target_diff.py
|
||||
```
|
||||
|
||||
With no unit arguments, it scans the report. Unit names omit the `main/` prefix
|
||||
and `.o` suffix. `--function` requires exactly one unit and still requires that
|
||||
function to be reported100. JSON goes to stdout unless `--output` is supplied.
|
||||
The output path cannot overwrite an input.
|
||||
|
||||
`--root` selects another prepared repository. `--report` and `--dol` override
|
||||
input paths; otherwise they are `ROOT/build/43U/report.json` and
|
||||
`ROOT/orig/43U/00000008.app`. Relative overrides are relative to the current
|
||||
working directory. The DOL must match the known 43U SHA1 by default; `--dol-sha1`
|
||||
accepts an explicitly verified alternative hash, including authored test inputs.
|
||||
The configuration remains 43U.
|
||||
|
||||
## What it checks
|
||||
|
||||
Only exact-name functions with equal source/reference extents and
|
||||
`fuzzy_match_percent == 100` are compared. Missing/ambiguous functions and
|
||||
unequal extents are reported as skips. Direct unconditional PPC `bl` instructions
|
||||
are decoded whether they have explicit `R_PPC_REL24`/`R_PPC_ADDR24` relocations or
|
||||
already contain resolved branch displacements. Signed relative and absolute
|
||||
encodings are supported.
|
||||
|
||||
Section-relative relocation targets and raw local branches resolve through
|
||||
actual function extents. Aliases sharing one function start are retained;
|
||||
overlapping distinct function starts remain unresolved. A matching reference
|
||||
symbol or an unambiguous configured address maps the source callee into the
|
||||
original address space. The original DOL supplies the reference call destination,
|
||||
and reference-object disagreement is an input error. Symbol spelling alone is
|
||||
not accepted as proof that two distinct callees are equivalent.
|
||||
|
||||
Rows classify:
|
||||
|
||||
- `different_destination`: source resolves to a different original address
|
||||
- `unresolved_destination`: source has no unique original-address mapping
|
||||
- `same_address_alias`: different names resolve to the same original address
|
||||
- `call_shape_difference`: only one object has a direct call at that offset
|
||||
|
||||
Rows are candidates for inspection, not automatic bug reports. Different
|
||||
constructor addresses can have identical bodies; a generated assignment or
|
||||
anonymous wrapper can implement the same operation under another name. Optional
|
||||
`alias_hint` evidence identifies identical original leaf bodies or a local
|
||||
relocation-free body identical to the original callee. Hints require compatible
|
||||
extents, equal bytes, and conservative branch checks; they do not suppress rows
|
||||
or establish general semantic equivalence. Helpers containing relocations may
|
||||
remain unresolved even when manual review can establish equivalence. No symbol
|
||||
name, constructor prefix, or wrapper prefix is silently excluded.
|
||||
|
||||
An unresolved symbol is not necessarily a missing implementation: it may be a
|
||||
renamed helper, external symbol, or ambiguous map entry. Check source definitions,
|
||||
linker aliases, receiver types, and the actual callee before proposing a fix.
|
||||
|
||||
## Provenance and exits
|
||||
|
||||
JSON includes both observed Git heads, the original-DOL SHA1, and SHA-256 hashes
|
||||
of the exact report, symbol/split maps, object files, tool, and shared ELF helper
|
||||
read by the scan. Inputs are fingerprinted again at the end. A changed head or
|
||||
input produces an error rather than a clean result. No binary or assembly bytes
|
||||
are written into the report.
|
||||
|
||||
These fingerprints identify what was inspected. They do not prove that cached
|
||||
objects were rebuilt from the stated source commit or that the report is fresh.
|
||||
For a serious candidate, preserve its inputs and independently rebuild/verify
|
||||
that specific commit before changing source.
|
||||
|
||||
Exit statuses are advisory:
|
||||
|
||||
- 0: no review rows, skips, or input errors in the selected scope
|
||||
- 1: rows or skipped functions require review
|
||||
- 2: invalid, missing, inconsistent, or changing input
|
||||
|
||||
An empty or entirely nonexact selection can check zero calls. Always inspect the
|
||||
counts. A clean result does not establish completion, whole-program equivalence,
|
||||
or correct callee bodies. Indirect calls, tail calls, conditional branch-and-link,
|
||||
nonexact callers, and unequal extents are outside this pass. Runtime behavior,
|
||||
callback effects, and floating-point evaluation order still require focused
|
||||
review.
|
||||
|
||||
## Tests
|
||||
|
||||
Tests build small ELF32/PPC objects and DOL containers from authored instruction
|
||||
words and metadata in temporary directories. They cover an incorrect relocation
|
||||
destination despite identical instruction bytes, raw local branches, shifted
|
||||
function locations, interior targets, address/constructor/wrapper aliases,
|
||||
undefined destinations, malformed references, original-DOL disagreement,
|
||||
selection skips, input changes, and JSON exit behavior. No original executable
|
||||
bytes or assembly are embedded in the tool, documentation, or tests.
|
||||
383
tools/decomp-assist/call_target_diff.py
Normal file
383
tools/decomp-assist/call_target_diff.py
Normal file
@@ -0,0 +1,383 @@
|
||||
"""Advisory direct-call destination audit for objdiff100, equal-size functions.
|
||||
|
||||
Run from the repository root: python tools/decomp-assist/call_target_diff.py [unit ...]
|
||||
Checks REL24/ADDR24 relocations and raw PPC direct BL against the original DOL.
|
||||
Exit 0: no review rows or skips; 1: review needed; 2: invalid/changing input.
|
||||
This is not a completion gate or a proof of callee behavior. See call_target_diff.md.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
from collections import Counter, defaultdict
|
||||
from dataclasses import dataclass
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
from elftools.common.exceptions import ELFError
|
||||
from elftools.elf.elffile import ELFFile
|
||||
|
||||
import jump_table_diff as elf
|
||||
|
||||
DOL_SHA1 = "26116613f624061ba99c8d1a299aaa6efa85670d"
|
||||
|
||||
|
||||
def signed(value, bits):
|
||||
return value - (1 << bits) if value & (1 << (bits - 1)) else value
|
||||
|
||||
|
||||
def branch_target(word, address):
|
||||
"""Return a PPC I-form direct BL destination, or None for other instructions."""
|
||||
if word >> 26 != 18 or not word & 1:
|
||||
return None
|
||||
displacement = signed(word & 0x03FFFFFC, 26)
|
||||
return (displacement if word & 2 else address + displacement) & 0xFFFFFFFF
|
||||
|
||||
|
||||
class Inputs:
|
||||
"""Fingerprint exact bytes read; detect changes before presenting a result."""
|
||||
|
||||
def __init__(self):
|
||||
self.hashes = {}
|
||||
|
||||
def read(self, path):
|
||||
path = Path(path)
|
||||
raw = path.read_bytes()
|
||||
digest = hashlib.sha256(raw).hexdigest()
|
||||
previous = self.hashes.setdefault(str(path), digest)
|
||||
if previous != digest:
|
||||
raise ValueError("input changed while reading: " + str(path))
|
||||
return raw
|
||||
|
||||
def changed(self):
|
||||
return [path for path, digest in self.hashes.items()
|
||||
if not Path(path).is_file()
|
||||
or hashlib.sha256(Path(path).read_bytes()).hexdigest() != digest]
|
||||
|
||||
|
||||
@dataclass
|
||||
class CallObject:
|
||||
info: elf.ObjectInfo
|
||||
data: dict
|
||||
|
||||
def function(self, name):
|
||||
symbol = self.info.symbol(name)
|
||||
if symbol.kind != "STT_FUNC" or symbol.value % 4 or symbol.size % 4:
|
||||
raise ValueError("expected an aligned function: " + name)
|
||||
return symbol
|
||||
|
||||
def location(self, section, offset, mechanism):
|
||||
functions = [s for s in self.info.symbols if s.section == section
|
||||
and s.kind == "STT_FUNC" and s.value <= offset < s.value + s.size]
|
||||
# Multiple names at one start are aliases; overlapping starts are ambiguous.
|
||||
starts = {s.value for s in functions}
|
||||
start = next(iter(starts)) if len(starts) == 1 else None
|
||||
names = sorted({s.name for s in functions}) if start is not None else []
|
||||
return {"mechanism": mechanism, "names": names,
|
||||
"addend": offset - start if start is not None else 0,
|
||||
"section": self.info.sections[section][0], "offset": offset}
|
||||
|
||||
def call(self, function, offset):
|
||||
at = function.value + offset
|
||||
word = int.from_bytes(self.data[function.section][at:at + 4], "big")
|
||||
raw_target = branch_target(word, at)
|
||||
if raw_target is None:
|
||||
return None
|
||||
relocation = self.info.relocs(function.section).get(at)
|
||||
if relocation is not None:
|
||||
expected = 2 if word & 2 else 10 # R_PPC_ADDR24 / R_PPC_REL24
|
||||
if relocation.kind != expected:
|
||||
raise ValueError("direct BL has an unsupported relocation kind")
|
||||
location = self.info.location(relocation)
|
||||
if location is not None:
|
||||
section, value = location
|
||||
if section not in self.data or value % 4 or not 0 <= value < len(self.data[section]):
|
||||
raise ValueError("call relocation is outside an allocated section")
|
||||
return self.location(section, value, "relocation")
|
||||
symbol = self.info.symbols[relocation.symbol]
|
||||
return {"mechanism": "relocation", "names": [symbol.name],
|
||||
"addend": relocation.addend}
|
||||
if word & 2:
|
||||
return {"mechanism": "raw_absolute", "names": [], "addend": 0,
|
||||
"address": raw_target}
|
||||
return self.location(function.section, raw_target, "raw_relative")
|
||||
|
||||
|
||||
def read_object(path, inputs):
|
||||
raw = inputs.read(path)
|
||||
info = elf.read_object(path)
|
||||
inputs.read(path) # Shared ELF helper reads the path; reject a concurrent change.
|
||||
parsed = ELFFile(io.BytesIO(raw))
|
||||
data = {i: section.data() for i, section in enumerate(parsed.iter_sections())
|
||||
if section["sh_type"] == "SHT_PROGBITS" and section["sh_flags"] & 2}
|
||||
return CallObject(info, data)
|
||||
|
||||
|
||||
class Dol:
|
||||
def __init__(self, raw, expected_sha1):
|
||||
self.sha1 = hashlib.sha1(raw).hexdigest()
|
||||
if self.sha1 != expected_sha1:
|
||||
raise ValueError("original DOL SHA1 does not match --dol-sha1")
|
||||
if len(raw) < 256:
|
||||
raise ValueError("truncated DOL header")
|
||||
self.raw = raw
|
||||
header = struct.unpack(">64I", raw[:256])
|
||||
self.spans = []
|
||||
for index in range(18):
|
||||
offset, address, size = header[index], header[18 + index], header[36 + index]
|
||||
if size:
|
||||
if offset < 256 or offset + size > len(raw) or address + size > 0x100000000:
|
||||
raise ValueError("DOL section is outside its file/address space")
|
||||
if any(max(address, a) < min(address + size, a + n) for a, n, _ in self.spans):
|
||||
raise ValueError("overlapping DOL address sections")
|
||||
self.spans.append((address, size, offset))
|
||||
|
||||
def read(self, address, size):
|
||||
matches = [self.raw[offset + address - start:offset + address - start + size]
|
||||
for start, length, offset in self.spans
|
||||
if start <= address and address + size <= start + length]
|
||||
if size <= 0 or len(matches) != 1:
|
||||
raise ValueError("address is not in one DOL section: 0x%X" % address)
|
||||
return matches[0]
|
||||
|
||||
def call(self, address):
|
||||
return branch_target(int.from_bytes(self.read(address, 4), "big"), address)
|
||||
|
||||
|
||||
def read_maps(symbol_text, split_text):
|
||||
names, sizes, splits = defaultdict(set), defaultdict(set), {}
|
||||
for line in symbol_text.splitlines():
|
||||
match = re.match(r"(.*?) = (\S+):0x([0-9a-fA-F]+);(.*)", line)
|
||||
if not match or not re.search(r"type:(function|label)\b", match[4]):
|
||||
continue
|
||||
address = int(match[3], 16)
|
||||
names[match[1]].add(address)
|
||||
size = re.search(r"size:(0x[0-9a-fA-F]+|\d+)\b", match[4])
|
||||
if "type:function" in match[4] and size:
|
||||
sizes[address].add(int(size[1], 0))
|
||||
unit = None
|
||||
for line in split_text.splitlines():
|
||||
if line and not line[0].isspace() and line.endswith(":") and line != "Sections:":
|
||||
unit = line[:-1]
|
||||
splits[unit] = {}
|
||||
match = re.match(r"\s+(\S+)\s+start:0x([0-9a-fA-F]+)", line)
|
||||
if match and unit:
|
||||
splits[unit][match[1]] = int(match[2], 16)
|
||||
return names, sizes, splits
|
||||
|
||||
|
||||
def addresses(destination, reference, bases, names):
|
||||
if destination is None:
|
||||
return set()
|
||||
if "address" in destination:
|
||||
return {destination["address"]}
|
||||
result = set()
|
||||
for name in destination["names"]:
|
||||
local = [s for s in reference.info.symbols if s.name == name
|
||||
and s.kind == "STT_FUNC" and isinstance(s.section, int)]
|
||||
if local:
|
||||
for symbol in local:
|
||||
section = reference.info.sections[symbol.section][0]
|
||||
if section in bases:
|
||||
result.add(bases[section] + symbol.value + destination["addend"])
|
||||
elif len(names.get(name, ())) == 1:
|
||||
result.update(address + destination["addend"] for address in names[name])
|
||||
return result
|
||||
|
||||
|
||||
def position_independent_leaf(raw):
|
||||
"""Conservative filter for body-alias hints, never an equivalence proof."""
|
||||
for offset in range(0, len(raw), 4):
|
||||
word = int.from_bytes(raw[offset:offset + 4], "big")
|
||||
op = word >> 26
|
||||
branch_op = (word >> 1) & 1023
|
||||
if op == 19 and branch_op in (16, 528) and (word & 1 or branch_op == 528):
|
||||
return False # Indirect call or tail transfer.
|
||||
if op in (16, 18):
|
||||
displacement = signed(word & (0x03FFFFFC if op == 18 else 0xFFFC),
|
||||
26 if op == 18 else 16)
|
||||
if word & 3 or not 0 <= offset + displacement < len(raw):
|
||||
return False # Link/absolute branches or transfers outside the body.
|
||||
return bool(raw) and len(raw) % 4 == 0
|
||||
|
||||
|
||||
def alias_hint(source, destination, source_addresses, target_address, dol, sizes):
|
||||
if destination is None or target_address is None or len(sizes.get(target_address, ())) != 1:
|
||||
return None
|
||||
size = next(iter(sizes[target_address]))
|
||||
target_body = dol.read(target_address, size)
|
||||
if not position_independent_leaf(target_body):
|
||||
return None
|
||||
# Different known addresses can share a retail body (e.g. folded constructors).
|
||||
if len(source_addresses) == 1:
|
||||
address = next(iter(source_addresses))
|
||||
if sizes.get(address) == {size} and dol.read(address, size) == target_body:
|
||||
return "identical_original_leaf_bodies; current callee implementation not proved"
|
||||
# Renamed local helpers need no guessed symbol pairing or name allowlist.
|
||||
for name in destination["names"]:
|
||||
try:
|
||||
function = source.function(name)
|
||||
except ValueError:
|
||||
continue
|
||||
if destination["addend"] or function.size != size:
|
||||
continue
|
||||
relocations = source.info.relocs(function.section)
|
||||
if any(function.value <= at < function.value + size for at in relocations):
|
||||
continue
|
||||
body = source.data[function.section][function.value:function.value + size]
|
||||
if body == target_body:
|
||||
return "identical_relocation_free_local_body; alias requires review"
|
||||
return None
|
||||
|
||||
|
||||
def compare_function(source, reference, name, bases, names, sizes, dol):
|
||||
src, ref = source.function(name), reference.function(name)
|
||||
if src.size != ref.size:
|
||||
raise ValueError("function extents differ")
|
||||
section = reference.info.sections[ref.section][0]
|
||||
if section not in bases:
|
||||
raise ValueError("no original section base for " + section)
|
||||
caller = bases[section] + ref.value
|
||||
counts, rows = Counter(), []
|
||||
for offset in range(0, ref.size, 4):
|
||||
left, right = source.call(src, offset), reference.call(ref, offset)
|
||||
expected = dol.call(caller + offset)
|
||||
if left is None and right is None and expected is None:
|
||||
continue
|
||||
counts["calls"] += 1
|
||||
for prefix, destination in (("source_", left), ("reference_", right)):
|
||||
if destination:
|
||||
counts[prefix + destination["mechanism"]] += 1
|
||||
actual = addresses(left, reference, bases, names)
|
||||
target = addresses(right, reference, bases, names)
|
||||
if right and right.get("section") in bases:
|
||||
target = {bases[right["section"]] + right["offset"]}
|
||||
if bool(right) != (expected is not None) or (target and target != {expected}):
|
||||
raise ValueError("reference ELF/DOL disagreement at +0x%X" % offset)
|
||||
if left and right and expected is not None and actual == {expected}:
|
||||
counts["same_destination"] += 1
|
||||
if set(left["names"]) == set(right["names"]):
|
||||
continue
|
||||
classification = "same_address_alias"
|
||||
elif not left or not right:
|
||||
classification = "call_shape_difference"
|
||||
elif len(actual) != 1:
|
||||
classification = "unresolved_destination"
|
||||
else:
|
||||
classification = "different_destination"
|
||||
hint = alias_hint(source, left, actual, expected, dol, sizes)
|
||||
rows.append({"function": name, "offset": offset, "caller_address": caller,
|
||||
"classification": classification, "source": left, "reference": right,
|
||||
"source_addresses": sorted(actual), "reference_addresses": sorted(target),
|
||||
"dol_destination": expected, "alias_hint": hint})
|
||||
counts[classification] += 1
|
||||
return counts, rows
|
||||
|
||||
|
||||
def git_head(root):
|
||||
result = subprocess.run(["git", "-C", str(root), "rev-parse", "HEAD"],
|
||||
capture_output=True, text=True)
|
||||
return result.stdout.strip() if result.returncode == 0 else None
|
||||
|
||||
|
||||
def audit(root, report_path, dol_path, expected_sha1, units=(), function=None):
|
||||
inputs = Inputs()
|
||||
inputs.read(Path(__file__))
|
||||
inputs.read(Path(elf.__file__))
|
||||
before = git_head(root)
|
||||
report = json.loads(inputs.read(report_path))
|
||||
names, sizes, splits = read_maps(
|
||||
inputs.read(root / "config/43U/symbols.txt").decode(),
|
||||
inputs.read(root / "config/43U/splits.txt").decode())
|
||||
dol = Dol(inputs.read(dol_path), expected_sha1)
|
||||
counts, rows, skipped, errors = Counter(), [], [], []
|
||||
selected = set(units)
|
||||
found = set()
|
||||
for unit in report["units"]:
|
||||
key = unit["name"].removeprefix("main/")
|
||||
if selected and key not in selected:
|
||||
continue
|
||||
found.add(key)
|
||||
functions = [f for f in unit.get("functions", [])
|
||||
if f.get("fuzzy_match_percent") == 100
|
||||
and (function is None or function == f["name"])]
|
||||
if not functions:
|
||||
if function:
|
||||
skipped.append({"unit": key, "function": function,
|
||||
"reason": "function absent or not objdiff100"})
|
||||
continue
|
||||
try:
|
||||
source = read_object(root / "build/43U/src" / (key + ".o"), inputs)
|
||||
reference = read_object(root / "build/43U/obj" / (key + ".o"), inputs)
|
||||
except (OSError, ValueError, ELFError) as error:
|
||||
errors.append({"unit": key, "reason": str(error)})
|
||||
continue
|
||||
bases = splits.get(unit.get("metadata", {}).get("source_path"), {})
|
||||
counts["units"] += 1
|
||||
for entry in functions:
|
||||
try:
|
||||
src, ref = source.function(entry["name"]), reference.function(entry["name"])
|
||||
if src.size != ref.size:
|
||||
raise ValueError("function extents differ")
|
||||
except ValueError as error:
|
||||
skipped.append({"unit": key, "function": entry["name"], "reason": str(error)})
|
||||
continue
|
||||
try:
|
||||
measured, result = compare_function(source, reference, entry["name"],
|
||||
bases, names, sizes, dol)
|
||||
except (ValueError, KeyError) as error:
|
||||
errors.append({"unit": key, "function": entry["name"], "reason": str(error)})
|
||||
continue
|
||||
counts.update(measured)
|
||||
counts["functions"] += 1
|
||||
rows.extend(dict(row, unit=key) for row in result)
|
||||
for missing in sorted(selected - found):
|
||||
errors.append({"unit": missing, "reason": "unit absent from report"})
|
||||
after = git_head(root)
|
||||
changed = inputs.changed()
|
||||
if before != after or changed:
|
||||
errors.append({"reason": "inputs changed during audit", "paths": changed})
|
||||
result = {"advisory": True, "counts": dict(counts), "rows": rows,
|
||||
"skipped_functions": skipped, "errors": errors,
|
||||
"provenance": {"head_before": before, "head_after": after,
|
||||
"dol_sha1": dol.sha1, "sha256": inputs.hashes},
|
||||
"limits": "Direct BL, objdiff100 and equal extents only. Alias hints require review; "
|
||||
"no callee-behavior, build-freshness or completion proof."}
|
||||
return result, 2 if errors else 1 if rows or skipped else 0
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("units", nargs="*")
|
||||
parser.add_argument("--root", type=Path, default=Path("."))
|
||||
parser.add_argument("--report", type=Path, help="default: ROOT/build/43U/report.json")
|
||||
parser.add_argument("--dol", type=Path, help="default: ROOT/orig/43U/00000008.app")
|
||||
parser.add_argument("--dol-sha1", default=DOL_SHA1)
|
||||
parser.add_argument("--function", help="one exact-name objdiff100 function; requires one unit")
|
||||
parser.add_argument("--output", type=Path, help="write JSON here instead of stdout")
|
||||
args = parser.parse_args(argv)
|
||||
if args.function and len(args.units) != 1:
|
||||
parser.error("--function requires exactly one unit")
|
||||
try:
|
||||
result, status = audit(args.root, args.report or args.root / "build/43U/report.json",
|
||||
args.dol or args.root / "orig/43U/00000008.app",
|
||||
args.dol_sha1, args.units, args.function)
|
||||
output = json.dumps(result, indent=2) + "\n"
|
||||
if args.output:
|
||||
if args.output.resolve() in {Path(p).resolve() for p in result["provenance"]["sha256"]}:
|
||||
raise ValueError("output must not overwrite an audit input")
|
||||
args.output.write_text(output)
|
||||
else:
|
||||
print(output, end="")
|
||||
return status
|
||||
except (OSError, ValueError, KeyError, ELFError) as error:
|
||||
print("ERROR: " + str(error), file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
316
tools/decomp-assist/test_call_target_diff.py
Normal file
316
tools/decomp-assist/test_call_target_diff.py
Normal file
@@ -0,0 +1,316 @@
|
||||
"""Authored synthetic PPC/ELF/DOL fixtures only; no original executable bytes."""
|
||||
import contextlib
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
from pathlib import Path
|
||||
import struct
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
import call_target_diff as tool
|
||||
|
||||
BASE = 0x80001000
|
||||
BLR = 0x4E800020
|
||||
LI_ONE = 0x38600001
|
||||
LI_TWO = 0x38600002
|
||||
|
||||
|
||||
def put_word(data, offset, word):
|
||||
data[offset:offset + 4] = word.to_bytes(4, "big")
|
||||
|
||||
|
||||
def bl(address, destination, absolute=False):
|
||||
displacement = destination if absolute else destination - address
|
||||
return 0x48000001 | (displacement & 0x03FFFFFC) | (2 if absolute else 0)
|
||||
|
||||
|
||||
def make_elf(text, symbols, relocations):
|
||||
"""Minimal ELF32 big-endian PowerPC relocatable, built from authored fields."""
|
||||
strings = bytearray(b"\0")
|
||||
symbol_data = bytearray(16)
|
||||
for name, value, size, section, kind in symbols:
|
||||
name_offset = len(strings)
|
||||
strings.extend(name.encode() + b"\0")
|
||||
symbol_data.extend(struct.pack(">IIIBBH", name_offset, value, size,
|
||||
0x10 | kind, 0, section))
|
||||
relocation_data = b"".join(struct.pack(">IIi", offset, index << 8 | kind, addend)
|
||||
for offset, index, kind, addend in relocations)
|
||||
section_names = b"\0.text\0.symtab\0.strtab\0.rela.text\0.shstrtab\0"
|
||||
entries = [("", 0, 0, b"", 0, 0, 0),
|
||||
(".text", 1, 6, bytes(text), 0, 0, 0),
|
||||
(".symtab", 2, 0, bytes(symbol_data), 3, 1, 16),
|
||||
(".strtab", 3, 0, bytes(strings), 0, 0, 0),
|
||||
(".rela.text", 4, 0, relocation_data, 2, 1, 12),
|
||||
(".shstrtab", 3, 0, section_names, 0, 0, 0)]
|
||||
raw = bytearray(52)
|
||||
headers = []
|
||||
for name, kind, flags, data, link, info, item_size in entries:
|
||||
while len(raw) % 4:
|
||||
raw.append(0)
|
||||
offset = len(raw) if data else 0
|
||||
raw.extend(data)
|
||||
headers.append(struct.pack(">10I", section_names.index(name.encode() + b"\0"),
|
||||
kind, flags, 0, offset, len(data), link, info,
|
||||
4 if name else 0, item_size))
|
||||
while len(raw) % 4:
|
||||
raw.append(0)
|
||||
section_offset = len(raw)
|
||||
raw.extend(b"".join(headers))
|
||||
ident = b"\x7fELF\x01\x02\x01" + bytes(9)
|
||||
raw[:52] = ident + struct.pack(">HHIIIIIHHHHHH", 1, 20, 1, 0, 0,
|
||||
section_offset, 0, 52, 0, 0, 40, 6, 5)
|
||||
return bytes(raw)
|
||||
|
||||
|
||||
def make_dol(text):
|
||||
header = [0] * 64
|
||||
header[0], header[18], header[36] = 256, BASE, len(text)
|
||||
return struct.pack(">64I", *header) + bytes(text)
|
||||
|
||||
|
||||
def fixture(root, source_call="B", source_raw=False, reference_raw=False,
|
||||
source_start=0x20, alias_body=False, renamed=False, score=100, size=8):
|
||||
text = bytearray(0x80)
|
||||
for offset, value in [(0, LI_ONE), (4, BLR), (8, LI_ONE if alias_body else LI_TWO),
|
||||
(12, BLR), (0x24, BLR)]:
|
||||
put_word(text, offset, value)
|
||||
reference_text = bytearray(text)
|
||||
put_word(reference_text, 0x20, bl(0x20, 8) if reference_raw else bl(0, 0))
|
||||
source_text = bytearray(text)
|
||||
put_word(source_text, source_start + 4, BLR)
|
||||
destination = 0 if source_call == "A" else 8
|
||||
put_word(source_text, source_start,
|
||||
bl(source_start, destination) if source_raw else bl(0, 0))
|
||||
symbols = [("A", 0, 8, 1, 2), ("B", 8, 8, 1, 2), ("caller", 0x20, 8, 1, 2)]
|
||||
source_symbols = [("renamed" if renamed and n == source_call else n,
|
||||
source_start if n == "caller" else v,
|
||||
size if n == "caller" else length, sec, kind)
|
||||
for n, v, length, sec, kind in symbols]
|
||||
source_relocs = [] if source_raw else [(source_start, 1 if source_call == "A" else 2, 10, 0)]
|
||||
reference_relocs = [] if reference_raw else [(0x20, 2, 10, 0)]
|
||||
source = root / "build/43U/src/toy.o"
|
||||
reference = root / "build/43U/obj/toy.o"
|
||||
for path, data in [(source, make_elf(source_text, source_symbols, source_relocs)),
|
||||
(reference, make_elf(reference_text, symbols, reference_relocs))]:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_bytes(data)
|
||||
put_word(text, 0x20, bl(BASE + 0x20, BASE + 8))
|
||||
dol = root / "original.dol"
|
||||
dol.write_bytes(make_dol(text))
|
||||
report = {"units": [{"name": "main/toy", "functions": [
|
||||
{"name": "caller", "fuzzy_match_percent": score}],
|
||||
"metadata": {"source_path": "toy.c"}}]}
|
||||
report_path = root / "build/43U/report.json"
|
||||
report_path.write_text(json.dumps(report))
|
||||
config = root / "config/43U"
|
||||
config.mkdir(parents=True, exist_ok=True)
|
||||
(config / "symbols.txt").write_text(
|
||||
"A = .text:0x80001000; // type:function size:0x8\n"
|
||||
"B = .text:0x80001008; // type:function size:0x8\n"
|
||||
"caller = .text:0x80001020; // type:function size:0x8\n")
|
||||
(config / "splits.txt").write_text("toy.c:\n\t.text start:0x80001000 end:0x80001080\n")
|
||||
return report_path, dol, hashlib.sha1(dol.read_bytes()).hexdigest()
|
||||
|
||||
|
||||
class CallTargetTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.root = Path(self.temp.name)
|
||||
|
||||
def run_fixture(self, **options):
|
||||
report, dol, sha = fixture(self.root, **options)
|
||||
return tool.audit(self.root, report, dol, sha)
|
||||
|
||||
def test_relocated_destination_matches_original_dol(self):
|
||||
result, status = self.run_fixture()
|
||||
self.assertEqual(status, 0)
|
||||
self.assertEqual(result["counts"]["same_destination"], 1)
|
||||
self.assertIn(str(self.root / "build/43U/src/toy.o"), result["provenance"]["sha256"])
|
||||
|
||||
def test_wrong_callee_detected_despite_identical_relocated_call_bytes(self):
|
||||
result, status = self.run_fixture(source_call="A")
|
||||
self.assertEqual(status, 1)
|
||||
row = result["rows"][0]
|
||||
self.assertEqual(row["classification"], "different_destination")
|
||||
self.assertEqual(row["source_addresses"], [BASE])
|
||||
self.assertEqual(row["dol_destination"], BASE + 8)
|
||||
self.assertIsNone(row["alias_hint"])
|
||||
|
||||
def test_raw_local_calls_with_shifted_source_function(self):
|
||||
result, status = self.run_fixture(source_raw=True, reference_raw=True, source_start=0x40)
|
||||
self.assertEqual(status, 0)
|
||||
self.assertEqual(result["counts"]["source_raw_relative"], 1)
|
||||
self.assertEqual(result["counts"]["reference_raw_relative"], 1)
|
||||
|
||||
def rewrite_source(self, symbols, relocations):
|
||||
text = bytearray(0x80)
|
||||
for offset, word in [(0, LI_ONE), (4, BLR), (8, LI_TWO),
|
||||
(12, BLR), (0x20, bl(0, 0)), (0x24, BLR)]:
|
||||
put_word(text, offset, word)
|
||||
(self.root / "build/43U/src/toy.o").write_bytes(make_elf(text, symbols, relocations))
|
||||
|
||||
def test_section_symbol_relocation_resolves_actual_function_start(self):
|
||||
report, dol, sha = fixture(self.root)
|
||||
self.rewrite_source([("A", 0, 8, 1, 2), ("B", 8, 8, 1, 2),
|
||||
("caller", 0x20, 8, 1, 2), ("", 0, 0, 1, 3)],
|
||||
[(0x20, 4, 10, 8)])
|
||||
result, status = tool.audit(self.root, report, dol, sha)
|
||||
self.assertEqual(status, 0)
|
||||
self.assertEqual(result["counts"]["same_destination"], 1)
|
||||
|
||||
def test_same_address_alias_is_visible_without_claiming_a_bug(self):
|
||||
report, dol, sha = fixture(self.root)
|
||||
self.rewrite_source([("A", 0, 8, 1, 2), ("alias", 0, 0, 0, 2),
|
||||
("caller", 0x20, 8, 1, 2)], [(0x20, 2, 10, 0)])
|
||||
with (self.root / "config/43U/symbols.txt").open("a") as stream:
|
||||
stream.write("alias = .text:0x80001008; // type:function size:0x8\n")
|
||||
result, status = tool.audit(self.root, report, dol, sha)
|
||||
self.assertEqual(status, 1)
|
||||
self.assertEqual(result["rows"][0]["classification"], "same_address_alias")
|
||||
|
||||
def test_unknown_external_symbol_is_unresolved_not_guessed(self):
|
||||
report, dol, sha = fixture(self.root)
|
||||
self.rewrite_source([("A", 0, 8, 1, 2), ("unknown", 0, 0, 0, 2),
|
||||
("caller", 0x20, 8, 1, 2)], [(0x20, 2, 10, 0)])
|
||||
result, status = tool.audit(self.root, report, dol, sha)
|
||||
self.assertEqual(status, 1)
|
||||
self.assertEqual(result["rows"][0]["classification"], "unresolved_destination")
|
||||
self.assertEqual(result["rows"][0]["source_addresses"], [])
|
||||
|
||||
def test_wrong_relocation_kind_or_symbol_is_an_input_error(self):
|
||||
for index, kind in [(2, 1), (99, 10)]:
|
||||
with self.subTest(index=index, kind=kind):
|
||||
report, dol, sha = fixture(self.root)
|
||||
self.rewrite_source([("A", 0, 8, 1, 2), ("B", 8, 8, 1, 2),
|
||||
("caller", 0x20, 8, 1, 2)], [(0x20, index, kind, 0)])
|
||||
result, status = tool.audit(self.root, report, dol, sha)
|
||||
self.assertEqual(status, 2)
|
||||
self.assertTrue(result["errors"])
|
||||
|
||||
def test_wrong_raw_local_callee_detected(self):
|
||||
result, status = self.run_fixture(source_call="A", source_raw=True)
|
||||
self.assertEqual(status, 1)
|
||||
self.assertEqual(result["rows"][0]["source"]["names"], ["A"])
|
||||
|
||||
def test_decoder_signed_absolute_and_non_call_cases(self):
|
||||
self.assertEqual(tool.branch_target(bl(0x20, 0), 0x20), 0)
|
||||
self.assertEqual(tool.branch_target(bl(0x20, 0x40), 0x20), 0x40)
|
||||
self.assertEqual(tool.branch_target(bl(0, 8, True), 0x100), 8)
|
||||
self.assertEqual(tool.branch_target(bl(0, -4, True), 0), 0xFFFFFFFC)
|
||||
self.assertIsNone(tool.branch_target(0x48000010, 0)) # tail branch
|
||||
self.assertIsNone(tool.branch_target(0x4E800421, 0)) # indirect bctrl
|
||||
|
||||
def test_raw_interior_destination_retains_addend(self):
|
||||
report, dol, sha = fixture(self.root, source_raw=True)
|
||||
obj = tool.read_object(self.root / "build/43U/src/toy.o", tool.Inputs())
|
||||
data = bytearray(obj.data[1])
|
||||
put_word(data, 0x20, bl(0x20, 12))
|
||||
obj.data[1] = bytes(data)
|
||||
destination = obj.call(obj.function("caller"), 0)
|
||||
self.assertEqual((destination["names"], destination["addend"]), (["B"], 4))
|
||||
|
||||
def test_renamed_relocation_free_wrapper_is_retained_for_review(self):
|
||||
result, status = self.run_fixture(renamed=True)
|
||||
self.assertEqual(status, 1)
|
||||
self.assertEqual(result["rows"][0]["classification"], "unresolved_destination")
|
||||
self.assertIn("identical_relocation_free_local_body", result["rows"][0]["alias_hint"])
|
||||
|
||||
def test_identical_original_bodies_do_not_silently_suppress_names(self):
|
||||
result, status = self.run_fixture(source_call="A", alias_body=True)
|
||||
self.assertEqual(status, 1)
|
||||
self.assertEqual(result["rows"][0]["classification"], "different_destination")
|
||||
self.assertIn("identical_original_leaf_bodies", result["rows"][0]["alias_hint"])
|
||||
|
||||
def test_nonexact_function_is_not_audited(self):
|
||||
result, status = self.run_fixture(score=99.9)
|
||||
self.assertEqual(status, 0)
|
||||
self.assertEqual(result["counts"].get("calls", 0), 0)
|
||||
|
||||
def test_unequal_extents_are_explicitly_skipped(self):
|
||||
result, status = self.run_fixture(size=4)
|
||||
self.assertEqual(status, 1)
|
||||
self.assertIn("extents differ", result["skipped_functions"][0]["reason"])
|
||||
|
||||
def test_missing_requested_function_or_unit_is_explicit(self):
|
||||
report, dol, sha = fixture(self.root)
|
||||
result, status = tool.audit(self.root, report, dol, sha, ["toy"], "missing")
|
||||
self.assertEqual(status, 1)
|
||||
self.assertTrue(result["skipped_functions"])
|
||||
result, status = tool.audit(self.root, report, dol, sha, ["missing"])
|
||||
self.assertEqual(status, 2)
|
||||
self.assertTrue(result["errors"])
|
||||
|
||||
def test_reference_dol_disagreement_is_an_input_error(self):
|
||||
report, dol, sha = fixture(self.root)
|
||||
raw = bytearray(dol.read_bytes())
|
||||
put_word(raw, 256 + 0x20, bl(BASE + 0x20, BASE))
|
||||
dol.write_bytes(raw)
|
||||
result, status = tool.audit(self.root, report, dol, hashlib.sha1(raw).hexdigest())
|
||||
self.assertEqual(status, 2)
|
||||
self.assertIn("ELF/DOL disagreement", result["errors"][0]["reason"])
|
||||
|
||||
def test_call_only_in_original_dol_is_not_missed(self):
|
||||
report, dol, sha = fixture(self.root)
|
||||
raw = bytearray(dol.read_bytes())
|
||||
put_word(raw, 256 + 0x24, bl(BASE + 0x24, BASE))
|
||||
dol.write_bytes(raw)
|
||||
result, status = tool.audit(self.root, report, dol, hashlib.sha1(raw).hexdigest())
|
||||
self.assertEqual(status, 2)
|
||||
self.assertIn("ELF/DOL disagreement at +0x4", result["errors"][0]["reason"])
|
||||
|
||||
def test_removed_source_call_is_a_shape_candidate(self):
|
||||
report, dol, sha = fixture(self.root)
|
||||
source = self.root / "build/43U/src/toy.o"
|
||||
text = bytearray(0x80)
|
||||
put_word(text, 0x20, LI_ONE)
|
||||
put_word(text, 0x24, BLR)
|
||||
source.write_bytes(make_elf(text, [("caller", 0x20, 8, 1, 2)], []))
|
||||
result, status = tool.audit(self.root, report, dol, sha)
|
||||
self.assertEqual(status, 1)
|
||||
self.assertEqual(result["rows"][0]["classification"], "call_shape_difference")
|
||||
|
||||
def test_dol_integrity_and_bounds_validation(self):
|
||||
raw = make_dol(bytes(8))
|
||||
with self.assertRaisesRegex(ValueError, "SHA1"):
|
||||
tool.Dol(raw, "0" * 40)
|
||||
with self.assertRaisesRegex(ValueError, "truncated"):
|
||||
tool.Dol(b"", hashlib.sha1(b"").hexdigest())
|
||||
image = tool.Dol(raw, hashlib.sha1(raw).hexdigest())
|
||||
with self.assertRaisesRegex(ValueError, "not in one DOL section"):
|
||||
image.read(BASE + 8, 4)
|
||||
|
||||
def test_no_body_alias_hint_for_position_dependent_code(self):
|
||||
self.assertFalse(tool.position_independent_leaf(bl(0, 0x100).to_bytes(4, "big")))
|
||||
self.assertFalse(tool.position_independent_leaf(bl(0, 0).to_bytes(4, "big")))
|
||||
|
||||
def test_changed_input_and_head_are_reported(self):
|
||||
path = self.root / "input"
|
||||
path.write_bytes(b"first")
|
||||
inputs = tool.Inputs()
|
||||
inputs.read(path)
|
||||
path.write_bytes(b"second")
|
||||
self.assertEqual(inputs.changed(), [str(path)])
|
||||
with self.assertRaisesRegex(ValueError, "changed while reading"):
|
||||
inputs.read(path)
|
||||
report, dol, sha = fixture(self.root)
|
||||
with patch.object(tool, "git_head", side_effect=["before", "after"]):
|
||||
result, status = tool.audit(self.root, report, dol, sha)
|
||||
self.assertEqual(status, 2)
|
||||
self.assertEqual(result["errors"][-1]["reason"], "inputs changed during audit")
|
||||
|
||||
def test_cli_json_exit_status_and_input_overwrite_protection(self):
|
||||
report, dol, sha = fixture(self.root)
|
||||
output = self.root / "results.json"
|
||||
args = ["--root", str(self.root), "--dol", str(dol), "--dol-sha1", sha]
|
||||
self.assertEqual(tool.main(args + ["--output", str(output)]), 0)
|
||||
self.assertTrue(json.loads(output.read_text())["advisory"])
|
||||
with contextlib.redirect_stderr(io.StringIO()):
|
||||
self.assertEqual(tool.main(args + ["--output", str(report)]), 2)
|
||||
self.assertIn("units", json.loads(report.read_text()))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user