Files
wii-ipl/AGENTS.md
2026-09-22 22:39:00 -05:00

372 lines
17 KiB
Markdown

# AGENTS.md
Guidance for AI agents working in this repository.
Keep code comments to cases where they are necessary.
## CRITICAL RULE: never touch upstream
**Do not open a pull request against `koopthekoopa/wii-ipl`. Do not push to it.
Do not open issues on it. Do not comment on it. Never. Not once.**
The upstream maintainers do not want AI-authored commits in their history. This
fork exists solely so that AI-assisted matching work has somewhere to live
without disturbing them. Breaking this rule is worse than producing no matches
at all.
Concretely:
- `origin` is this fork, `coah80/wii-ipl`. Only the orchestrator pushes
validated leaf branches here.
- `upstream` is `koopthekoopa/wii-ipl`. Its push URL is intentionally set to
the sentinel `DISABLED_never_push_to_upstream` so an accidental
`git push upstream` fails instead of uploading.
- Never run `gh pr create` with an upstream base, and never pass
`--repo koopthekoopa/wii-ipl` to any `gh` command.
- If a task appears to require contacting upstream, stop and ask the human.
If you are reading this because you are about to open a PR against upstream:
don't. PRs for validated leaf branches belong only to this fork.
## What this repository is
A matching decompilation of the Wii Menu (system menu) 4.3U. "Matching" means
the compiled output must be byte-identical to the original binary, not merely
functionally equivalent. Correct behaviour that compiles to different assembly
is a failure.
## Build and verify
```
python3 configure.py --version 43U
ninja
```
Ninja lives at `/home/cole/projects/tests/.venv/bin/ninja` in the author's
environment.
The acceptance gate for any change is the DOL hash:
```
build/43U/main.dol -> 26116613f624061ba99c8d1a299aaa6efa85670d
```
Objects listed as `NonMatching` in `configure.py` are **not linked** into the
DOL. Editing a `NonMatching` translation unit cannot change the DOL hash, which
makes it safe to iterate on a partial match without breaking the build. Verify
progress against the object file instead of the DOL:
```
build/43U/src/src/<path>.o # what you just built
build/43U/obj/src/<path>.o # the original object, ground truth
```
## Tooling for measuring progress
`objdiff` scores each function:
```
build/tools/objdiff-cli report generate -p . -o report.json -f json
```
A function counts as matched only at `fuzzy_match_percent == 100.0`. Anything
below that is not a match, regardless of how good it looks.
Helper tools live in `tools/decomp-assist/`:
- `ctxdiff.py <unit> <symbol>` — compact instruction-level diff of one function
between your object and the original. This is the main iteration loop.
- `disasm_fn.py <object> <symbol>` — disassembly with relocation-resolved call
names, so you can see which real functions are being called.
- `pool_diff.py` — diffs the string pool in `.data` between your object and the
original, string by string. See below; this is the highest-signal tool.
- `export_pyghidra.py` — regenerates `ghidra_decomp.txt`, decompiled C for the
functions still missing.
The final completion assertion is machine-checkable:
```
python3 tools/check_decomp_complete.py build/43U/report.json \
--dol build/43U/main.dol
```
It must pass before claiming full completion. GitHub Actions runs the same
assertion when manually dispatched with `require_complete=true`.
Normal push and pull-request builds publish progress without failing on the
current incomplete state. Use the manual `require_complete=true` dispatch for
the terminal all-100% gate.
`tools/test_workflow_guards.py` exercises the completion checker, and
`tools/worktree_check.py` validates the main, leaf, and cleanup states before
integration.
## The single most useful finding: the string pool
MWCC emits a translation unit's string literals into `.data` in the order the
compiler encounters them, and the compiler encounters them in **source-token
order**, not runtime order. Every later string's offset therefore depends on
every earlier one.
This matters enormously. If a function is missing from a translation unit, or
references its strings in a different order than the original, then every
string after it lands at the wrong offset, and *every function that reads those
strings fails to match* — even functions you have not touched.
Observable evidence: `TMDFile::Open` sat at 99.92 percent purely because a
string several functions earlier was 8 bytes off. Adding the correct
`checkContentsNum` block moved it to 99.96 without touching `Open`.
**Always run `pool_diff.py` and fix the first divergence before tuning register
allocation.** It points at exactly which function block is wrong or missing.
### Which branch you write changes the pool
Because emission follows source-token order, writing
```cpp
if (cond) { A(); } else { B(); }
```
versus
```cpp
if (!cond) { B(); } else { A(); }
```
emits the strings referenced by `A` and `B` in the opposite order. This is a
free lever: if a function's codegen looks right but its strings are swapped,
invert the condition rather than restructuring the logic. It fixed both the
pool and the instruction stream for `GetValidTicketIndex` in one change.
### Missing functions must all be present
A string block cannot be partially correct. A function that is entirely absent
leaves a hole, and everything downstream is shifted. Implement functions in
**object-order** (the order their addresses appear in `config/43U/symbols.txt`
and `splits.txt`), not in whatever order looks convenient.
## Workflow that produced matches
1. Find the next missing function in object order.
2. Get its decompiled C from `ghidra_decomp.txt` (regenerate with
`export_pyghidra.py` if needed).
3. Write it as literally as possible: same order of operations, same branch
direction, same number of temporaries.
4. Insert it at the correct position in the source file.
5. Build only that object: `ninja build/43U/src/src/<path>.o`.
6. `pool_diff.py` first. Fix ordering divergences before anything else.
7. `ctxdiff.py` next. Work through the instruction diff.
8. Only then look at register allocation.
## Subagent and PR workflow
This repository targets only Wii Menu 4.3U. Do not configure or build 43E,
43J, or 43K while doing matching work.
Most workers should pursue exact matching or safe linking; assign the remainder
to decompilation. Keep up to 75 live workers when the runtime allows it, while
reserving capacity for fixer and validation workers. Fill completed slots with
disjoint tasks. Never duplicate a source file or function
range. Workers return changed paths, exact objdiff measurements, and validation
results. Fuzzy-only results are not accepted.
Workers must do the work through tool calls and avoid conversational messages.
Send findings to the parent only through the collaboration tool, with concise
paths, measurements, and validation evidence. Put PR discussion on the PR when
needed. Do not send user-facing status chatter; the orchestrator follows the
reporting rule below.
### Worker loop
Workers iterate until exact-name `objdiff` reports `100.0%`. They must not stop
at a fuzzy score, push, open a PR, or merge. A local commit is fine. Compiler
tie-breaks should be reported with evidence, not hidden behind artificial
assembly or uninitialized values. Exact matching is the handoff point, not
acceptance. A candidate is incomplete until every owned code and data section
and its link metrics are 100%; matching a few functions alone is not enough.
Write new matches as readable C or C++ and keep new inline assembly out of the
source unless the existing target structure makes it unavoidable.
Worker measurements are advisory. Only a fresh orchestrator verification can
accept, push, open, or merge a candidate. Workers are persistent across
validation. After reaching a candidate result,
the worker enters a paused or awaiting-parent state while the orchestrator
performs its checks. Keep the worker handle and its leaf assignment alive. If
pool, ctxdiff, source review, or any other gate fails, send the failure evidence
back to that same worker and resume it on the same leaf. Do not discard a
near-match and silently replace it with a new worker. A worker may be closed
only after its leaf is accepted, explicitly abandoned, or the orchestrator is
stopping the entire goal. A wait timeout is not worker termination; poll the
same handle or inspect its authoritative status before taking recovery action.
The orchestrator must independently rebuild the owned object and verify:
1. `pool_diff.py` reports identical pools, with no first divergence.
2. Exact-name `objdiff` reports `100.0%` for the requested symbol.
3. `ctxdiff.py` reports `diffs 0` and identical instruction counts.
4. Unit-level code, data, function, and link measures for the owned range are
100%, including every owned section.
5. The focused source diff is minimal, readable, and free of artificial
assembly, uninitialized values, or unrelated edits.
6. The full 4.3U build passes and `build/43U/main.dol` has SHA1
`26116613f624061ba99c8d1a299aaa6efa85670d`.
If any orchestrator gate fails, do not promote, push, open a PR, or merge the
candidate. Restore it or send the same disjoint task back to its worker with
the failing evidence. Stage the candidate branch for a PR only after every
gate passes, then regenerate the live progress report before accepting the
next candidate.
For each candidate, the worker must:
1. Run `pool_diff.py` before tuning code generation.
2. Build the 43U object with `/home/cole/projects/tests/.venv/bin/ninja -C . build/43U/src/src/<path>.o`.
3. Require `objdiff` to report `100.0%` for the function and `ctxdiff.py` to report `diffs 0`.
4. Iterate until exact-name objdiff reaches `100.0%`, while preserving the
pool and source-quality requirements above.
5. Return the focused exact match to the parent agent for review. The parent
reruns every gate, runs `tools/decomp_status.py` and
`tools/check_decomp_complete.py`, and runs `/home/cole/projects/tests/.venv/bin/ninja -C . build/43U/ok`;
the worker does not push, open a PR, or merge.
The main agent reviews every worker result before accepting it. After each
accepted merge, record the new report from `build/43U/report.json` and keep
the DOL hash at `26116613f624061ba99c8d1a299aaa6efa85670d`. Never push to
`upstream`; only the fork's `origin` is in scope.
### Orchestrator ownership boundary
The orchestrator does not perform decompilation or hand-tune worker source.
All source-level reverse engineering, implementation, compiler experiments,
and match iteration belong to the persistent subagents. The orchestrator may
only select disjoint leaves, start and pause workers, resume a worker with
failure evidence, run independent verification commands, review diffs, record
wave state, commit validated leaf-branch changes, push the leaf branch, open
and merge the PR, fast-forward `origin/main`, and publish progress. If a
candidate fails validation, return it to its worker; do not fix the candidate
inline in the orchestrator.
A new thread must begin by reading this file, checking the live 43U report and
remote branch, and dispatching the worker pool. The parent must not claim a
match without fresh independent verification.
### Worktrees, branches, and PR integration
Every leaf gets its own worktree and branch created from the latest
`origin/main`, for example `agent/<wave>/<leaf>`. No two workers may share a
worktree, branch, or source file. A worker may commit locally on its leaf branch
if that is needed to preserve its exact result, but it must not push, open a PR,
or merge.
Use the repository worktree checker before integration and cleanup:
```
python3 tools/worktree_check.py main --synced
python3 tools/worktree_check.py leaf <leaf-worktree> agent/<wave>/<leaf>
python3 tools/worktree_check.py cleanup <leaf-worktree> agent/<wave>/<leaf>
```
The cleanup check is read-only. It succeeds only for a clean leaf branch that
is already an ancestor of `origin/main`; the orchestrator then performs the
explicit `git worktree remove` and local branch deletion.
After the worker reaches its handoff point, the orchestrator validates the
worker's worktree from a fresh build. If every gate passes, the orchestrator
pushes that leaf branch to the fork, opens a PR against `coah80/wii-ipl:main`,
and merges it only after the same gates are recorded in the PR. Never create a
PR against `koopthekoopa/wii-ipl`.
Merge one accepted leaf at a time. After each merge, fetch the new
`origin/main`, regenerate progress, and rebase or recreate all not-yet-merged
leaf branches before their next validation. Re-run objdiff, pool, ctxdiff, the
full build, and the DOL hash after rebasing. The PR itself is not acceptance;
the post-merge main branch is the authority.
The orchestrator may resolve mechanical branch conflicts such as unchanged
context or generated bookkeeping. It must not resolve a semantic source
conflict by inventing decompilation. Send semantic conflicts back to the same
leaf worker, resume it in its worktree, and require the full gates again. Keep
the worktree and worker handle until the leaf is merged, explicitly abandoned,
or the whole goal is stopped; clean them up only after that terminal state.
After a successful merge:
1. Confirm the PR is merged into the fork's `main` and the leaf worktree has no
uncommitted changes.
2. In the integration/main worktree, run `git fetch origin` followed by
`git merge --ff-only origin/main` (or an equivalent non-destructive fast
forward). Do not use `git reset --hard` or overwrite unrelated user work.
3. Re-run the live progress report and the full 4.3U/DOL gates on the updated
main branch.
4. Rebase each still-active leaf branch onto the new `origin/main` in its own
worktree, resolve only mechanical conflicts centrally, and send semantic
conflicts back to that leaf's worker.
5. Only after the leaf is merged, clean, and no longer needed, remove its
worktree and local branch. A failed or conflicted leaf keeps its worktree
and worker alive.
### User-facing progress reports
Keep user-facing messages to a minimum and use as few tokens as possible.
Do not send routine status updates or commentary while workers are running or
PRs are open. After each matching-work PR is merged and the live progress
report has been regenerated, send only the current linked, fuzzy, and data
percentages in this format:
```
Linked: <percent> | Fuzzy: <percent> | Data: <percent>
```
Do not report stale values, unmerged PR results, or add other text to that
update. Continue all required verification and coordination silently.
## Goal completion contract
The decompilation loop does not stop after a successful wave, a convenient
near-match, or a code-only milestone. Keep dispatching and resuming workers
until the live 43U report proves all of the following:
- Code is 100.00% exact-matched, not merely fuzzy-matched.
- Code is 100.00% fully linked, with no remaining unlinked code units.
- Data is 100.00% extracted/decompiled and 100.00% fully linked.
- Every source unit and function in the 4.3U target is accounted for; no
`NonMatching`, missing, fuzzy-only, or unlinked unit remains.
- The final full build passes, the DOL hash remains correct, and the report,
object files, and linked DOL agree.
Until every condition is true and independently verified, the goal remains
active. A worker wave with zero accepted matches is a failed attempt, not a
completion condition; preserve useful workers, resume fixable leaves, and
dispatch the next disjoint batch.
## Things that do NOT work
- **Frame pragmas.** `#pragma ppc_iro_level 0` fixed exactly one function out
of roughly forty attempts and made others worse. A search across pragma
variants produced almost no movement. Do not spend time here.
- **Reverting to pristine upstream to unblock.** The unmodified upstream tree
is needed as a baseline, but it does not help you match anything.
- **Searching other forks for the missing functions.** The eleven unimplemented
`iplESMisc` functions do not exist in any of the twenty-three public forks.
## Known hard cases
Many remaining diffs are pure compiler tie-breaks — register allocation and
floating-point scheduling — with no source-level lever. If `ctxdiff.py` shows
the instruction count already matches and the surviving differences are
callee-saved register names, you are probably looking at a tie-break. Record it
and move on rather than thrashing.
## Current status
Do not copy a progress number from this file. Regenerate the live 43U report
with `ninja -C . progress build/43U/report.json`; the report and DOL hash are
the authority for current status. `iplESMisc.cpp` remains `NonMatching` and
unlinked, so work there is DOL-safe.
## Reference
`tools/decomp-assist/iplESMisc.reference.cpp` is a snapshot of `iplESMisc.cpp`
with the matching work applied, kept as a reference for the string order and
function layout that the pool depends on.