Skip to content

Commit 96922cf

Browse files
Merge pull request #141 from RetroReversing/feat/integer-and-call-layout-repairs
2 parents c58a3a6 + 7bf992c commit 96922cf

13 files changed

Lines changed: 771 additions & 6 deletions

‎README.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,14 @@ local declaration order before random permutation. It keeps original line layout
3636
These hypotheses run through the same compiler and exact judge as agents, in ROM and supported 32-bit imported
3737
projects. They use local CPU, with no model API calls. Rejected candidates do not replace the best saved C.
3838

39+
The same pass can widen a byte/short local when arithmetic or a later conversion suggests truncation, test
40+
candidate-owned callee parameter/return widths, and split or merge bounded integer lifetimes. It also tries
41+
calls inside argument-selecting branches, copies terminal goto leaves, and reorders switch cases with no
42+
fallthrough. Missing no-op zero cases can restore a compiler's original dispatch layout. Shared headers and
43+
project-wide prototypes stay unchanged. These are hypotheses: an improved similarity score is still a draft,
44+
and only a byte-exact result counts as matched. The families also reach iterative/deep search and
45+
`try_local_repairs`; loops, incoming jumps, shadowing and ambiguous declarations limit what is proposed.
46+
3947
For close ROM misses it also tests cached compiler flag alternatives, two probes per pass. Later passes skip
4048
already tested choices and can reach optimization levels as well as debug flags. Only exact results qualify for a
4149
narrow compiler-range change. When automatic compiler changes are disabled, the result becomes a suggestion for

‎ROADMAP.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,10 @@
11
# Roadmap
22

3+
Bounded integer/call repairs now include evidence-led local widening, candidate-owned callee prototype widths,
4+
integer lifetime splits/merges, call distribution into branches, terminal goto leaves and switch layout variants.
5+
They share the ROM/imported-project judge and existing budgets; a better similarity score remains a draft.
6+
General CFG liveness, fallthrough switch rewriting and shared-header prototype inference remain future work.
7+
38
What's planned for decompdecomp, roughly in order, and why. Every item keeps the rule that matters most: a match is
49
byte-identical after linking (ROM mode) or objdiff 100% (fork mode). Nothing here accepts code that only does the
510
same thing.
Lines changed: 203 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,203 @@
1+
"""Alternative call/leaf and switch layouts, with bounded work and explicit control-flow guards."""
2+
import re
3+
4+
from .integer_repair_support import IDENT, INTEGER, NUMBER, edit, integer, view
5+
from .pattern_repairs import close
6+
7+
8+
def call_statement(text, start):
9+
"""A direct call statement (optionally assigned), with nested argument parentheses kept intact."""
10+
hit = re.match(rf'\s*(?:{IDENT}\s*=\s*)?({IDENT})\s*\(', text[start:])
11+
if not hit or hit[1] in {'if', 'for', 'while', 'switch', 'sizeof', 'return'}:
12+
return None
13+
end = close(text, start+hit.end()-1)
14+
tail = re.match(r'\s*;', text[end:]) if end else None
15+
return (start+hit.start(), end+tail.end()) if tail else None
16+
17+
18+
def flat_assignments(text):
19+
return bool(re.fullmatch(rf'(?:\s*{IDENT}\s*=(?!=)[^;{{}}]*;)+\s*', text)) and not re.search(
20+
r'\b(?:return|goto|break|continue|case|default|for|while|do)\b', text)
21+
22+
23+
def terminal_case(text):
24+
terminal = re.search(r'\b(?:break\s*;|return\b[^;{}]*;|goto\s+'+IDENT+r'\s*;)\s*$', text)
25+
if not terminal or text[:terminal.start()].count('{') != text[:terminal.start()].count('}'):
26+
return False
27+
prefix = text[:terminal.start()].rstrip()
28+
return not prefix or prefix[-1] in ';}'
29+
30+
31+
def controls(part):
32+
"""Braced controls and their ends; an unrecognised scope is a boundary for leaf traversal."""
33+
result = {}
34+
for pos in range(part.opening+1, part.end-1):
35+
if part.clean[pos] != '{':
36+
continue
37+
prefix = part.clean[:pos].rstrip()
38+
kind = 'else' if re.search(r'\belse$', prefix) else ''
39+
if prefix.endswith(')'):
40+
depth = 1
41+
start = len(prefix)-2
42+
while start >= 0 and depth:
43+
depth += (prefix[start] == ')')-(prefix[start] == '(')
44+
start -= 1
45+
hit = re.search(r'\b(if|switch|for|while)\s*$', prefix[:start+1])
46+
kind = hit[1] if hit else ''
47+
if re.search(r'\bdo$', prefix):
48+
kind = 'do'
49+
end = close(part.clean, pos, '{', '}')
50+
if end:
51+
result[end-1] = (pos, kind)
52+
return result
53+
54+
55+
def leaf_transfer(part, start, scopes):
56+
"""Follow closing if arms past their else arms, stopping at one explicit terminal transfer."""
57+
pos = start
58+
while True:
59+
while pos < part.end and part.clean[pos].isspace():
60+
pos += 1
61+
if part.clean[pos:pos+1] != '}':
62+
break
63+
if pos not in scopes or scopes[pos][1] not in {'if', 'else'}:
64+
return None
65+
pos += 1
66+
other = re.match(r'\s*else\s*{', part.clean[pos:])
67+
if other:
68+
end = close(part.clean, pos+other.end()-1, '{', '}')
69+
if not end:
70+
return None
71+
pos = end
72+
elif re.match(r'\s*else\b', part.clean[pos:]):
73+
return None # shortcut: leave unbraced/else-if continuation to a full CFG parser
74+
terminal = re.match(rf'\s*(?:goto\s+({IDENT})|return(?:\s+[^;{{}}]+)?|break)\s*;', part.clean[pos:])
75+
return (pos, pos+terminal.end(), terminal[1]) if terminal else None
76+
77+
78+
def variants(source, name):
79+
part = view(source, name)
80+
if not part:
81+
return []
82+
result = []
83+
clean = part.clean
84+
# Distribute a call following an if/else which only selects scalar arguments. Retain assignments:
85+
# they may still be read afterwards, and evaluating an RHS twice would change its meaning.
86+
for hit in re.finditer(r'\bif\s*\(', part.body):
87+
cond_end = close(clean, part.opening+1+hit.end()-1)
88+
arm = re.match(r'\s*{', clean[cond_end:]) if cond_end else None
89+
if not arm:
90+
continue
91+
left = cond_end+arm.end()-1
92+
left_end = close(clean, left, '{', '}')
93+
other = re.match(r'\s*else\s*{', clean[left_end:]) if left_end else None
94+
if not other:
95+
continue
96+
right = left_end+other.end()-1
97+
right_end = close(clean, right, '{', '}')
98+
if not right_end or not all(flat_assignments(clean[a+1:b-1]) for a, b in [(left, left_end), (right, right_end)]):
99+
continue
100+
call = call_statement(clean, right_end)
101+
if not call:
102+
continue
103+
statement = source[call[0]:call[1]]
104+
result.append(('branch_call', edit(source, [(left_end-1, left_end-1, statement),
105+
(right_end-1, right_end-1, statement), (*call, '')])))
106+
if sum(family == 'branch_call' for family, _ in result) >= 4:
107+
break
108+
# Inline a leaf with its terminal transfer, never a label whose following code would fall through.
109+
# Duplicate local names could change lookup at the new site, so leave those functions alone.
110+
decls = list(re.finditer(rf'(?:^|[;{{}}])\s*(?!goto\b|return\b|break\b|continue\b)'
111+
rf'(?:{IDENT}\s+)+\**\s*({IDENT})\s*(?=[;=,\[])', part.body))
112+
declarations = [decl[1] for decl in decls]
113+
nested = any(part.body[:decl.start(1)].count('{') != part.body[:decl.start(1)].count('}') for decl in decls)
114+
if not nested and len(declarations) == len(set(declarations)):
115+
scopes = controls(part)
116+
# Unbraced loops can change a copied break's destination too; don't reason through them.
117+
unbraced = False
118+
for loop in re.finditer(r'\b(?:for|while)\s*\(', part.body):
119+
loop_start = part.opening+1+loop.start()
120+
if loop[0].startswith('while') and any(kind == 'do' and ending < loop_start
121+
and not clean[ending+1:loop_start].strip() for ending, (_, kind) in scopes.items()):
122+
continue # the latch of a braced do/while is not a second, unbraced loop
123+
loop_end = close(clean, part.opening+1+loop.end()-1)
124+
if loop_end and not re.match(r'\s*{', clean[loop_end:]):
125+
unbraced = True
126+
127+
def break_scope(pos):
128+
enclosing = [(opening, ending) for ending, (opening, kind) in scopes.items()
129+
if opening < pos < ending and kind in {'switch', 'for', 'while', 'do'}]
130+
return max(enclosing, default=None)
131+
132+
for label in re.finditer(rf'\b({IDENT})\s*:(?!:)', part.body):
133+
start = part.opening+1+label.end()
134+
call = call_statement(clean, start)
135+
if not call:
136+
continue
137+
terminal = leaf_transfer(part, call[1], scopes)
138+
if not terminal:
139+
continue
140+
transfer_start, end, destination = terminal
141+
leaf = source[start:call[1]]+source[transfer_start:end]
142+
if destination == label[1]:
143+
continue
144+
for jump in re.finditer(rf'\bgoto\s+{re.escape(label[1])}\s*;', part.body):
145+
pos = part.opening+1+jump.start()
146+
if start <= pos < end:
147+
continue
148+
if re.search(r'\bbreak\b', clean[transfer_start:end]) and (
149+
unbraced or break_scope(pos) is None or break_scope(pos) != break_scope(start)):
150+
continue
151+
result.append(('goto_leaf', edit(source, [(pos, part.opening+1+jump.end(), '{'+leaf+'}')])) )
152+
if sum(family == 'goto_leaf' for family, _ in result) >= 4:
153+
break
154+
if sum(family == 'goto_leaf' for family, _ in result) >= 4:
155+
break
156+
for switch in re.finditer(r'\bswitch\s*\(', part.body):
157+
condition = close(clean, part.opening+1+switch.end()-1)
158+
arm = re.match(r'\s*{', clean[condition:]) if condition else None
159+
if not arm:
160+
continue
161+
opening = condition+arm.end()-1
162+
end = close(clean, opening, '{', '}')
163+
if not end:
164+
continue
165+
inside = clean[opening+1:end-1]
166+
labels = [hit for hit in re.finditer(rf'\b(?:case\s+(-?{NUMBER})|default)\s*:', inside)
167+
if inside[:hit.start()].count('{') == inside[:hit.start()].count('}')]
168+
if not labels or inside[:labels[0].start()].strip():
169+
continue
170+
# Reject non-literal/range labels, not merely the subset our regex understands.
171+
all_labels = [hit for hit in re.finditer(r'\b(?:case\b[^:]*|default)\s*:', inside)
172+
if inside[:hit.start()].count('{') == inside[:hit.start()].count('}')]
173+
if len(labels) != len(all_labels):
174+
continue
175+
if not any(label[1] is None or integer(label[1].lstrip('-')) == 0 for label in labels):
176+
result.append(('switch_layout', edit(source, [(opening+1, opening+1, '\ncase 0: break;\n')])))
177+
starts = []
178+
pending = labels[0].start()
179+
valid = True
180+
for index, label in enumerate(labels):
181+
stop = labels[index+1].start() if index+1 < len(labels) else len(inside)
182+
code = inside[label.end():stop]
183+
if not code.strip():
184+
continue # grouped labels travel with their one body
185+
# Only block-scoped declarations can move with a case. No fallthrough or cross-case labels.
186+
if re.search(rf'\b(?:{INTEGER}|void|float|double)\s*\**\s*{IDENT}', code):
187+
valid = False
188+
break
189+
if not terminal_case(code):
190+
valid = False
191+
break
192+
starts.append((pending, stop))
193+
pending = stop
194+
if not valid or not 2 <= len(starts) <= 6:
195+
continue
196+
chunks = [source[opening+1+a:opening+1+b] for a, b in starts]
197+
orders = [list(range(rotation, len(chunks)))+list(range(rotation)) for rotation in range(1, len(chunks))]
198+
orders += [list(reversed(range(len(chunks))))]
199+
for order in orders[:6]:
200+
result.append(('switch_layout', edit(source, [(opening+1+starts[0][0], opening+1+starts[-1][1],
201+
''.join(chunks[index] for index in order))])))
202+
break # bounded per function, not factorial permutations of every nested switch
203+
return list(dict.fromkeys(result))
Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
"""Repair candidate-owned scalar callee prototypes without editing a project's shared declarations.
2+
3+
Wrong short formals can add conversions at every call. Masked results can also reveal a wrongly inferred
4+
return width. Unknown, conditional, variadic and indirect declarations remain with the real compiler.
5+
"""
6+
import re
7+
8+
from ..forks.csource import find
9+
from .integer_repair_support import INTEGER, calls, edit, split_args, view
10+
from .repair_preflight import _PROTO
11+
12+
NARROW = {'s8', 'u8', 's16', 'u16', 'char', 'short', 'signed char', 'unsigned char', 'signed short', 'unsigned short'}
13+
14+
15+
def variants(source, name):
16+
part = view(source, name)
17+
if not part or re.search(r'^\s*#', part.clean, re.M):
18+
return []
19+
prototypes = list(_PROTO.finditer(part.clean[:part.opening]))
20+
result = []
21+
for proto in prototypes:
22+
callee = proto[1]
23+
if callee == name or sum(p[1] == callee for p in prototypes) != 1 or find(source, callee):
24+
continue
25+
sites = list(calls(part, callee))
26+
if not sites or not proto[2].strip() or '...' in proto[2]:
27+
continue
28+
# A local object/parameter can shadow this function name.
29+
without_proto = part.clean[:proto.start()]+part.clean[proto.end():]
30+
if re.search(rf'\b{re.escape(callee)}\b(?!\s*\()', without_proto):
31+
continue
32+
params = split_args(proto[2])
33+
for start, end in params:
34+
param = proto[2][start:end]
35+
match = re.fullmatch(rf'\s*({INTEGER})(?:\s+[A-Za-z_]\w*)?\s*', param)
36+
if not match or ' '.join(match[1].split()) not in NARROW:
37+
continue
38+
offset = proto.start(2)+start
39+
result.append(('callee_parameter_width', edit(source, [(offset+match.start(1), offset+match.end(1), 'int')])))
40+
if len(result) >= 6:
41+
break
42+
# Infer return-width choices only from a captured direct result subsequently masked in this caller.
43+
returns = re.match(rf'\s*(?:extern\s+)?({INTEGER})\s+{re.escape(callee)}\s*\(', proto[0])
44+
if returns and ' '.join(returns[1].split()) in {'s32', 'u32', 'int', 'signed int', 'unsigned int'}:
45+
for start, _ in sites:
46+
# Call whitespace can vary; locate the assignment ending at the callee token instead.
47+
assignment = re.search(rf'\b([A-Za-z_]\w*)\s*=\s*{re.escape(callee)}\s*\($', part.clean[part.opening+1:start])
48+
if not assignment:
49+
continue
50+
variable = re.escape(assignment[1])
51+
masks = re.findall(rf'\b{variable}\s*&\s*0x(FF|FFFF)\b', part.body, re.I)
52+
for mask in dict.fromkeys(masks):
53+
for typ in (('unsigned char', 'signed char') if len(mask) == 2 else ('unsigned short', 'short')):
54+
offset = proto.start()+returns.start(1)
55+
result.append(('callee_return_width', edit(source, [(offset, proto.start()+returns.end(1), typ)])))
56+
if masks:
57+
break
58+
if len(result) >= 10:
59+
break
60+
return list(dict.fromkeys(result))[:10]

‎src/decompnet/app/draft_repairs.py‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,15 @@
99
from .. import m2cfix, seeds
1010
from ..forks.csource import _blank, find
1111
from . import (
12+
branch_layout_repairs,
1213
branch_locals,
14+
callee_prototype_repairs,
1315
captured_repairs,
1416
expression_repairs,
17+
integer_lifetime_repairs,
1518
lifetime_repairs,
1619
load_pair_repairs,
20+
local_widening_repairs,
1721
local_width_repairs,
1822
mmio_repairs,
1923
near_repairs,
@@ -74,6 +78,13 @@ def variants(source: str, name: str, arch: str, asm: str = '', flags=(), labels=
7478
for text in local_width_repairs.variants(source, name):
7579
proposals.append(text)
7680
labels[text] = 'local_width'
81+
for text in local_widening_repairs.variants(source, name):
82+
proposals.append(text)
83+
labels[text] = 'local_widening'
84+
for module in (callee_prototype_repairs, integer_lifetime_repairs, branch_layout_repairs):
85+
for family, text in module.variants(source, name):
86+
proposals.append(text)
87+
labels[text] = family
7788
for text in pointer_cast_repairs.variants(source, name, arch):
7889
proposals.append(text)
7990
labels[text] = 'pointer_cast_inline'

‎src/decompnet/app/guide/workflow.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,3 +30,10 @@ Goal: C that the game's own compiler turns into exactly the bytes in the ROM. No
3030
## Priorities
3131
Get the instruction count and control flow right first, then operands, then register allocation.
3232
Plausible original code beats contortions: games were written by people, quickly.
33+
34+
Before sending many hand-written variants, use `try_local_repairs` on the current revision. Its bounded
35+
families include `local_widening`, `callee_parameter_width`, `callee_return_width`, `split_integer_lifetime`,
36+
`merge_integer_lifetimes`, `branch_call`, `goto_leaf` and `switch_layout`. They can test lost coordinate bits,
37+
unwanted call conversions, integer register lifetimes and alternative dispatch layouts without model-written
38+
source variants. Candidate-owned prototypes change only in that candidate; confirm a discovered signature
39+
before suggesting a shared header edit. A lower score is useful evidence, not a proof of correctness.

0 commit comments

Comments
 (0)