Pad/doc/ime.md

The IME contract and its failure modes

What the editor relies on when talking to Android's IME (Gboard), what stock gioui does structurally, and the failure modes observed on device — with their log signatures and the invariants that keep the system safe. This is the map for the next time text lands in the wrong place.

The three models

An IME session involves three independent copies of the text that must be kept consistent:

1. The file (logic goroutine, the truth).
2. The pushed snippet — the 32 KB window around the caret that we push to the IME (key.SnippetCmd). All IME commit positions are in this coordinate space (absolute file runes, Range.Start = window start).
3. Gboard's local model — Gboard's private copy of the snippet plus its own caret/selection/composition state, which it updates from our pushes and from its own bookkeeping of the commits it sends.

Every bug in this area is a loss of consistency between model 3 and model

1. The commit that arrives names a range in model 3; applying it to model 1 is only correct if the two still agree.

Invariants the code must maintain

• Commits map against the whole buffer, never a stale window. HandleIMECommit converts the commit's rune range to bytes with a whole-buffer scan (runeToByteWhole), so it stays exact while the visible window moves (a fling in flight). A renderer-side IME model was tried and abandoned: it drifts whenever the render window moves (commit f54ca2f).
• The snippet window is hysteresis-gated and decoupled from the render window (State.computeIMESnippetWindow): 32 KB, re-anchored only when the caret comes within 4 KB of an edge. Re-anchoring reaches Gboard as a restartInput, which starts a fresh input session that re-derives auto-capitalization — a per-scroll or per-tap re-push is what caused the random mid-word caps (commit dd9493b).
• The post-commit caret is Range.Start + len(text), not Range.End + len(text). Identical for insertions; for a replacement (autocorrect over a selection) the old formula lands (End−Start) runes past the inserted text and desynchronizes the IME at the exact moment its model is being updated (commit 51344b9). The logic side (applyIMECommitBytes) and the renderer's immediate push (ApplyIMECommitToModel) must compute the same caret.
• A legitimate commit is always local: at the caret, inside the live selection, or a correction a few runes left of the caret. The guard (HandleIMECommit, maxCommitDistance = 1024) snaps anything else to the caret and arms the resync. A 5-rune autocorrect 19,000 runes from the caret is never legitimate; applying it verbatim clobbers distant text (commit a83cc1a).
• Key-event edits desync the IME exactly like a dropped commit — and the IME's next commit is anchored at the pre-edit caret. Gboard sends Enter as a KEY EVENT, not a commit (the logic's handleKey(NameReturn) inserts the \n); the IME's model never sees that edit unless a re-syncing snippet push reaches it first, and the push can lose the race — the next commit can arrive in the very same input batch as the key event, so no frame has drawn. While one such edit is pending (EditorState.imeStaleEditByte), the guard accepts a small (≤ 2 rune) TEXT commit that ends at or before the edit position (IME STALE-OK): everything before the edit is byte-identical in both models, so applying it verbatim is exact (the usual shape is the IME appending to, or correcting, the word it last committed). The pending edit is consumed and a resync is armed. Never relaxed for: empty-text commits (a stale range deletion is how documents get clobbered), commits ending past the edit (the range crosses the shift), or anything without a pending edit (commit a83cc1a's guard stays intact otherwise). The pending edit is set by noteNonIMEEdit (Enter, hardware backspace/delete, cut, paste), consumed by the next IME commit or key edit, cleared by a tap.
• A desynchronized IME is healed only by a restart. See the structural fact below: selection pushes after a commit are deduplicated away, so the only mechanism that makes Gboard re-read the truth is a snippet change (restartInput). IMEForceResync arms on the first anomaly, not a streak: the next frame ships the snippet trimmed by one rune (one restart), the following frame ships the full window (one more), then quiet. One-shot, so normal typing never pays for it (commits cb8ebc0, a83cc1a).
• Pre-commit frames must not flush the IME — and the hold is armed with the drained frame's own seq. ApplyIMECommitToModel updates the pushed model on the drain pass and arms imeHold with the EditSeq of the frame being drained (it is the pre-commit frame — built before the logic processed the commit — and its snippet would clobber the just- updated model). FlushIME skips frames whose EditSeq is at most the armed one and resumes on the next higher; the logic's HandleIMECommit always markDirties, so the post-commit frame exists and always advances the seq. The hold is bounded (8 frames) so a stalled logic can never stick the renderer. Two details that are load-bearing: the frame's ui.TextField.EditSeq is copied from the state in EditorLayout — without it every frame reads 0, EditSeq <= holdSeq matches every frame, the hold releases only via the 8-frame timeout, and resync pushes (and all IME sync) are silently swallowed (this is why the resync armed by the 2026-09-19 drift-snap never reached Gboard, letting it keep committing against a stale model); and the seq must be the drained frame's, not "the last drawn frame's" — the drained frame is pre-commit even when its seq already differs from earlier ones (a key event landed in between), and arming with the older value releases the hold on the very frame whose snippet is stale.

What stock gioui does structurally (worked around, not patched)

All of the following is stock v0.10.2 behavior. The fixes live entirely in pad; the fork at ~/gioui carries diagnostic logging only (the PADIME lines) and can be dropped for stock gioui at any time without behavioral change.

1. Post-commit selection pushes are deduplicated away. The commit callback (callbacks.EditorReplace) advances the window's stored imeState.Selection before the frame's state comparison, so a SelectionCmd pushed in the same frame as a commit is "no change" and imm.updateSelection is never called. Consequence: after a commit, the IME's caret can only be corrected by a snippet restart. This is why the resync exists and why "just re-push the selection" is not a fix.
2. restartInput shadows updateSelection. In window.EditorStateChanged, a snippet change takes an early-return branch: the selection update in the same state change is dropped. Gboard re-fetches the selection on restart, so this is harmless — but it means a restart frame is the only frame where selection and text are guaranteed to be re-read together.
3. Selection commands are focus-gated. keyQueue.setSelection silently drops a SelectionCmd whose tag is not the current key-focus (req.Tag != state.focus). A selection push can vanish with no log and no feedback — the IME never acks selections. If taps stop moving Gboard's caret, check focus first.
4. There is no ack anywhere. Snippet pushes, selection pushes, and commits are fire-and-forget. The system can only detect desync indirectly (a commit landing where no text expects it), which is what the drift guard does.

Gboard behaviors observed

• Gboard sends Enter as a key event (InputConnection.sendKeyEvent), not a text commit: the app inserts the \n via handleKey(NameReturn) and the IME's model never learns about it until a re-syncing snippet push (see the key-event-edit invariant above). The emulator's Gboard does the same; its word-appending is milder than the phone's AICore Gboard, so the desync shows as a caret-off-by-one there and as duplicated text on the phone.
• Gboard keeps a word selection on its own (a tap near a word selects it) and commits autocorrect as a replacement over its local selection — the commit range is then as wide as the word, not a point insertion.
• While composing, Gboard re-sends the whole word on every keystroke ("t", "th", "thi", … each replacing the previous), and on backspace it re-sends the shrinking word down to text="". Both are normal; the drift guard must not treat them as anomalies (they are at the caret).
• When its local model desynchronizes, Gboard can enter an endless empty-fix-up loop (text="", one commit per ~150 ms) trying to reconcile text that is not in its model. The file is never damaged (the guard snaps each one to the caret); the resync breaks the loop.
• The phone's AICore Gboard is stricter/more aggressive than the emulator's Gboard. A flow that is clean on the emulator can still desync on device.

Log signatures → diagnosis

Permanent, low-volume lines (see "Diagnostics"):

Signature in logcat	Meaning
IME TAP … cursor=N winStart=M	A tap moved the logic cursor (bytes). Compare with the next IME COMMIT ranges (runes) — they should be in the same neighborhood (rune ≈ byte − multibyte count, compute it, don't assume a constant).
IME COMMIT range=[a,b) text=… -> [x,y)	The logic applied the commit at bytes [x,y). The pre--> range is Gboard's coordinate space; the post--> range is where it actually landed.
IME DRIFT-SNAP … snap to caret + resync	Anomaly detected: a commit far from the caret/selection (or a stale small commit) was applied at the caret instead. One = a tap/scroll desync just happened; a run = Gboard was looping.
IME PUSH snippet=[a,b) … (restart)	We re-anchored the IME window → Gboard restarts its input session. Should be rare: app open, file switch, caret crossing a window margin. Frequent restarts = the hysteresis gate is being defeated (check render-window coupling).
IME PUSH sel=[s,e)	We pushed a caret/selection to Gboard. If a tap does not produce one (or it repeats forever), the selection pipeline is broken upstream (focus gate, dedup, float-noise re-emit).
PADIME EditorStateChanged sel A -> B / PADIME updateSelection sel=[…] / PADIME restartInput	The gio→Gboard boundary: what actually crossed into Android. If IME PUSH shows a value the PADIME lines never show, the drop is inside gioui (focus gate / dedup); if PADIME shows it and Gboard still misbehaves, the IME ignored it.

Capture the log at incident time. Logcat rotates fast; the incident that started this document was only reconstructable because the capture happened minutes later.

Test layers and what each can catch

Layer	What it exercises	Catches
e2e harness (internal/test/e2e)	Logic + renderer headless; commits injected directly (HandleIMECommit)	Commit mapping math, window invariants, guard/resync logic, edit/scroll/selection state. Never the IME contract: Gboard is not in the loop.
Emulator stress loops (/tmp/loop4.sh, /tmp/stress.sh)	Real app + adb touches, real Gboard	Fling/tap/typing interplay, restart storms, caps regressions. Coarse: no boundary visibility, scenario shape matters (warm-session fling-tap-type was the original shape; it missed the cold-restore → far-scroll → tap → fast-typing shape entirely).
Full-path scenario (/tmp/s8.sh)	Cold open with restored cursor → far scroll → tap into a known word → fast Gboard typing → backspaces; asserts tap position vs commit ranges vs exact file diff	The desync class: text landing anywhere but the tap. The byte→rune conversion must be computed from the file, never assumed.

Rule: a bug in the IME boundary is invisible to every layer except the last one. When IME behavior is touched, run S8.

Operational pitfalls hit in these sessions

• The launcher activity is pad.pad/org.gioui.GioActivity, not .MainActivity. am start failures are silent if stderr is redirected — check the output once, trust it after.
• adb install/build steps fail with "no adb device" when the emulator is mid-reconnect; the APK may be built but not installed. Verify dumpsys package … lastUpdateTime after installs.
• The "All files access" settings toggle can be a red herring: appops set <pkg> MANAGE_EXTERNAL_STORAGE allow is the reliable grant, and appops get output has been observed to lag; dumpsys appops is authoritative.
• difflib.SequenceMatcher on ~1 MB strings is minutes slow; use a linear first-diff scan for test assertions on large files.
• Float-equality re-emit is a recurring trap (LastLineY, caret px): any "re-emit when changed" comparison on a float sum needs an epsilon gate, or it spins a loop that re-pushes IME state every frame and starves real work.
• When a tap's pixel→text mapping is in question, the IME TAP line gives both the local point and the resulting cursor — log both, or the mapping bug is undiagnosable from the outside.

Diagnostics currently enabled (temporary)

• Pad-side, permanent: IME TAP, IME COMMIT, IME DRIFT-SNAP, IME PUSH snippet/sel/commit-range (bounded: only on change).
• Fork-side, temporary (in ~/gioui, gated by the replace in go.mod): PADIME boundary logs in GioView.updateSelection, GioView.restartInput, and window.EditorStateChanged.

Cleanup steps (when informal on-device testing is settled)

The fork is stock v0.10.2 plus the PADIME lines and nothing else (verify first: git -C ~/gioui diff --stat must show only app/GioView.java and app/os_android.go); behavior is identical on stock gioui.

# 1. Restore the fork to clean v0.10.2
git -C ~/gioui checkout -- .
git -C ~/gioui diff --stat            # must be empty

# 2. Drop the replace and re-tidy pad against stock gioui
cd ~/pad
go mod edit -dropreplace gioui.org
go mod tidy

# 3. Verify everything green on stock gioui
go build ./...
go test -race ./...
./scripts/check.sh

# 4. Rebuild both APKs and smoke-test
./scripts/build_emu.sh                # install to emulator
bash /tmp/s8.sh                       # full-path scenario must still PASS
./scripts/build_phone.sh              # install to phone, type + tap once

# 5. Commit and push the go.mod/go.sum change
git add -A && git commit -m "build: drop the temporary gioui diagnostic fork" && git push origin main

# 6. Remove this cleanup section (and the "temporary" fork-side bullet
#    above) from ime.md; commit + push.

If step 1 shows more than the two diagnostic files, do NOT checkout -- . blindly: review the diff and keep anything that is a real (non-log) change, porting it upstream or documenting it in this file first.