Pad/doc/ime.md

# The IME contract and its failure modes

What the editor relies on when talking to Android's IME (Gboard), what
stock gioui does structurally, and the failure modes observed on device —
with their log signatures and the invariants that keep the system safe.
This is the map for the next time text lands in the wrong place.

## The three models

An IME session involves **three independent copies of the text** that must
be kept consistent:

1. **The file** (logic goroutine, the truth).
2. **The pushed snippet** — the 32 KB window around the caret that we push
   to the IME (`key.SnippetCmd`). All IME commit positions are in *this*
   coordinate space (absolute file runes, `Range.Start` = window start).
3. **Gboard's local model** — Gboard's private copy of the snippet plus its
   own caret/selection/composition state, which it updates from our pushes
   *and* from its own bookkeeping of the commits it sends.

Every bug in this area is a loss of consistency between model 3 and model
1. The commit that arrives names a range in model 3; applying it to model 1
is only correct if the two still agree.

## Invariants the code must maintain

- **Commits map against the whole buffer, never a stale window.**
  `HandleIMECommit` converts the commit's rune range to bytes with a
  whole-buffer scan (`runeToByteWhole`), so it stays exact while the
  visible window moves (a fling in flight). A renderer-side IME model was
  tried and abandoned: it drifts whenever the render window moves
  (commit `f54ca2f`).
- **The snippet window is hysteresis-gated and decoupled from the render
  window** (`State.computeIMESnippetWindow`): 32 KB, re-anchored only when
  the caret comes within 4 KB of an edge. Re-anchoring reaches Gboard as a
  `restartInput`, which starts a fresh input session that re-derives
  auto-capitalization — a per-scroll or per-tap re-push is what caused the
  random mid-word caps (commit `dd9493b`).
- **The post-commit caret is `Range.Start + len(text)`**, not
  `Range.End + len(text)`. Identical for insertions; for a replacement
  (autocorrect over a selection) the old formula lands `(End−Start)` runes
  past the inserted text and desynchronizes the IME at the exact moment its
  model is being updated (commit `51344b9`). The logic side
  (`applyIMECommitBytes`) and the renderer's immediate push
  (`ApplyIMECommitToModel`) must compute the *same* caret.
- **A legitimate commit is always local**: at the caret, inside the live
  selection, or a correction a few runes left of the caret. The guard
  (`HandleIMECommit`, `maxCommitDistance = 1024`) snaps anything else to
  the caret and arms the resync. A 5-rune autocorrect 19,000 runes from
  the caret is never legitimate; applying it verbatim clobbers distant
  text (commit `a83cc1a`).
- **Key-event edits desync the IME exactly like a dropped commit — and the
  IME's next commit is anchored at the pre-edit caret.** Gboard sends
  Enter as a KEY EVENT, not a commit (the logic's `handleKey(NameReturn)`
  inserts the `\n`); the IME's model never sees that edit unless a
  re-syncing snippet push reaches it first, and the push can lose the race
  — the next commit can arrive in the very same input batch as the key
  event, so no frame has drawn. While one such edit is pending
  (`EditorState.imeStaleEditByte`), the guard accepts a small (≤ 2 rune)
  TEXT commit that ends at or before the edit position (`IME STALE-OK`):
  everything before the edit is byte-identical in both models, so applying
  it verbatim is exact (the usual shape is the IME appending to, or
  correcting, the word it last committed). The pending edit is consumed and
  a resync is armed. Never relaxed for: empty-text commits (a stale range
  deletion is how documents get clobbered), commits ending past the edit
  (the range crosses the shift), or anything without a pending edit
  (commit `a83cc1a`'s guard stays intact otherwise). The pending edit is
  set by `noteNonIMEEdit` (Enter, hardware backspace/delete, cut, paste),
  consumed by the next IME commit or key edit, cleared by a tap.
- **A desynchronized IME is healed only by a restart.** See the structural
  fact below: selection pushes after a commit are deduplicated away, so
  the *only* mechanism that makes Gboard re-read the truth is a snippet
  change (`restartInput`). `IMEForceResync` arms on the **first**
  anomaly, not a streak: the next frame ships the snippet trimmed by one
  rune (one restart), the following frame ships the full window (one more),
  then quiet. One-shot, so normal typing never pays for it
  (commits `cb8ebc0`, `a83cc1a`).
- **Pre-commit frames must not flush the IME — and the hold is armed with
  the drained frame's own seq.** `ApplyIMECommitToModel` updates the pushed
  model on the drain pass and arms `imeHold` with the **EditSeq of the
  frame being drained** (it is the pre-commit frame — built before the
  logic processed the commit — and its snippet would clobber the just-
  updated model). `FlushIME` skips frames whose EditSeq is at most the
  armed one and resumes on the next higher; the logic's `HandleIMECommit`
  always markDirties, so the post-commit frame exists and always advances
  the seq. The hold is bounded (8 frames) so a stalled logic can never
  stick the renderer. Two details that are load-bearing: the frame's
  `ui.TextField.EditSeq` is copied from the state in `EditorLayout` —
  without it every frame reads 0, `EditSeq <= holdSeq` matches *every*
  frame, the hold releases only via the 8-frame timeout, and resync pushes
  (and all IME sync) are silently swallowed (this is why the resync armed
  by the 2026-09-19 drift-snap never reached Gboard, letting it keep
  committing against a stale model); and the seq must be the drained
  frame's, not "the last drawn frame's" — the drained frame is pre-commit
  even when its seq already differs from earlier ones (a key event landed
  in between), and arming with the older value releases the hold on the
  very frame whose snippet is stale.

## What stock gioui does structurally (worked around, not patched)

All of the following is stock v0.10.2 behavior. The fixes live entirely in
pad; the fork at `~/gioui` carries **diagnostic logging only** (the
`PADIME` lines) and can be dropped for stock gioui at any time without
behavioral change.

1. **Post-commit selection pushes are deduplicated away.** The commit
   callback (`callbacks.EditorReplace`) advances the window's stored
   `imeState.Selection` *before* the frame's state comparison, so a
   `SelectionCmd` pushed in the same frame as a commit is "no change" and
   `imm.updateSelection` is never called. Consequence: after a commit, the
   IME's caret can only be corrected by a snippet restart. This is why the
   resync exists and why "just re-push the selection" is not a fix.
2. **`restartInput` shadows `updateSelection`.** In
   `window.EditorStateChanged`, a snippet change takes an early-return
   branch: the selection update in the same state change is dropped.
   Gboard re-fetches the selection on restart, so this is harmless — but
   it means a restart frame is the *only* frame where selection and text
   are guaranteed to be re-read together.
3. **Selection commands are focus-gated.** `keyQueue.setSelection`
   silently drops a `SelectionCmd` whose tag is not the current key-focus
   (`req.Tag != state.focus`). A selection push can vanish with no log and
   no feedback — the IME never acks selections. If taps stop moving
   Gboard's caret, check focus first.
4. **There is no ack anywhere.** Snippet pushes, selection pushes, and
   commits are fire-and-forget. The system can only *detect* desync
   indirectly (a commit landing where no text expects it), which is what
   the drift guard does.

## Gboard behaviors observed

- Gboard sends Enter as a **key event** (InputConnection.sendKeyEvent),
  not a text commit: the app inserts the `\n` via `handleKey(NameReturn)`
  and the IME's model never learns about it until a re-syncing snippet
  push (see the key-event-edit invariant above). The emulator's Gboard
  does the same; its word-appending is milder than the phone's AICore
  Gboard, so the desync shows as a caret-off-by-one there and as
  duplicated text on the phone.
- Gboard keeps a word selection on its own (a tap near a word selects it)
  and commits autocorrect as a **replacement over its local selection** —
  the commit range is then as wide as the word, not a point insertion.
- While composing, Gboard re-sends the whole word on every keystroke
  (`"t"`, `"th"`, `"thi"`, … each replacing the previous), and on
  backspace it re-sends the shrinking word down to `text=""`. Both are
  normal; the drift guard must not treat them as anomalies (they are at
  the caret).
- When its local model desynchronizes, Gboard can enter an endless
  empty-fix-up loop (`text=""`, one commit per ~150 ms) trying to
  reconcile text that is not in its model. The file is never damaged (the
  guard snaps each one to the caret); the resync breaks the loop.
- The phone's AICore Gboard is stricter/more aggressive than the emulator's
  Gboard. A flow that is clean on the emulator can still desync on device.

## Log signatures → diagnosis

Permanent, low-volume lines (see "Diagnostics"):

| Signature in logcat | Meaning |
|---|---|
| `IME TAP … cursor=N winStart=M` | A tap moved the logic cursor (bytes). Compare with the next `IME COMMIT` ranges (runes) — they should be in the same neighborhood (rune ≈ byte − multibyte count, **compute it, don't assume a constant**). |
| `IME COMMIT range=[a,b) text=… -> [x,y)` | The logic applied the commit at bytes [x,y). The pre-`->` range is Gboard's coordinate space; the post-`->` range is where it actually landed. |
| `IME DRIFT-SNAP … snap to caret + resync` | Anomaly detected: a commit far from the caret/selection (or a stale small commit) was applied at the caret instead. One = a tap/scroll desync just happened; a run = Gboard was looping. | | `IME STALE-OK … staleEdit=N -> apply as-is + resync` | A key-event edit (usually Enter) is pending and the IME's commit was anchored at the pre-edit caret; it was applied at its reported range (byte-identical there) and a resync armed. Expected once per key-edit → keystroke sequence; a run without intervening commits = something is replaying. |
| `IME PUSH snippet=[a,b) … (restart)` | We re-anchored the IME window → Gboard restarts its input session. Should be rare: app open, file switch, caret crossing a window margin. Frequent restarts = the hysteresis gate is being defeated (check render-window coupling). |
| `IME PUSH sel=[s,e)` | We pushed a caret/selection to Gboard. If a tap does not produce one (or it repeats forever), the selection pipeline is broken upstream (focus gate, dedup, float-noise re-emit). |
| `PADIME EditorStateChanged sel A -> B` / `PADIME updateSelection sel=[…]` / `PADIME restartInput` | The gio→Gboard boundary: what actually crossed into Android. If `IME PUSH` shows a value the `PADIME` lines never show, the drop is inside gioui (focus gate / dedup); if `PADIME` shows it and Gboard still misbehaves, the IME ignored it. |

**Capture the log at incident time.** Logcat rotates fast; the incident
that started this document was only reconstructable because the capture
happened minutes later.

## Test layers and what each can catch

| Layer | What it exercises | Catches |
|---|---|---|
| e2e harness (`internal/test/e2e`) | Logic + renderer headless; commits injected directly (`HandleIMECommit`) | Commit mapping math, window invariants, guard/resync logic, edit/scroll/selection state. **Never** the IME contract: Gboard is not in the loop. |
| Emulator stress loops (`/tmp/loop4.sh`, `/tmp/stress.sh`) | Real app + adb touches, real Gboard | Fling/tap/typing interplay, restart storms, caps regressions. Coarse: no boundary visibility, scenario shape matters (warm-session fling-tap-type was the original shape; it missed the cold-restore → far-scroll → tap → fast-typing shape entirely). |
| Full-path scenario (`/tmp/s8.sh`) | Cold open with restored cursor → far scroll → tap into a known word → fast Gboard typing → backspaces; asserts tap position vs commit ranges vs **exact file diff** | The desync class: text landing anywhere but the tap. The byte→rune conversion must be computed from the file, never assumed. |

Rule: **a bug in the IME boundary is invisible to every layer except the
last one.** When IME behavior is touched, run S8.

## Operational pitfalls hit in these sessions

- The launcher activity is `pad.pad/org.gioui.GioActivity`, not
  `.MainActivity`. `am start` failures are silent if stderr is redirected —
  check the output once, trust it after.
- `adb install`/build steps fail with "no adb device" when the emulator is
  mid-reconnect; the APK may be built but not installed. Verify
  `dumpsys package … lastUpdateTime` after installs.
- The "All files access" settings toggle can be a red herring: `appops
  set <pkg> MANAGE_EXTERNAL_STORAGE allow` is the reliable grant, and
  `appops get` output has been observed to lag; `dumpsys appops` is
  authoritative.
- `difflib.SequenceMatcher` on ~1 MB strings is minutes slow; use a linear
  first-diff scan for test assertions on large files.
- Float-equality re-emit is a recurring trap (LastLineY, caret px): any
  "re-emit when changed" comparison on a float sum needs an epsilon gate,
  or it spins a loop that re-pushes IME state every frame and starves real
  work.
- When a tap's pixel→text mapping is in question, the `IME TAP` line gives
  both the local point and the resulting cursor — log both, or the mapping
  bug is undiagnosable from the outside.

## Diagnostics currently enabled (temporary)

- Pad-side, permanent: `IME TAP`, `IME COMMIT`, `IME DRIFT-SNAP`,
  `IME PUSH snippet/sel/commit-range` (bounded: only on change).
- Fork-side, **temporary** (in `~/gioui`, gated by the `replace` in
  `go.mod`): `PADIME` boundary logs in `GioView.updateSelection`,
  `GioView.restartInput`, and `window.EditorStateChanged`.

### Cleanup steps (when informal on-device testing is settled)

The fork is stock v0.10.2 plus the `PADIME` lines and nothing else
(verify first: `git -C ~/gioui diff --stat` must show only
`app/GioView.java` and `app/os_android.go`); behavior is identical on
stock gioui.

```sh
# 1. Restore the fork to clean v0.10.2
git -C ~/gioui checkout -- .
git -C ~/gioui diff --stat            # must be empty

# 2. Drop the replace and re-tidy pad against stock gioui
cd ~/pad
go mod edit -dropreplace gioui.org
go mod tidy

# 3. Verify everything green on stock gioui
go build ./...
go test -race ./...
./scripts/check.sh

# 4. Rebuild both APKs and smoke-test
./scripts/build_emu.sh                # install to emulator
bash /tmp/s8.sh                       # full-path scenario must still PASS
./scripts/build_phone.sh              # install to phone, type + tap once

# 5. Commit and push the go.mod/go.sum change
git add -A && git commit -m "build: drop the temporary gioui diagnostic fork" && git push origin main

# 6. Remove this cleanup section (and the "temporary" fork-side bullet
#    above) from ime.md; commit + push.
```

If step 1 shows more than the two diagnostic files, do NOT `checkout --
.` blindly: review the diff and keep anything that is a real (non-log)
change, porting it upstream or documenting it in this file first.