Wyatt_K bca47a27bc fix(agent): skill_search tokenizer keeps single CJK runes ...

cloudAgentSkillSearchTokens dropped any token shorter than 2 runes.
That filter was copied from English stop-word logic (a / I), but a single
Han rune such as the ones meaning hook, gag or twist is already a complete
semantic unit in Chinese. When the whole keyword consisted of such runes,
tokens came back empty and cloudAgentSearchSkills degraded to listing the
full index of enabled skills - the user thinks a search happened but gets
an unsorted dump.

Fix: keep a token when it is exactly one Han rune (unicode.Han); single
ASCII/digit runes and empty strings are still dropped as before.

Test: TestCloudAgentSkillSearchTokensKeepsSingleCJKRune fails on the old
logic (got [反转 钩子], want [反转 钩子 梗 钩]) and passes after the fix.
recall_lessons has an isomorphic copy of the same tokenizer, fixed
separately in #593.

2026-09-22 16:30:07 +08:00

61 KiB

Raw Blame History

View Raw