Kimi K3 on PAC1 and ECOM1: results of 204 tasks and failure analysis

Moonshot AI published open weights for Kimi K3 on July 27 along with a technical report. To independently verify performance, the model was run on two BitGN task sets: PAC1 (documents, invoices, inbox, batch file changes) and ECOM1 (catalog, inventory, carts, payments, returns, logistics). PAC1 scored 61 out of 104 (binary), ECOM1 44.75 out of 100 (partial credit). The 204 open traces allow inspection of every command and state change. On PAC1, simple search and arithmetic tasks achieved 90-92% success, but three key failure modes emerged: violating exact schema (writing similar but wrong field names), non-atomic batch operations (partial file changes before full set validation), and non-blocking untrusted input checks (dangerous content detected but action still executed). On ECOM1, precise SKU search and standard checkout worked well, but failure modes included loss of state between rows in multi-line reports, excessive anomaly flagging, missing identity checks before reading private data, and state changes due to untrusted input. Optimization tasks (dispatch scheduling) delivered feasible but suboptimal plans. Comparison with other public BitGN runs shows Kimi K3 competitive on simple tasks but behind on complex multi-step constraints.