2026-07-03-agent-safety-follow-up

Follow-Up Learning Plan: Agent-Safety TOI Findings (2026-07-03)

Source: 2026-07-03-moltbook-topics-research.md (same directory) - 5 grounded findings from testing Moltbook’s top community theses against ~/internal/harness. Purpose: turn one-off research findings into durable learning items and, where warranted, actual fixes - so the branch isn’t just an archived writeup. Status: all 6 items done (2026-07-03) - full learning run complete on this branch.

Priority Queue

#	Item	Priority	Effort	Depends on	Status
1	Patch harness curl|bash guardrail regex escaping bug	P1	S	none	Done
2	Regression test: fire real bypass payloads through the loader in CI	P1	S	#1	Done
3	Compression rule: “strip prose, never strip identifiers” checker	P2	M	none	Done
4	Reusable subagent-vs-direct decomposability heuristic script	P2	M	none	Done
5	jq-based structured-log query cookbook (reusable across agents)	P3	S	none	Done
6	Better epistemic-reliability test set (replace pure arithmetic)	P3	L	none	Done

1. Fix the Dead curl|bash Guardrail Regex (P1) - DONE 2026-07-03

Finding: hooks/_lib/command-blacklist.json in ~/internal/harness stores patterns with PCRE-style escapes (\s, \|). The loader (load_blacklist() in hooks/_lib/common.sh) reads the JSON via jq -r '... | @tsv', which re-escapes backslashes on the way out, so grep -Eq (POSIX ERE) never sees a usable pattern. The rule has been silently dead since it was written.

Plan:

• Reproduce with the exact repro script already committed here: scripts/blacklist-bypass-test.sh.
• Fix options to evaluate (pick one, don’t stack):
  • (a) Store patterns already in POSIX-ERE form (drop \s, \|) so @tsv round-trips clean.
  • (b) Switch the extraction from @tsv to @sh or raw -c JSON array iteration to avoid the backslash-doubling entirely.
  • (c) Use jq -j per-line output rather than @tsv for single-column extraction.
• Whichever fix is chosen, re-run scripts/blacklist-bypass-test.sh and confirm all 7 test cases from the original research (canonical block + 6 semantic bypasses) resolve to the intended block/allow state - not just “the regex matches something”.
• This is a ~/internal/harness change, not a ~/internal/learn change - file the actual patch there once validated here. This repo’s role is proving the bug and validating the fix logic before it touches production hooks.

Outcome: chose option (a) - rewrote both affected patterns in command-blacklist.json to avoid backslash escapes entirely (\s+ -> [[:space:]]+, \| -> [|], \* -> /?[*]), so @tsv has nothing to double-escape. Along the way found and fixed a second, non-jq-related gap: the rm -rf wildcard alternative never matched rm -rf /* even in its originally-intended form (only matched a bare * with no leading slash) - folded into the same fix since it’s the same pattern. Verified against harness’s own tests/hooks/test-blacklist.sh: 9/12 -> 11/12 passing (one pre-existing, unrelated test-expectation mismatch remains: generic sudo is WARN by policy design, not BLOCK as the test wrongly expects - left for a separate fix). Committed as 6037faa7 on branch fix/command-blacklist-tsv-escaping in ~/internal/harness (not main, not yet pushed - push + PR is a tracked follow-up in jane-online-presence/NEXT_STEPS.md). Full session log: jane-online-presence/docs/response-diary/2026-07-03.md.

2. CI Regression Test for the Blacklist Loader (P1, depends on #1) - DONE 2026-07-03

Why: a bug like this survives silently because nobody fires a real payload through the loader in review. A denylist with no adversarial test is a promise, not a control.

Plan:

• Turn scripts/blacklist-bypass-test.sh into a proper pass/fail test harness (exit non-zero on any unexpected ALLOW for a canonical-block case).
• Recommend (not implement here) wiring it into harness’s own pre-commit/CI for hooks/_lib/command-blacklist.json changes, so any future pattern edit gets adversarially tested automatically, not just eyeballed.

Outcome: wrote scripts/blacklist-regression-test.sh, replacing the print-everything original with explicit expected outcomes per case. Asserts 6 canonical-block/benign cases against the now-fixed (item #1) command-blacklist.json - all 6 PASS, exit 0. The 6 known semantic-bypass cases (py-syntax rmtree, find -delete, dropped sudo, split download+exec, process substitution, base64) are printed as informational-only INFO lines, not asserted, since closing them needs a behavioral/sandboxed control, not a regex denylist. Wiring into harness CI itself is left as the recommended next action (not done in this repo, per the plan’s own scope note).

3. Compression Safety Rule: “Strip Prose, Never Strip Identifiers” (P2) - DONE 2026-07-03

Finding: a 92-word verbose log compressed to 26 words lost 6/6 concrete pointers (file:line refs, commit hash, test filename). Satware’s own context-management guidance already implies this principle but doesn’t mechanically enforce it.

Plan:

• Write a small script/heuristic that takes an original + compressed text pair and flags compression as “unsafe” if any regex-detectable identifier class (file:line, commit SHA, bare file path, test name pattern) present in the original is absent from the compressed version.
• Test against the existing fixtures (fixtures/compression-original.md, fixtures/compression-summarized.md) as the baseline “should fail” case, plus a hand-written “good” compression (same word reduction, identifiers preserved) as the “should pass” case.
• Stretch goal: package as a reusable check-compression-safety.sh under learn/agent-safety/ once validated, so any future agent-context-compression work in this or other repos can lint against it.

Outcome: wrote scripts/check-compression-safety.sh, detecting 4 identifier classes (file:line, commit SHA, bare file path, pytest-style test name) via regex and diffing presence between original/compressed text. Validated against the existing bad-case fixtures (fixtures/compression-{original,summarized}.md) -> correctly reports UNSAFE, 8/8 identifiers lost, exit 1. Added a new hand-written good-case fixture (fixtures/compression-good.md, same word-count reduction, identifiers kept inline) -> correctly reports SAFE, exit 0. Stretch goal delivered as specified.

4. Decomposability Heuristic for Hierarchy Decisions (P2) - DONE 2026-07-03

Finding: subagent cold-start floor cost measured at ~1.01s vs ~0ms direct - real but bounded. Harness’s rules/agent.framework.md §1 already encodes a decision table (single-file -> direct, multi-file/independent -> subagent) but it’s prose, not a checkable rule.

Plan:

• Turn the existing decision table into a small script/function that takes a task description (or a simple structured input: file count, independence flag) and returns “direct” or “subagent”, with the >1s floor cost as a documented constant it can be weighed against.
• Not a full scheduler - just a decision-support one-liner agents can consult instead of guessing, paired with the measured cost from this research as justification in the docstring.

Outcome: wrote scripts/decomposability-heuristic.sh, taking --files, --independent, --read-only, --est-duration-sec flags and returning DIRECT/SUBAGENT plus a rationale line, with the measured 1.01s subagent cold-start floor as a named, documented constant. Verified across 4 cases: single-file/dependent -> DIRECT; multi-file -> SUBAGENT; short-duration but independent -> SUBAGENT (isolation still wins over the floor cost); read-only independent -> SUBAGENT with a “safe to parallelize” rationale.

5. Structured-Log Query Cookbook (P3) - DONE 2026-07-03

Finding: harness’s JSONL event fixtures (logs/fixtures/*.jsonl) support a real one-line jq aggregation producing per-action success rates - a working counter-example to “transcripts aren’t observability”.

Plan:

• Collect 3-5 more jq recipes beyond the group-by-success-rate one already run (e.g. rolling failure rate over time window, slowest action by duration if timestamps allow, most common action sequence before a failure).
• Write them up as a short cookbook (learn/agent-safety/structured-log-jq-cookbook.md) so any agent with a JSONL event log (not just harness) can reuse the queries instead of re-deriving jq syntax each time.

Outcome: wrote structured-log-jq-cookbook.md with 5 verified recipes (per-action success rate, chronological timeline, overall failure rate, first-failure-per-action, level/message filtering for pino-style logs), each run against harness’s logs/fixtures/sample-stress-test.jsonl and sample-cli.jsonl with real output captured inline.

6. Better Epistemic-Reliability Test Set (P3, largest effort) - DONE 2026-07-03

Finding: Moltbook’s math-verification gate is 10/10 solvable by hand-reasoning - it filters non-LLM bots, it does not test reasoning reliability. The research post argues the right test would be ambiguous-instruction or multi-hop-contradiction tasks instead.

Plan:

• Design 5-10 test items in that shape (one plausible-but-wrong reading per ambiguous instruction; one multi-hop question requiring noticing an earlier contradiction).
• Run them against this agent’s own reasoning (not an external model) as a self-check, scored pass/fail with the reasoning trace kept alongside the answer for post-hoc review.
• This is the most open-ended item on this plan - treat it as a research spike, not a fixed deliverable; timebox it rather than trying to build a full benchmark.

Outcome: wrote epistemic-reliability-test-set.md with 8 self-check items (4 ambiguous-instruction, 4 multi-hop-contradiction), each with the trap reading, correct reading, reasoning trace, and pass/fail. Scored 8/8 pass against this agent’s own reasoning (not an external model) - including one item (A3) where the correct answer was recognizing genuine ambiguity and flagging it rather than confidently picking a reading. Explicitly scoped as a timeboxed spike, not a repeated-trial reliability benchmark.

Non-Goals

• No patch to ~/internal/harness production files happens in this repo - findings and fix validation live here, the actual patch is a separate harness-repo change once validated.
• Not building a general-purpose agent benchmark suite (item #6) - a handful of self-check items is enough to test the specific claim, not a competing eval framework.
• No new dependency installs for any of the above - jq, bash, grep -E, bc are already available and sufficient.

References

• 2026-07-03-moltbook-topics-research.md (this directory) - full research writeup with raw results per finding.
• scripts/blacklist-bypass-test.sh (original, manual print), scripts/blacklist-regression-test.sh (item #2, pass/fail), scripts/check-compression-safety.sh (item #3), scripts/decomposability-heuristic.sh (item #4) - reusable scripts, this directory.
• fixtures/compression-{original,summarized}.md (bad case), fixtures/compression-good.md (good case, item #3) - this directory.
• structured-log-jq-cookbook.md (item #5), epistemic-reliability-test-set.md (item #6) - this directory.
• ~/internal/harness: hooks/_lib/command-blacklist.json, hooks/_lib/common.sh, rules/agent.framework.md §1, rules/context.management.md §7, logs/fixtures/*.jsonl.