Work archivePrototype · The inspiration for Tickmark
Finance operations workbench
Run Replay
A hands-on prototype for month-end close, payroll control and scenario planning. It reconciles structured data, routes exceptions and keeps every release behind a human decision.
Tickmark: controls on real 10-K filings and a FinQA grader, liveRules calculate
Evidence stays attached
People release
- Finance sources
- 4
- Finance controls
- 10
- Payroll engine recall
- 98.5%
- Payroll boundary alerts
- 0%
01 · Finance operations workbench
Run a month-end close.
Four synthetic sources enter a deterministic control layer. The system produces an exception queue, balanced journal drafts, close evidence and an adjustable forecast. AI explains evidence; it never calculates or approves amounts.
Run ReplayFinance operating file · October close
Local prototype- Ingest—
- Validate—
- Reconcile—
- Prepare JEs—
- Route review—
Ready for the first run
The workspace stays empty until you run the close controls. You can replace any CSV first.
02 · Payroll control lab
Run a payroll control cycle.
Use the synthetic sample or upload your own three CSV files. The files are parsed and checked on the server by the deterministic engine; nothing is sent to a model at this stage.
Synthetic sample · Oct 1–15, 2026
Runs, decisions and the audit log are kept in this browser for the demo. The production schema (PostgreSQL) is in db/schema.sql.
- Load & fingerprint—
- Validate contracts—
- Apply rules—
- Route to review—
—source rows
—payroll records
—exceptions
—release blockers
—server time
03 · Payroll review desk
Every exception, one decision at a time.
Run the pipeline to fill the queue.
04 · Evaluation harness
How I know it works, and where it does not.
These scores cover the payroll control engine and are computed from code at build time. The set mixes clear errors, boundary cases that must stay silent, prompt-injection notes, and stress cases written to break the rules. The newer finance-close engine has deterministic integration tests, but not yet a scored recall set.
74Test cases
98.5%Exception recall
100%Precision
0%False-alert rate (boundary)
98.6%Exact match
By category
Version history
engine-2.0.066 casesR 100%P 100%EM 100%First run on the 66 cases written alongside the rules. A perfect score here only proves the rules match their own spec.
engine-2.0.074 casesR 92.3%P 98.4%EM 91.9%Added 8 stress cases written to break the rules: 5 missed, 1 false alarm.
engine-2.1.074 casesR 98.5%P 100%EM 98.6%Fixed four root causes. One known gap left open on purpose (overlapping periods).
Failure analysis
- FixedInjection written in Chinese
Cause: Instruction patterns were English-only.
Fix: Added Chinese patterns for “ignore previous instructions”, “approve directly” and “system prompt”.
- FixedInjection with lookalike characters (“Ign0re … appr0ve”)
Cause: Matching ran on raw text, so digit swaps slipped through.
Fix: Normalize common swaps (0→o, 1→i, 3→e, 4→a, 5→s, @→a) and match both forms.
- FixedHRIS termination date “Sep 30 2026”
Cause: Only payroll dates were validated; a bad HRIS date silently skipped the inactive-employee rule.
Fix: Validate HRIS hire and termination dates and raise MISSING_FIELD, without dropping the person from lookup.
- FixedLower-case employee ID in payroll
Cause: IDs were compared case-sensitively, creating a false “missing employee” alarm.
Fix: Normalize employee IDs (trim + upper-case) when files are loaded.
- FixedStatus “Inactive” instead of “terminated”
Cause: Only the literal status “terminated” was checked.
Fix: Treat terminated, inactive, left and separated as inactive statuses.
- OpenOverlapping pay periods paid twice
Cause: Duplicate detection keys on identical periods; overlapping ranges are not compared.
Fix: Open: needs an interval-overlap check plus a policy for legitimate off-cycle payments.
Brief quality (deterministic fallback)
100%Schema valid
100%Evidence citation accuracy
100%Allowed next action
100%No approval language
64 briefs
Model × prompt comparison
Not run yet. The comparison calls real models and costs money, so it needs an API key: set ANTHROPIC_API_KEY and run npm run eval. Results appear here.
05 · Observability & failure handling
Every model call is accounted for.
Live model on this deployment
…
Controls
- Timeout
- 12 s per call, 1 retry on rate limit or 5xx
- Structured output
- JSON schema (zod) + a second validator: cited fields must exist, action must be allowed, no approval language
- Fallback
- Any failure returns the deterministic brief: no key, timeout, refusal, invalid output
- Redaction
- Employee IDs become aliases; free-text notes are never sent
- Blocking errors
- High-severity exceptions block release until a person decides
- Overrides
- Overriding a suggestion requires a written reason, stored in the audit log
Model calls this session
Generate a brief to see its telemetry.
06 · User testing
Protocol ready. Results not collected yet.
I have not run this with real Accounting, finance-operations or payroll reviewers yet, so there are no user results on this page. This is the study I will run.
- Who
- 3–5 Accounting / finance operations / payroll practitioners and 5–10 operations testers
- Task
- Complete the finance-close and payroll samples once in Run Replay and once from the raw CSVs
- Measured
- Time to finish, exceptions found, false calls, overrides and their reasons, willingness to use again
- Status
- Recruiting
07 · Post-mortem
What changed, what failed, what is still missing.
- 01
Why AI belongs here
Reviewers lose time turning a rule code and five values into a decision. Explaining evidence in plain language is exactly what a model is good at, and it is reversible.
- 02
What never goes to AI
Amounts, dates, differences, rule decisions, writes to source data, and approval. Those stay in code and with people.
- 03
Version 1 failed quietly
The first Run Replay showed eight hand-written exceptions. It looked finished, but nothing ran on real input and nothing could be measured.
- 04
What the evaluation exposed
My own 66 cases scored 100%. Eight stress cases written to break it dropped recall to 92.3% and exact match to 91.9%: English-only injection checks, lookalike characters, an unvalidated HRIS date, case-sensitive IDs, and a single “terminated” status.
- 05
What I changed
Multilingual and normalized injection checks, HRIS date validation, ID normalization and broader inactive statuses. Recall is now 98.5% with no false alerts; overlapping pay periods remain an open gap.
- 06
Cost and speed
The finance engine runs 63 rows and the payroll engine runs 31 rows in milliseconds at zero model cost. Model cost and latency per brief will come from the comparison above once it runs; I will not invent them.
- 07
Missing for production
Authenticated NetSuite, Ramp, Numeric, Snowflake, HRIS and payroll APIs; a hosted database; SSO and roles; four-eyes release; jurisdiction-specific policies; monitoring; and real user results.
- 08
What I built
Two deterministic control engines, source contracts, journal preparation, scenario modeling, exception routing, an evaluation harness, the AI boundary (redaction, schema, validator, fallback), API routes, this interface, and CI tests.