I'm Building an Auditable Payroll Pipeline from WhatsApp Photos

This is a project in active development, not yet running live payroll. The Phase 1 architecture is built and tested on real photos. The audit layer catches real bugs (documented below). The workflow is now being piloted with the admin who runs payroll for a 30-store retail chain in South Africa.

She spends several hours each week typing handwritten timesheets from WhatsApp photos into Excel. Pay day is the 20th. Errors become disputes.

---

The friction I found

The admin who runs payroll for this chain has a hard deadline. The 20th of every month. Before that, she has to:

• Receive ~240 WhatsApp photos of paper timesheets from store managers
• Read the handwriting on each one (good days are good, bad days are not)
• Type the hours into a master Excel
• Apply the lunch rule (worked more than 7h that day = deduct 1h)
• Cross-match employees who work across multiple stores
• Produce a final payable list, per store, per week, with totals

Estimated time: several hours each week. Exact number to be measured during pilot.

She doesn’t want a new app. The store managers definitely don’t want a new app. The paper-and-WhatsApp loop has worked for years and the team is comfortable with it. The only thing that can change is what happens between the photo landing in her phone and the number landing in the Excel.

---

What I built (Phase 1)

A Custom GPT plus an Excel Engine that:

1. Takes a batch of WhatsApp photos as input
2. Extracts the handwritten timesheet table via vision LLM
3. Normalizes the raw extraction (“11 to 7” becomes 8h, “9-5” becomes 8h)
4. Applies the lunch deduction rule
5. Cross-matches every row with a unique employee ID (more on this in the accordion)
6. Writes everything into a three-tab Excel master: Audit, Weekly, Final

Phase 1 is semi-manual on purpose. The admin reviews the extraction before validating each batch. I’m not putting an LLM-only step into a financial file without a human review layer until the error rate has held flat across enough months.

If Phase 1 holds through the pilot, Phase 2 is a full API version with a stricter audit layer, moving human review from “every batch” to “audit-by-sample”.

---

Why I didn’t touch the input

The workflow intentionally preserves the existing paper + WhatsApp process. The store managers keep their paper. The admin keeps her WhatsApp. What I automate is what happens between the photo and the Excel cell.

---

The technical architecture

Stack

• Python via Code Interpreter for deterministic math, not LLM prediction
• Vision LLM (GPT-4o, Gemini as backup) for OCR on handwritten timesheets
• pandas + openpyxl for Excel manipulation
• Custom GPT as the Phase 1 human-in-the-loop interface
• Three-tab Excel master: VERIFICATION_AUDIT, Weekly, Final

Why Code Interpreter and not pure LLM

The lunch rule is “if worked more than 7h that day, deduct 1h”. A pure LLM might predict the right answer most of the time and predict it wrong on edge cases I can’t anticipate. Code Interpreter runs deterministic Python on the extracted data. The math is auditable, not probabilistic.

For an auditable payroll file, that distinction is the whole point.

Multi-step prompt architecture

1. Extract the raw table from the photo
2. Parse the hours strings into numeric durations
3. Apply the lunch deduction function
4. Cross-match against the Employee_ID reference list

Each step is auditable on its own. When something goes wrong I know which step broke.

The Employee_ID trick

Names are unreliable. Two people named “John” across two stores. Handwriting that turns “Sarah” into “Sarat”. I forced every employee record to be matched on a unique South African employee ID (NSS format) before anything else. Names are display-only, useful for the human reviewer, not for matching.

That single decision killed a class of errors I would have spent weeks debugging otherwise.

Versions

• V1: monthly batch. Worked on test data, broke at real scale.
• V4: switched to weekly cadence. Smaller batches for the LLM, smoother workload for the admin.
• V5: fixed date segmentation bug.
• V6: built the official hourly rate grid in (employees don’t all earn the same rate).

---

What broke

The two bugs worth telling about:

• The 140h bug. Vision LLM extracted the weekly timesheets correctly, but the consolidation logic cumulated the whole month into the last week’s column. One employee showed up with 140 hours in week 4. That would have triggered an overtime payment, then an employee dispute when corrected. Caught by the audit tab (any cell above 50h flagged for review). Fix: separate the weekly consolidation step from the monthly rollup.
• The Manager error. On one photo the vision LLM saw the store manager’s name in the header zone and added him as a payable employee. He doesn’t get paid hourly. Caught by sample audit before the file shipped. Fix: explicit prompt instruction to ignore the header zone, plus the Employee_ID cross-check (the manager isn’t in the NSS list).

The pattern across both: every bug got caught because there’s a verification layer between the LLM extraction and the final payable file. No LLM-only step ships straight into the payroll file.

---

Where this lives now

Phase 1 architecture is functional and tested on real photos. The admin is piloting the pipeline on her weekly batches: she submits the photos, reviews the LLM extraction, validates into the Excel master. The hours of manual entry are replaced by a review step. Exact time savings to be measured during pilot.

Phase 2 (full API, sample-audit instead of batch-by-batch review) is planned next. Volunteer side project.