How I Leveraged LLMs to Capture Terabytes of Historical Email Attachment Compliance Data Into a User-Friendly System
Welcome back to Bits Of Progress, a space where I reflect on hands-on technical experiences, explore systems thinking, and break down the design patterns that keep our infrastructure moving forward.
Lately, I’ve been thinking about the true nature of technical capability in the modern IT landscape. For a long time, the barrier to executing enterprise-grade infrastructure projects was syntax fluency—knowing exactly how to write multi-threaded scripts, configure complex serverless deployment files, or map data schemas from memory.
But a recent project forced me to confront a new reality: if you possess strong systems thinking and clean logic, you can bridge significant technical knowledge gaps by treating LLMs as your on-demand engineering team.
We recently executed a company-wide migration from Google Workspace to Microsoft 365 at International Warehouse & Shipping LLC. The migration went smoothly, but it brought a massive compliance liability to light: our teams were routinely routing and stashing critical customer compliance imagery directly within individual employee email inboxes. These photos verify freight condition, container numbers, and seal integrity. Under our logistics compliance rules, we must retain and easily query these records for years. Leaving these massive binary attachment footprints inside active user mailboxes would eventually exhaust active exchange quotas and trap crucial audit trails in silos.
I needed to design a decoupled, automated serverless ecosystem to offload this storage weight into Azure, optimize full-text search capability, and surface the results straight to our team’s active workspace. I named this macro architecture Project CargoLens.
Because I didn't have the deep Python or data engineering expertise required to build this out from scratch, I co-engineered the entire ecosystem by securely orchestrating three distinct AI models: Perplexity AI, Gemini, and Claude. Here is the real-world workflow of how we built it.
My Tri-AI Engineering Team
Instead of using a single LLM for everything, I realized early on that different models have distinct algorithmic strengths. I structured my workflow by treating them as specialized engineers reporting to me as the lead systems architect:
Perplexity AI (The Research Specialist): Used exclusively to scout live cloud documentation, compare library version dependencies, and clarify the nuances of the Python v2 Azure Functions programming model.
Gemini (The Systems & Optimization Thinker): Used to map data pipelines, untangle multi-core processing errors, and design the logical layout for data consistency.
Claude (The Code Syntactician): Used for deep algorithmic code generation, complex multi-line regular expression construction, and robust front-end refactoring.
Phase 1: Clearing the Legacy Backfill (Perplexity + Gemini)
Before building a real-time ingestion queue, I had to deal with the historical debt: gigabytes of legacy mailbox archive (.mbox) backups sitting in our local extraction directories.
Researching the Azure Ecosystem with Perplexity
I started by asking Perplexity to outline the modern Python v2 Azure Functions directory pattern and verify the compatibility matrices for our primary requirements : azure-functions, azure-storage-blob, azure-data-tables, and azure-search-documents. Perplexity saved me hours of wading through fragmented documentation by giving me the exact library versions required to ensure our local scripts wouldn't throw environment conflicts when pushed to production.
Designing Parallel Workflows with Gemini
Processing massive .mbox archives sequentially would have bottlenecked our environment. I wanted to design a local multi-threaded pipeline to handle zip unpacking and mailbox pruning, but I lacked the syntax confidence for Python's concurrency modules.
I gave Gemini my structural design requirements, and it generated a multi-core processing architecture utilizing a ProcessPoolExecutor. When my early test runs threw multi-threaded terminal errors, I dumped the raw stack traces back into Gemini. It acted as a patient senior debugger, explaining that I needed a clean isolation layer (move_kept_messages.py) to copy validated compliance records out to a clean directory and purge processed items from the origin source to prevent overlaps.
[Legacy MBOX Archives]
│
▼
(bulkMBOXProcessor.py) [cite_start]──► Utilizes ProcessPoolExecutor (All but 1 CPU core) [cite: 134, 135]
│
▼
(move_kept_messages.py) [cite_start]─► Isolates messages matching compliance regex [cite: 135, 136]
│
▼
(ingest_mbox_shipment_images.py) [cite_start]──► Verifies 25 Search Index fields before cloud upload [cite: 29, 30]
Gemini also co-designed our automated schema integrity guard. Before shipping any data payloads across the wire, the ingestion loader automatically queries the live Azure AI Search cluster and asserts that all 25 required metadata fields exactly match our configuration types. If a type mismatch is identified, the program throws a terminal error and aborts, completely shielding our cloud index from data pollution.
Phase 2: Live Serverless Ingestion (Claude)
With the historical debt cleared, we shifted to the continuous real-time capture engine: prod-shipment-function-app. When a live shipment notification hits our logistics mailboxes, a Power Automate flow captures the incoming parameters and forwards a structured JSON webhook payload to our Azure Function endpoint.
Writing Clean Parsers with Claude
Because incoming email formatting is highly volatile, the function needed to extract strict tracking fields from unstructured text bodies. I used Claude to generate the core parsing logic.
Claude wrote an efficient html_to_text() sanitization routine that strips out hidden script blocks, inline styling sheets, line-break anomalies, and non-breaking spaces before text evaluation. I then fed Claude our anonymized text samples, and it generated highly precise multi-line regular expressions to parse multi-value arrays for Item Numbers, Rip Numbers, PO Identifiers, and Seals on the fly.
# Context: shipment_ingestion/parser.py co-generated with Claude
def parse_body_metadata(body_text, subject_shipment_id=None):
text = body_text or ""
shipment_ids = extract_field_values([r"Shipment\s*ID\s*:\s*([A-Za-z0-9\-]+)"], text)
item_numbers = extract_field_values([r"Item\s*Number\s*:\s*([A-Za-z0-9\-_\/\.]+)"], text)
# [cite_start]Generates safe string arrays for multi-value search indexing [cite: 107, 108]
Solving Idempotency with Core Math
One of our biggest system risks was duplicated cloud data caused by automated Power Automate retries during network hiccups. I explained this problem to Claude, and it introduced me to "defensive programming" through deterministic hashing.
Claude generated a formula that creates a unique, collision-resistant identifier for every single file attachment by calculating a localized SHA-256 hash combining the immutable shipment code, the network message ID, and the attachment's specific file signature:
$$\text{ID} = \text{SHA-256}(\text{ShipmentID} \parallel \text{"|"} \parallel \text{MessageID} \parallel \text{"|"} \parallel \text{AttachmentSHA256})$$
This constraint guarantees that if the exact same image is processed multiple times, it safely executes a merge upsert across Azure Blob, Table, and AI Search rather than generating duplicate entries.
Claude also helped me build a custom RETURN_PARSED_ONLY diagnostic circuit breaker. Flipping this environment configuration switch tells the application to run text extraction and JSON validation normally, but gracefully short-circuits execution before committing physical writes to active production storage—giving us a completely safe sandbox to test regex updates.
Phase 3: Secure Asset Queries & SharePoint Embedding (Perplexity + Claude)
The final layer was consumption: exposing our index data via an embedded Next.js interface housed directly inside a modern SharePoint page framework.
[SharePoint Modern Canvas]
│
[cite_start]▼ (Renders Framed Next.js Interface via iframe) [cite: 150]
[Next.js Client UI]
│
[cite_start]▼ (Secure POST request with Shared API Key) [cite: 8, 256]
[image-search-function-app]
│
[cite_start]├───► (Queries Azure AI Search Index via OData Filter) [cite: 3, 21]
│
[cite_start]└───► (Generates Ad-Hoc 30-Min Read SAS Token for Blob URL) [cite: 7, 28]
Building OData Queries with Perplexity
To query the search service efficiently, our query API backend (image-search-function-app) needs to construct precise OData filter strings based on user inputs. I used Perplexity to double-check the proper formatting constraints for OData time calculations and string-escaping rules.
The resulting code includes an input scrubbing step (_escape_odata) that cleans user parameters and dynamically handles query filtering. It successfully limits lookups to sliding date ranges (email_date ge {utc_cutoff}) and filters parameters by directional movement types or facility codes.
Enforcing Container Security with Claude
We could not make our compliance image storage container public, yet our SharePoint users needed to view photos instantly. I asked Claude how to bridge this gap securely. It helped me write a backend streaming routine using the generate_blob_sas library.
Instead of returning raw, insecure storage endpoints to the client browser, the search function acts as a secure intermediary. It reads the document metadata from the index, verifies our application-level header key (x-search-api-key) , and dynamically generates a short-lived Shared Access Signature (SAS) token appended to the image URL. This signature grants read-only access to that specific file and expires automatically after exactly 30 minutes.
Securing the Next.js Iframe Configuration
Finally, embedding a Next.js application inside a SharePoint tenant framework throws you right into a maze of cross-origin browser security policies. I used Claude to help structure our next.config.ts routing profile. Claude wrote a clean Content Security Policy (CSP) that safely allows enterprise framing while stopping clickjacking vectors cold:
[cite_start]// Context: next.config.ts co-generated with Claude [cite: 162]
const contentSecurityPolicy = [
"default-src 'self'",
"img-src 'self' data: blob: https:",
[cite_start]"frame-ancestors 'self' https://*.sharepoint.com https://*.sharepoint-df.com", // Restricting frame authority [cite: 162]
].join('; ');
My Blueprint for Secure AI Prompting
When you are using generative models to write enterprise software, you have an absolute responsibility to maintain data privacy. You cannot blindly paste production logs or configuration settings into public LLMs. My prompting workflow relies on a strict isolation model:
Anonymize the Schema: I never fed public models real warehouse identifiers, customer names, or routing information. I created mock JSON strings (e.g., using
SHP-204881andphoto1.jpg) to teach the AI our payload definitions.Abstract Environment Settings: I scrubbed our
local.settings.jsonfile completely before asking for help with configuration parsing. The AI only saw placeholder variables likeAZURE_SEARCH_ENDPOINT: "https://your-search-service...".Local Assembly Only: The LLMs generated the algorithmic logic, but the structural combination of our actual account connections and private access keys was managed strictly on my local machine.
Bits of Progress: Systems Thinking Over Syntax
Project CargoLens moved our organization away from unorganized email attachments to a secure, event-driven search catalog. More importantly, it completely changed my perspective on what it means to be a technical builder.
A lack of deep, syntax-level coding experience is no longer a terminal blocker for creating enterprise systems. If you can break a business problem down into clear logical requirements, map data flows accurately, and maintain rigid security guardrails, you can use generative AI to handle the heavy implementation lifting.
Don't use AI just to copy-paste code you don't understand. Use it as an interactive technical masterclass that explains the why behind every design pattern as you build.
Thanks for tuning into this edition of Bits Of Progress—until next time, keep building, keep learning, and keep moving forward!
