AI-powered automation has been a game changer in many industries, and I recently had the opportunity to integrate AI-driven email classification and entity creation into an ERP system. The goal? To automatically process orders, invoices, and other structured data from various email formats (EML, MSG, PDFs) and correctly map them into our internal system.
It sounded simple on paper—just extract key information, match it with the database, and automate entity creation. But in reality, the process involved unexpected challenges, fine-tuning prompts, and a lot of learning.
AI-powered automation is exciting—until you try to implement it. Then it becomes a mix of “wow, this is amazing” and “why AI is not hearing me?”
The Challenge: Automating Email Data Processing
Every business handles a flood of transactional emails—orders, invoices, requests—all arriving in different formats, structures, and levels of clarity. Instead of manually reviewing and inputting this data, I wanted AI to:
Classify emails into categories like orders, invoices, requests Extract key details from text & attachments (dates, amounts, order IDs) Match extracted data with existing records in the ERP system Minimize manual intervention while maintaining high accuracy
AI Models: OpenAI vs. AWS Claude
I tested both OpenAI’s GPT-4 Turbo and AWS Claude 3.5 Sonnet for this task.
Both delivered a satisfactory level of accuracy, so no drama, sorry. But the real story was in cost, prompt optimization, and working with files.
AWS Claude: Offers multiple AI models (some optimized for cost, others for accuracy, so you can pick the one that would execute each and separate task with the best cost to value balance). This flexibility is nice, but picking the right model requires trial and error. OpenAI GPT-4 Turbo: One model (with a lot of its versions :)) fits all, making it simpler to use while still delivering strong results.
Spoiler alert: AI pricing is not human-friendly.
Key Learnings from AI ERP Integration
Writing the “Perfect” Prompt is an Art—and Size Matters
I assumed that the more precise my prompt, the better the AI’s response.
Instead, I learned:
Too many constraints hurt performance. AI worked better when I gave it more freedom rather than micromanaging every step.
Additionally, prompt size impacts cost.
Longer prompts use more tokens, which means higher costs per request. When choosing an AI model, it’s important to:
Give clear but concise instructions Avoid unnecessary repetition Optimize structure instead of adding more words
AI processes everything you send it. If you make your prompt too long, you’re not just paying for thinking—you’re paying for rambling.
AI Pricing: It’s All About Inputs & Outputs
Understanding AI pricing was another unexpected challenge. Both OpenAI and AWS Claude use tokens, but how does that translate into real money?
AI pricing is based on two things: Input Tokens – Everything you send to AI (prompt, documents, metadata). Output Tokens – Everything AI sends back (responses, structured data).
Some AI services batch requests or use cached inputs, which reduces token usage and lowers costs.
This means:
Shorter prompts & efficient output formatting = lower costs. Batch processing (if available) can significantly reduce expenses.
Honestly, I wouldn’t be surprised if, in the near future, we see entire teams dedicated to AI cost optimization, just like DevOps exists for cloud infrastructure.
Here’s a direct cost comparison of OpenAI vs. AWS Claude based on processing a single same order:
| Model | Cost Per Order |
|---|---|
| OpenAI GPT-4 Turbo | $0.0357 (3.57 cents) |
| AWS Claude 3.5 Sonnet | $0.0831 (8.31 cents) |
OpenAI was more than 2x cheaper. AWS offers batch processing, which might reduce costs for high-volume use cases. AWS also provides different AI models, making it more flexible but harder to optimize without testing multiple setups.
Both OpenAI and AWS offer input and output caching, which significantly reduces costs.
Working with Files: PDFs Are More Challenging Than Expected
One major challenge we faced was extracting structured data from PDFs and other complex formats. Neither AWS Claude nor OpenAI provided reliable extraction for our use case. You can find huge threads in developers’ community trying to resolve this challenge.
We had to build our own data extractor.
AWS Claude handled images better—meaning that converting a PDF into an image first and then extracting the information worked more reliably than trying to process the raw PDF.
OpenAI recently announced broader file support, which could improve things, but if you’re planning to fall back to another AI model when needed, you’ll still need an extractor in place to standardize inputs.
One way or another, we had to write our own. AI can help process information, but it still requires custom-built support for real-world business workflows.
AI Can Enrich Data, Not Just Extract It
Beyond simply pulling order IDs and invoice numbers, AI could: Fill in missing CRM details (addresses, websites, contacts) Detect inconsistencies in names, suppliers, and metadata Suggest corrections when data looked incomplete
For businesses that struggle with keeping CRM and ERP data up to date, AI can save a huge amount of manual work.
Imagine AI auto-updating your CRM every time a new email comes in—no more hunting for company websites or manually correcting names.
That alone makes AI worth integrating.
AI is Smart, But It Won’t Explain Itself
While the AI provided accurate results most of the time, when it failed, understanding why was nearly impossible.
Why did it misclassify an order as an invoice? Why did it match the wrong supplier? Why did it ignore key details in one email but not another?
The lack of reasoning transparency made debugging frustrating.
It would be game-changing if AI could provide: A confidence score for its answers A brief explanation of its decision-making process
Right now, AI acts like a student who always writes the correct answer but never shows their work. Helpful, but frustrating when they’re wrong.
Final Thoughts: AI is Powerful, But Not Plug & Play
This AI integration took: 1.5 sprints (part-time) for OpenAI PoC 2 sprints (full-time) for AWS Claude PoC & implementation 1 sprint with a team of 8 to deliver full UI & business flow to end users
AI is still far from perfect, but it’s a big step forward for automation.
We’re already seeing areas for further fine-tuning, like optimizing data matching, but the potential is undeniable.
My key take aways are:
Optimize prompts for clarity & cost—concise instructions perform better and cost less. Budget for AI processing costs—longer prompts and unnecessary tokens can quickly add up. Extractor needs to be on your side—raw PDFs don’t work well, and OCR helps improve AI accuracy. Hope for better AI reasoning tools—debugging is hard when AI doesn’t explain itself.
AI isn’t replacing human input yet, but it absolutely speeds up manual processes when optimized correctly. More “wow” moments are coming, and I can’t wait to see what’s next.
#AIinBusiness #ERP #Automation #AIIntegration #MachineLearning #ProcessAutomation