As organizations accelerate their adoption of artificial intelligence, choosing the right tool is rarely as simple as picking the model with the highest benchmark score on public leaderboards. For New Jersey Resources (NJR), operating within an established Microsoft enterprise ecosystem meant that Microsoft Copilot was naturally available out of the box. However, initial user experiences revealed performance variability across complex financial, legal, and operational workflows. To cut through the marketing noise and determine the optimal path forward, NJR launched the AI Pepsi Challenge—a rigorous, data-driven evaluation comparing Microsoft Copilot against Perplexity across 232 scored trials spanning six foundational models and over ten core business functions.
Phase 1: Structuring the Evaluation Framework
The initiative began by establishing a standardized, defensible testing framework to ensure a strict "apples-to-apples" comparison. Rather than relying on subjective impressions, the evaluation was structured around rigorous operational dimensions:
Defining Real-World Use Cases: The evaluation covered complex enterprise scenarios, including General Ledger accounting reviews, FP&A forecasting, contract lease benchmarking, insurance standards evaluation, Robotic Process Automation (RPA) design, SOX compliance assessment, and technical Excel workbook audits.
Controlling Variables: Identical prompts, source materials, and file attachments were submitted across all participating platforms and models, isolating each tool's ability to interpret and execute complex instructions.
Holistic Scoring Criteria: Every trial was evaluated across a standardized 1-to-5 scoring matrix measuring accuracy, completeness, relevance, clarity, structure, business usefulness, and hallucination risk.
Phase 2: Execution and the Discovery of Platform Visibility
Across the 232 scored trials, the evaluation uncovered a profound technical finding that reshaped NJR’s AI strategy: Platform integration and file visibility matter more than foundational model selection alone.
While certain competing models demonstrated niche strengths in narrative financial summarizing, the evaluation proved that Microsoft Copilot held an insurmountable advantage in Microsoft Office-heavy workflows. Because Copilot is deeply integrated into the M365 architecture, it possessed native visibility into workbook components that competing platforms could not inspect. Specifically, Copilot successfully analyzed:
Complex cell formulas and named ranges
Embedded VBA macros and automation modules
Power Query transformations and data lineage
Cross-workbook dependencies and internal object structures
In contrast, external platforms frequently encountered strict file-type and structural barriers when attempting to parse .xlsm files or proprietary query logic.
Phase 3: Enterprise Reporting and Strategic Business Value
To translate raw scoring data into actionable executive insights, the evaluation culminated in structured reporting dashboards and comparative scorecards (such as Michael Rubillo's evaluation yielding top-tier performance for Copilot configurations with 262 combined score points across 8 passed test suites).
This reporting infrastructure delivered direct, measurable value to NJR in three key ways:
Maximized ROI & Licensing Consolidation: By empirically proving that Microsoft Copilot could effectively handle technical auditing, contract review, and automation documentation alongside general tasks, NJR avoided the substantial overhead and security friction of acquiring and governing redundant third-party listening or research tools.
Defensible AI Governance: The Pepsi Challenge provided internal audit and compliance teams with a transparent, auditable framework for AI adoption. By documenting model limitations, hallucination risks, and data boundaries, NJR established a blueprint for responsible, enterprise-grade AI deployment.
Operational Efficiency & Risk Reduction: Standardizing on a single enterprise platform streamlined user training, reduced cybersecurity review burdens, and empowered financial and operational analysts to leverage AI safely across daily workflows—ultimately accelerating decision-making while safeguarding institutional data.
