Why Data Quality will matter even more in 2025

In partnership with

Dear Reader…

Happy New Year

After the holiday break many of us will be back working on the unfinished business of 2024, and starting to turn our attention to planning the big projects of 2025. As LLM’s are developing the capacity for much larger context windows - essentially enabling them to remember all your interaction with them, Data Quality is going to play an even more significant role in your projects this year.

🧱The Intersection of Data Quality and the dawn of Near-Infinite Context Windows

Two major forces are reshaping how we handle and extract value from data. On one side, data engineering is continually evolving to keep pace with the rising volumes and varieties of information. On the other side, AI engineering is unveiling new frontiers such as near-infinite context windows, allowing AI models to process and remember unprecedented amounts of information. This week we explore why data quality is going to play an even more consequential role as we make the transition to Autonomous AI Agents.

📚 An Immutable Law of Data engineering

Regardless of how advanced AI systems become, the garbage in, garbage out principle remains true. High-quality data is the bedrock for trustworthy analytics, reliable AI models, and that enable the provision of actionable insights.

Foundation for Trustworthy AI

Any AI engine depends on accurate, consistent, and timely data to function optimally. Even with vast context capabilities, if that context is riddled with errors or incomplete elements, the insights produced will be suspect or outright misleading.

• Data Contracts: One emerging way to ensure data quality in distributed environments such as data meshes is through data contracts. These define the structure, format, and quality standards for data exchange, so all teams adhere to consistent technical guidelines.

• Continuous Monitoring: Regularly checking for anomalies, missing values, and integrity issues keeps data pipelines healthy. In a data mesh implementation, processes like data quality checks and stewardship become essential to gradually improve and maintain data reliability.

Governance and Collaboration

Ensuring consistent quality in highly decentralised data architectures requires strong governance frameworks. Data Stewards tasked with oversight, clear ownership, and universal data standards help enforce quality across domains.

• Data Mesh Frameworks: By distributing data ownership, mesh architectures can scale more effectively, but they also demand rigorous governance to make sure each data product meets enterprise-wide quality benchmarks.

• AI Agentic Workflows: As AI agents handle multi-step processes autonomously, they rely on validated, standardised data sets to avoid compounding errors at each step.

Supporting RAG, LLMs, and Beyond

In retrieval-augmented generation (RAG) setups, far more is at stake than just convenience. LLMs fetch contextual data on the fly—if the quality of that external data is low, the model’s answers degrade accordingly. High-quality data directly impacts how well large language models learn, recall, and apply knowledge.

∞ The Rise of Near-Infinite Context Windows

Recent breakthroughs in AI research point to an ever-expanding capacity for context windows—the amount of data a single model can keep in “memory” and reason about. Spotlights have been cast on context windows possibly reaching up to a million tokens or more.

Enhanced Reasoning and Comprehension

Big context windows unlock the ability to:

• Maintain complex conversations without losing earlier threads of context.

• Integrate large, multi-faceted datasets in one go, reducing the need for segmentation or frequent data pipeline resets.

• Spot subtle patterns that might otherwise slip through smaller windows.

Data Quality in the Face of AI’s Expanded Memory

While having a large context window is a significant advantage, it magnifies any flaws in the underlying data. High-volume does not necessarily mean high-value:

• Noise Amplification: A model ingesting flawed data at scale could generate erroneous or biased outcomes—and repeat them across a vast range of queries.

• Governance Challenges: As models comb through bigger data sets, ensuring data privacy, compliance, and security is likely to grow more complex.

The Data Engineer’s Role

For data engineers, near-infinite context windows offer new opportunities and challenges:

• Archival to Real-Time: You may incorporate historical archives directly into a model’s working context, potentially unveiling hidden trends. But the onus is on engineers to ensure that these archives are clean, integrated, and relevant.

• Expanded Use Cases: You can feed AI with entire data dictionaries, complex schematics, or lengthy logs. This can expedite tasks like intelligent data cataloguing and semantic search across millions of records—provided that these records are accurate.

• Data Stewardship Automation: AI agents can automate tasks like data validation and anomaly detection at scale. However, they need to start from known-good baselines so that subsequent iterative refinements are incremental improvements rather than compounding errors.

Try Artisan’s All-in-one Outbound Sales Platform & AI BDR

Ava automates your entire outbound demand generation so you can get leads delivered to your inbox on autopilot. She operates within the Artisan platform, which consolidates every tool you need for outbound:

• 300M+ High-Quality B2B Prospects, including E-Commerce and Local Business Leads

• Automated Lead Enrichment With 10+ Data Sources

• Full Email Deliverability Management

• Multi-Channel Outreach Across Email & LinkedIn

• Human-Level Personalization

Book a demo to see what Ava can do.

🤝 Bridging Data Engineering and AI Engineering

1. Common Ground: Data Quality
   Both Data engineers and AI engineers must converge on data quality principles to power next-generation systems. Strong governance frameworks (including data contracts, stewardship, and standardised schemas) ensure data is analytically ready.

2. Real-Time Feedback Loops
   With agentic workflows and near-infinite context models, feedback loops get more complex. We would recommend that data pipelines should incorporate real-time validation checks, error monitoring, and automated corrective measures so that AI outputs remain trustworthy, especially as they handle ever-larger contexts.

3. Shared Tooling
   A unified toolchain—incorporating lineage tracking, version control, and data quality metrics—will help manage the interplay of massive ingestion and large context windows. Traditional data engineering tasks (like ETL and pipeline orchestration) need to adapt for ongoing, iterative knowledge assimilation.

🔭 Looking Ahead

As infinite context windows become mainstream, data engineers are poised to play an even more strategic role. The value of data quality will only increase, because the kudos (or blame) for an AI system’s performance depends largely on how meticulously the data was curated.

• Opportunity for Innovation: Data contracts, real-time anomaly detection, and agentic feedback loops will pave new avenues for data engineers to influence AI’s evolution in the enterprise.

• Cultural Shift: Hybrid teams—comprising both Data and AI professionals—will collaborate more closely on data governance, extraction, and cleaning. The lines between Data engineering and AI engineering will blur, converging around shared responsibility for data excellence.

The next wave of AI might deliver new forms of value and competitive edge, but it all rests on the same fundamental principle: quality data is the key to success. In this symbiotic future, data engineering’s unwavering focus on data reliability and governance harmonises perfectly with AI engineering’s quest for expanded reason and context. By working hand-in-hand on robust governance, agile pipelines, and continuous improvement, we can elevate both the practice and outcomes of data-driven innovation—one data point at a time.

That’s a wrap for this week

Happy New Year Data Pro’s