Data Integration Is Solved. So Why Is Getting Usable Data Still So Hard?
xLytix Perspectives
POSITION PAPER · DATA INTEGRATION
Modern platforms have largely solved the mechanics of moving data. The next challenge is making that capability accessible, sustainable and economically viable across the organisation.
10 min read · August 2026
In This Paper
- We Don’t Have a Shortage of Integration Technology
- The Integration Pipeline is Only the Beginning
- The Hidden Cost of the Modern Data Stack
- One Problem. Three Different Realities
- The Enterprise Challenge: Complexity
- The Mid-Market Challenge: Capacity
- The Smaller Organisation Challenge: Affordability
- Where xLytix Takes a Different View
- Make the Common Case Simple
- From Pipeline Status to Data Readiness
- Complement the Data Platform. Don’t Replace it.
- The Economic Opportunity
- What Should Organisations Measure?
- From Technology Capability to Organisational Capability
We Don’t Have a Shortage of Integration Technology
There has arguably never been a better time to build a data platform.
Organisations today can choose from managed data integration services, cloud-native platforms, open-source technologies, enterprise integration suites, streaming technologies and highly programmable engineering frameworks.
Modern platforms can connect large numbers of applications and databases, move data incrementally, capture source changes, orchestrate pipelines, respond to schema changes and increasingly use AI to assist with data engineering. Major platforms including Microsoft Fabric and Databricks continue to expand these capabilities, while specialist platforms such as Fivetran and Airbyte provide sophisticated managed or extensible approaches to data movement.
The technology is extraordinarily capable.
And yet many organisations still experience the same conversation:
“We know the data exists. How long before we can actually use it?”
That question exposes the real problem.
The technology problem is increasingly solved. The operating-model problem is not.
The Integration Pipeline Is Only the Beginning
Consider a seemingly straightforward business requirement.
A commercial director wants to understand customer profitability using information from CRM, ERP and service-management systems.
Connecting those systems may no longer be particularly difficult.
But once the data arrives, someone still needs to determine:
Which customer identifier should be used?
Which records are duplicates?
Which fields are reliable?
How should products and customers be joined?
What constitutes revenue, cost and margin?
Who owns the resulting dataset?
What happens if the ERP schema changes?
Which reports depend upon an affected field?
Can an AI assistant safely reason over this information?
A technically successful integration can therefore leave the organisation a considerable distance from a trusted business answer.
Moving data and making data usable are not the same problem.
The real lifecycle is closer to:
Connect -> Understand -> Validate -> Structure -> Govern -> Model -> Analyse -> Operate
Most organisations can acquire technologies for every stage.
The challenge is the organisational effort required to make those stages work together.
The Hidden Cost of the Modern Data Stack
Individual technology products can make modern integration appear inexpensive.
A connector can be purchased as a service.
Cloud storage is readily available.
Infrastructure can be consumed on demand.
But technology licensing is only one part of the economics.
The organisation also needs people who can configure platforms, understand source systems, create and maintain pipelines, investigate failures, manage schema changes, transform data, implement quality controls, document business meaning and support downstream consumers.
The real equation looks more like:
THE REAL COST OF DATA INTEGRATION
Technology
+ Engineering
+ Integration
+ Maintenance
+ Monitoring
+ Governance
+ Support
+ Change
+ Business Waiting Time
A problem can therefore be technically solvable while still being economically difficult to solve at scale.
And that problem looks very different depending on the organisation.
One Problem. Three Very Different Realities
| Large Enterprise | Mid-Market | Smaller Organisation | |
| Technology | Extensive and sophisticated | Increasingly enterprise-grade | SaaS, databases, spreadsheets and BI |
| Data organisation | Multiple specialist teams | Small specialist team | Few or no dedicated engineers |
| Primary challenge | Complexity | Capacity | Access |
| Economic issue | Duplication and operating cost | Skills and team cost | Affordability |
| Typical symptom | Excessive hand-offs | Engineering backlog | Manual processes |
| Opportunity | Simplify | Multiply | Enable |
The technology may be similar. The organisational problem is not.
For the Enterprise, the Problem is Complexity
A large enterprise may already own several excellent integration technologies. It may operate multiple clouds, legacy ETL platforms, departmental pipelines, central engineering teams, regional data teams and sophisticated governance functions.
The problem is rarely a lack of functionality.
Instead, a routine business requirement may travel through several teams before the resulting data becomes usable.
Business provides the requirement.
An analyst interprets it.
Engineering onboards the data.
Another team transforms it.
Governance documents or validates it.
Analytics eventually consumes it.
Every hand-off may be perfectly reasonable.
Collectively, they create latency, duplicated understanding and operational complexity.
For the enterprise, the opportunity is therefore not necessarily another way of transporting data.
The opportunity is a more consistent way to onboard, understand, govern and operationalise data across the technology investments the organisation already has.
For the Mid-Market, the Problem Is Capacity
Mid-sized organisations increasingly use many of the same technology foundations as much larger enterprises.
They may operate Snowflake, Azure, AWS, Google Cloud, Power BI, Databricks or combinations of specialist SaaS platforms.
But the data organisation may contain only a handful of people.
One team can become responsible for:
Integration;
SQL and transformation;
data models;
quality;
reporting;
platform administration;
production incidents;
and an ever-growing queue of business requests.
Every new data source consumes part of the same limited capacity.
The question therefore becomes:
How do we achieve enterprise-grade data capability without building an enterprise-sized data organisation?
For this segment, simplification isn’t merely about user experience.
It directly affects the economics of the data function.
For Smaller Organisations, the Problem is Affordability
Smaller organisations experience the problem differently again.
Their information may be distributed across CRM, accounting software, operational databases, industry applications, Excel files and BI reports.
The business may understand exactly what it wants.
What it often does not have is a dedicated data engineering organisation.
Integration consequently happens through manual exports, spreadsheets, Power Query, scripts, contractors or consultants.
These approaches can work extremely well for a period.
But as the organisation grows, hidden costs appear:
manual effort;
dependency on individuals;
inconsistent data;
repeated reconciliation;
limited governance;
and slow response to new questions.
The challenge is therefore not simply finding a cheaper connector.
It is making the operating capability surrounding modern data integration affordable
Where xLytix Takes a Different View
xLytix starts from a simple premise:
Successful data integration should not end when the data arrives. It should begin the journey towards trusted, usable business data.
xLytix Sync is therefore designed as the controlled entry point into a wider data lifecycle.
Sync -> Model -> Govern -> Analyse -> Operate
Rather than treating integration as an isolated activity, information established while data is being onboarded can remain connected to what happens afterwards.
Source structures feed modelling.
Movement creates the beginning of lineage.
Quality can be associated with the assets being created.
Analytics can remain connected to underlying models and data.
Operations can monitor the processes keeping those assets current.
xLytix’s current platform positioning reflects this wider approach: Sync sits alongside Model, Govern, Analyse and Operate in a common lifecycle, while the platform is designed to work with existing warehouse and cloud investments rather than require their replacement.
The objective is therefore not simply:
build pipelines faster.
It is:
Reduce the distance between a business requirement and usable data.
Make the Common Case Simple: Keep the Complex Case Possible
Simplification does not mean pretending that all data engineering is easy.
Complex CDC architectures, highly specialised security, proprietary protocols, extreme volumes, streaming workloads and unusual integration patterns will continue to require specialist engineering.
They should.
But not every requirement belongs in that category.
Onboarding a standard database table, file or application dataset should not necessarily require the same operating model as designing a complex enterprise streaming architecture.
xLytix is therefore built around a principle:
Specialist engineering should be applied where specialist engineering is required – not as the default starting point for every data requirement.
For common requirements, the experience should become understandable:
Select Source
Discover Data
Select Objects
Configure Movement
Validate
Run
Monitor
Continue the Lifecycle
Analysts and less-specialised users should be able to perform more controlled activity themselves. Engineers should retain the controls required for sophisticated scenarios.
The objective is not to remove data engineers.
It is to increase the leverage of data engineers.
From Pipeline Status to Data Readiness
Traditional pipeline monitoring naturally answers an important technical question:
Did the pipeline run?
But the business needs to answer a broader question:
Is the resulting data ready to use?
Imagine a customer dataset showing:
- Sync – Complete
- Freshness – Current
- Schema – Expected
- Quality – 3 warnings
- Model – Available
- Governance – Classified
That provides a very different understanding from simply:
Pipeline: Successful
The distinction matters because a pipeline can technically succeed while delivering incomplete, unexpected or poor-quality information.
The future operating model should connect technical execution with the readiness of the data assets the organisation actually consumes.
Complement the Data Platform: Don’t replace It.
For many organisations, replacing their strategic data infrastructure would make little sense. They have already invested in cloud platforms, warehouses, databases, security models and internal skills.
xLytix is therefore not based on the assumption that those investments should disappear.
Snowflake can remain Snowflake.
BigQuery can remain BigQuery.
Redshift, Synapse, PostgreSQL, ClickHouse and cloud object storage can continue providing the underlying infrastructure.
xLytix’s own platform proposition is to operate across existing stack choices and provide a common lifecycle above them.
That changes the conversation from:
“Should we replace our data platform?”
to:
“How do we get substantially more organisational value from the platform we already have?”
The Economic Opportunity
The next major efficiency gain in integration may not come from reducing the cost of moving another gigabyte of data.
It may come from reducing the human effort surrounding it.
Consider the effect of reducing;
routine engineering effort;
the number of hand-offs before a dataset becomes usable;
the time required to understand where information originated;
the effort required to diagnose common failures;
the amount of metadata reconstructed later;
the cost of maintaining ordinary integration patterns;
and the dependence on specialists for requirements that do not genuinely require specialist engineering.
Those savings compound.
For a large enterprise, they can reduce complexity.
For the mid-market, they increase the effective capacity of a constrained team.
For a small organisation, they can determine whether modern data capability is economically achievable at all.
What Should Organisations Actually Measure?
Connector count alone tells us surprisingly little about whether an integration operating model is successful.
A more useful evaluation asks:
Delivery
How long does it take to onboard a new source?
How long from the original request until the resulting data is usable?
How many engineering hours does routine onboarding require?
How many data requirements can the team support?
Operations
How quickly can failed integrations be identified?
How quickly can they be understood?
Can users distinguish technical pipeline success from the health of the resulting data?
Organisational leverage
How much routine onboarding requires specialist engineering?
Can analysts safely perform more controlled work themselves?
Are engineers spending their time on genuinely complex problems?
Governance
Is source and movement information retained?
Can downstream lineage be understood?
Can users determine ownership, freshness and trust?
Economics
What does it actually cost to introduce and maintain another source?
How much consultant or specialist dependency remains?
How much business time is lost waiting for data to become available?
Measure the operating model—not merely pipeline throughput.
From Technology Capability to Organisational Capability
Modern organisations already have access to extraordinary data technology.
The challenge has moved.
It is increasingly about making that technology usable by more people.
Preserving context as information moves through the lifecycle.
Reducing unnecessary specialist dependency.
Allowing constrained teams to support more of the organisation.
And making modern data capability economically viable for organisations that cannot – or should not need to – construct large specialist teams around every data requirement.
That is the problem xLytix Sync is designed around.
Not simply:
Can we move the data?
But:
How quickly, affordably and safely can we turn that data into something the organisation can actually use?
About xLytix Sync
xLytix Sync provides a controlled starting point for bringing data from operational systems, databases, files and other sources into an organisation’s chosen data environment and connecting that information to modelling, governance, analytics and operations.
It is part of the wider xLytix approach to making the data lifecycle more accessible without requiring organisations to abandon the infrastructure investments they already have.
Move the data. Preserve its context. Make it usable.
Continue Exploring
POSITION PAPER · DATA MODELLING
From Transformation Pipelines to Business-Ready Data Products
Modern transformation technology is powerful. The harder problem is consistently translating technical structures into data the business can understand and reuse.
POSITION PAPER · DATA GOVERNANCE
Why Data Governance Must Move Into the Flow of Work
Governance becomes difficult when it is separated from the processes that create and consume data.
xLYTIX PERSPECTIVE · AI & BUSINESS CONTEXT
AI Doesn’t Just Need More Data. It Needs Business Context.
AI can query information, but trusted business answers require meaning, relationships, rules, lineage and context beyond the raw schema.
Coming Soon
