Jeff Kelly, News Editor
Published: 25 Aug 2009
While the use of data quality software has hit an all-time high, companies, by their
own admission, are still losing boatloads of money because of inaccurate data, according
to a recent survey.
The average organization surveyed by Gartner said it loses $8.2 million annually
through poor data quality. Further, of the 140 companies surveyed, 22% estimated
their annual losses resulting from bad data at $20 million. Four percent put that
figure as high as an astounding $100 million.
Much of this loss is due to lost productivity among workers who, realizing their
data is incorrect, are forced to compensate for the inaccuracies or create workarounds
when using both operational and analytic applications, Ted Friedman, an analyst
with the Stamford, Conn.-based research firm, said in an interview.
Still, losses could be even higher were it not for the increasing adoption of data
quality tools. According to Gartner, the data quality tools market grew by 26% in
2008, to $425 million.
Of those companies that use data quality tools, the survey found, many have begun
deploying them to support projects other than business intelligence (BI) and data
warehousing (DW), previously the two most common data quality use cases.
"The tools are not cheap, so people are doing the right thing by finding multiple
ways to use them," Friedman said.
Specifically, around 50% of survey respondents said they are using data quality
tools to support master data management (MDM) initiatives, and more than 40% are
using data quality technologies to assist in systems and data migration projects.
As the survey indicates, however, most companies still have a long way to go to
achieve comprehensive data quality processes. A common shortcoming, Friedman said,
is that most data quality tools are difficult for non-power users to understand
and consequently are used by only a small group of workers, usually IT staff.
According to the survey, only between one and five workers regularly interact with
data quality tools at 58% of organizations. Another 22% said between six and 10
workers use data quality tools.
To improve data quality throughout the organization, Friedman said, vendors must
make data quality tools simpler to use so that business types can use them and begin
taking responsibility for the quality of their own data.
"In particular, providing data profiling and visualization functionality (reporting
and dashboarding of data quality metrics and exceptions) to a broader set of business
users would increase awareness of data quality issues and facilitate data stewardship
activities," Friedman wrote in an accompanying report.
"Directly engaging users in specifying and maintaining business rules for cleansing,
matching and monitoring would also aid a shift in culture toward the business having
responsibility and accountability for properly managing data," he wrote.
Friedman also said organizations are increasingly applying data quality to data
domains other than customer data, but more still needs to be done. He said the quality
of financial data in particular costs some companies considerable money in the form
of fines for incorrect regulatory filings.
Companies should also invest in technology that applies data quality rules to data
at the point of capture or creation, he said, not just "downstream," as when loading
data into a data warehouse.
According to the survey, less than half of respondents currently use data quality
tools at the point of capture or creation, which often happens in operational systems,
like CRM software.
"Historically, data quality tools have been most often used in an offline, batch
mode -- cleansing data at a point in time outside the boundaries of operational
applications and processes," Friedman wrote. "Gartner advises clients to consider
pervasive data quality controls throughout their infrastructure, ensuring conformance
of data to quality rules at the point of capture and maintenance, as well as downstream."