Why Data Integration getting more important?

The basic definition for data integration is "combining data from different sources and providing unified view of data to users to support decision making".

When we have a quick look why data integration is getting more important?

• Technology is taking place in all processes within an enterprise and this requires data integration from transactional and operational systems to decision support systems

• Everyday new technologies, which give new opportunities to enterprises, are being developed. This requires data integration between current systems and new systems

• Data is everywhere, in every type and produced by everyone. Unstructured data use is growing and importance of Big Data is increasing which also require new technologies in data integration field

• Data production and consumption cycle is getting shorter and real-time data requirements are increasing which also requires developments in the field

In AberdeenGroup's report, "Ever Harder and Faster: Managing the New Demands of Data Integration", we can see the increase in unstructured data use. I do not have any reference for more up-to-date data but I guess increase in external unstructured data use is higher. Thanks to developments in Big Data, it is possible now to analyze external data from internet and especially from social media to support decision making process. (e.g. to manage marketing campaign)

(http://v1.aberdeen.com/launch/report/perspective/8428-AI-business-data-integration.asp?lan=US)

In the same report, it is stated that "use of integration tools maximize the types and quantities of data that can be integrated and delivered to business managers". 

Data integration tools provide data quality and validation controls and make data transformation easy. If number of systems and data volume is small, coding might seem as an easy and applicable solution. However, when data requirements are increased, new data sources are added to the system, coding will get more complicated and performance will degrade. Data integration tools make it easy to expand data sources and to adopt new data  requirements. Other than these, tools provide other benefits like, real-time data process, tracebility (also by business users), automation ...etc.

There are different vendors providing tools or packages for data integration. At this point, it is good to have a look at Gartner's Magic Quadrant for Data Integration Tools.

Gartner's Magic Quadrant for Data Integration Tools

  (http://www.gartner.com/technology/reprints.do?id=1-1HBEFSF&ct=130717&st=sb)

You can find detailed strengths and cautions analysis of vendors in the report. Whether you are searching for a data integration tool or you are new to the field and trying to understand what should be expected from a data integration tool, this report will help you.

How to progress?

Technological developments in data management field increase the need for qualified experts. Rapid changes in technology and never ending requests from business force practitioners in the field to follow new developments and improve their technical skills everyday. While information technologies are being integrated with all processes in business, it is better for all proffessionals to get used to basic terms and have a general understanding of IT structures.

In this blog, I (maybe together with guest bloggers in future) will,

• share information about specific data management tools and skills for IT practitioners, 

• give information about general concepts and provide definitions for business users to understand technological environment of data management 

• and share information and analysis about developments in the field. 

A quick look at Data Management

As enterprises are growing, data need at all levels and all types is increasing and decision making is getting more complicated. What business users see and ask for more is reports that they can access aggregated and analysed data in the form of information. On the other hand, what IT developers need to do to meet this expectation is much more than creating reports. 

Other than reporting purposes, data start to act as a commodity between different departments and processes in an enterprise and also between different enterprises. This extensive need for and use of data led the expansion of data management field and accelerated technological developments.


“What we’re seeing now is an absolute explosion in data management technology and it’s come about because of the complexity of data problems” 

Mike Olson, the CEO of Hadoop company Cloudera.

Data Management is defined as "Administrative process by which the required data is acquired, validated, stored, protected, and processed, and by which its accessibility, reliability, and timeliness is ensured to satisfy the needs of data users." in http://www.businessdictionary.com.

To understand the scope of data management and to get familiar with the field, we can have a look at  10 Data Management Functions defined by The Data Management Association.

• Data Governance – The exercise of authority, control and shared decision-making (planning, monitoring and enforcement) over the management of data assets. Data Governance is high-level planning and control over data management.

• Data Architecture Management – The development and maintenance of enterprise data architecture, within the context of all enterprise architecture, and its connection with the application system solutions and projects that implement enterprise architecture.

• Data Development – The data-focused activities within the system development lifecycle (SDLC), including data modeling and data requirements analysis, design, implementation and maintenance of databases data-related solution components.

• Database Operations Management – Planning, control and support for structured data assets across the data lifecycle, from creation and acquisition through archival and purge.

• Data Security Management – Planning, implementation and control activities to ensure privacy and confidentiality and to prevent unauthorized and inappropriate data access, creation or change.

• Reference & Master Data Management – Planning, implementation and control activities to ensure consistency of contextual data values with a “golden version” of these data values.

• Data Warehousing & Business Intelligence Management – Planning, implementation and control processes to provide decision support data and support knowledge workers engaged in reporting, query and analysis.

• Document & Content Management – Planning, implementation and control activities to store, protect and access data found within electronic files and physical records (including text, graphics, image, audio, video)

• Meta Data Management – Planning, implementation and control activities to enable easy access to high quality, integrated meta data.

• Data Quality Management – Planning, implementation and control activities that apply quality management techniques to measure, assess, improve and ensure the fitness of data for use.

In addition to defined functions, "Data Integration & Interoperability" is defined as "a new knowledge area in order to highlight its emerging importance in the field. (http://www.dama.org/i4a/pages/index.cfm?pageid=3733)

Data Management Knowledge Areas, DAMA