{"id":712,"date":"2018-05-31T05:19:09","date_gmt":"2018-05-31T02:19:09","guid":{"rendered":"http:\/\/kusuaks7\/?p=317"},"modified":"2026-05-26T13:11:15","modified_gmt":"2026-05-26T13:11:15","slug":"metadata-management-as-a-strategic-imperative","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/metadata-management-as-a-strategic-imperative\/","title":{"rendered":"Metadata Management as a Strategic Imperative"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tReady to learn Data Science? Browse courses<\/a>\u00a0like\u00a0Data Science Training and Certification<\/a> developed by industry thought leaders and Experfy in Harvard Innovation Lab.<\/em><\/strong>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tData is growing at an astounding rate.\u00a0Humanity is moving to a universe of sensors or \u201cresolution revolution\u201d where more granular data is being produced at a rate than can be truly analyzed and actioned upon.\u00a0As we move into a world of IoT, Industrial IoT, social media, people analytics ultimately to automation, machine learning and artificial intelligence, the hot topics seem to focus on the end of the value chain of data science rather than the bulk of the process in data management.\u00a0The hype in the \u201cwhat could be\u201d or artificial intelligence curing cancer supersedes the effective and tangible roadmap to get there.\u00a0The industry must consider alleviating the bottlenecks of data management and leverage metadata management as a strategy baked into product management and innovation at the organizational level to unlock the value of data science.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThis paper outlines the foundations of big data, new programming skills and changing environments, time required for data preparation and value in managing metadata management as a system or strategy directly correlated to success in business analytics, BI and data science.\u00a0Metadata management will be defined as the factory in productionizing data science and the defining factor in yielding meaningful insights.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

What is Big Data?<\/strong><\/h3>\n<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe term Big Data means different things to different people.\u00a0What we call Big Data is the data measured in Terabytes, and not Gigabytes.\u00a0Well, how much space is one Gigabyte?\u00a0We need to understand how storage is measured.\u00a0A bit, the smallest storage space, is equivalent to 1=0, and off =0.\u00a0A byte, is 8 \u2013 28 = 256 and 16 bits = double-byte = 256 X 256 = 65,536 bits.\u00a0This goes forward to storage measurements using the international system of units, every 1,000 bytes (103) is given a distinct name, being Kilobyte (103); Megabyte (106); Gigabyte (109); Terabyte (1012); Petabyte (1015), Exabyte (1018), Zettabyte (1021) and Yottabyte (1024).\u00a0The industry has seen that traditional databases struggle to keep up at the Terabyte level.\u00a0However, what do these numbers mean?\u00a0In a post from Business Insider in 2013, the company announced that Facebook has about 1.15 billion users and each user having over 200 pictures.\u00a0That amounts to about 230,000,000,000 imagines, and if each image is about 3MB, then the picture database size at Facebook is 690,000,000,000,000,000 bytes or 690 Petabytes or PB.\u00a0That same post also indicated that this size of the image storage increases by 350 million photos each day.1 Over a decade ago, it was published that Google processed 24 Petabytes (compare that to the 50 petabytes of all of the written works of mankind from the beginning of recorded history in all languages) of data per day, all suggesting the volume categorization of Big Data.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tIn addition, the velocity and processing issue is pushing new highs, and in 2014 ACI information group announced that every minute:\n
    \n \t
  • Facebook users share nearly 2.5 million pieces of content<\/li>\n \t
  • Twitter users tweet nearly 300,000 times<\/li>\n \t
  • Instagram users post nearly 220,000 photos<\/li>\n \t
  • YouTube users upload 72 hours of new video content<\/li>\n \t
  • Apple users download nearly 50,000 apps<\/li>\n \t
  • Email users send over 200 million messages<\/li>\n \t
  • Amazon generates over $800,000 in online sales<\/li>\n<\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tWe are dealing with data growth, and the 4 \u201cV\u201ds or characteristics of Big Data are data grows very fast (Velocity) such as 24 PB per day (Google); Data is in all kinds of formats, both structured and unstructured (Variety); Data size is humongous (Volume) such as 1.5 PB (Facebook); Data is not clean nor reliable (Veracity).\u00a0We need to be able to process humongous amount of information from all kinds of sources at a fast pace.\u00a0Companies are struggling to store, manage and analyze the data they do have, nonetheless take advantage of the emergence of data science.\u00a0The question that many organizations are faced with is how to store these types of data with traditional computer architectures, disk storage and relational databases?\u00a0We have come to realize that the answer is indeed no, and came the totally different approach to how we store the information and how to process it.\u00a0\u00a0Welcome to the world of Big Data, parallel processing and NoSQL.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tWith the inability to scale-up to meet the needs of ever-growing data, data growth required a scale-out approach due to coordination overhead, communication delay, remote file access, remote procedure calls, sharing disks, sharing memory, etc \u2013 all issues that any distributed system must deal with.\u00a0The distributed system is multiple computers that work together cohesively with parallel programming support, distributed (or parallel) operating system, distributed file system and distributed fault tolerance.\u00a0Welcome to the world of Hadoop and NoSQL.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\nThus, Hadoop is a reliable, fault-tolerant, high performance distributed parallel programming framework for large scale data written in Java.\u00a0Cloudera and Hortonworks have Hadoop environments to have a UI framework, SDK, Workflow, Scheduling, Metadata, Data Integration, Scheduling, Metadata, Languages, Compliers, Read\/Write Access and Coordination.\u00a0Hue, Oozie, Pig, Hive, Hbase, Flume, Scoop, Zookeeper are evolving into other preferred environments and languages.\u00a0It\u2019s interesting to note that Pig was created at Yahoo as an easier programming language to their limited skills within their company\u2026 using Hadoop to focus more on analyzing large data sets and spend less time having to write mapper and reducer programs. Like actual pigs, who eat almost anything, the Pig programming language is designed to handle any kind of data.\u00a0The takeaway is that it is very difficult to automate or keep this orchestration of Big Data moving and flowing in the right direction, namely to the business or organization to answer questions.\u00a0It must therefore be deduced that managing the Big Data orchestration is critical to gleaning insights.\u00a0Insights or pattern exploration with unsupervised learning, machine learning, deep learning is the next frontier, but we must flow data constantly to leverage those techniques.\u00a0This entire orchestration of Big Data is complex, but it assumes the music will continue to play if data is available to it if all those complexities are overcome.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tThe players that leverage Big Data are well-known, those being Google, Facebook, Apple and Amazon.\u00a0They have so much data they can provide hyper-customized deals and have a deeper understanding of their customers and business, automatic pattern and trend detection for the metric, data-driven organization providing value.\u00a0They have the skillsets to harness and deploy productionized data environments for near real-time changes in this ever-changing marketplace.\u00a0In many cases, they have only scraped the surface with structured data at this rate, but will continue to make advances as they automate missing data and impute with success.\u00a0However, if this is the case, the rest must look to data preparation not as a major bottleneck, but rather, as a strategic and fundamental imperative in feeding Big Data, Data Science and the business.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t

    Data Preparation as Most Time-Consuming Task<\/strong><\/h3>\n<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tData preparation accounts anywhere from 80-90% of the work of data scientists.\u00a0They spend 60% of their time on cleaning and organizing data and 19% of their time on collecting data sets, meaning data scientists spend around a whopping 80% of their time preparing and cleaning their data for analysis.5, what I refer to as the black hole. The number is about the same for their least enjoyable parts of data science.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    The black hole is a massive amount of time met with an equal proportion of disdain by those doing that work.\u00a0And again, most of the love is given to the shiny presentation layer, the end-results of analytics or visualization.\u00a0However, we must be able to serve the data to the business more quickly under an increasingly stressed data organization with Big Data volume and velocity.\u00a0The answer lies in metadata.\u00a0David Lyle, VP of Business Transformative Services at Informatica, wrote that \u201cthe difference between success and failure is proportional to the investment an organization makes in its metadata management system\u201d.6\u00a0He goes on to state to automate the process to reuse the cleansed data or production through careful preparation, annotation, and semantically define all data to make it useable and trustworthy.\u00a0These small patterns that are repeating further bogs down the machine towards information production.\u00a0If we harness the power of metadata, we construct the factory with metadata as the solid foundation that all information tools and data discovery can stand upon.\u00a0It is vital for organizations to meet the increasing need for analytics and decision making driven by data needed by marketing professionals.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t

    Metadata Management as Core Competency to Data-Driven Organizations<\/strong><\/h3>\n<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tWith Big Data increasing and the time-consuming task of data preparation, the industry is looking for better ways to improve efficiency and speed up the change of data into meaningful information.\u00a0With that said, when analyzing the data journey and the constituents data serves, metadata is the common recurring theme that enables an organization that replaces \u201cdata wrangling\u201d with data discovery.\u00a0One has to look at the opportunity costs of a limited time and value data scientists or data analysts spending his or her time on data preparation.\u00a0Data Scientists or computer programmers are considered some of the highest paid jobs, yet we have their time allocated to tasks that truly do not affect the bottom line.\u00a0The bulk of their time, as we have seen, are spent on potentially getting to those bottom line questions.\u00a0BI groups put in lots of valuable time and effort to manually discover, find and understand metadata while the business loses precious time to market.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tMetadata management must be a core competency, a place of innovation and of strategic importance.\u00a0Organizations must regularly audit their metadata and data preparation processes in order to compete.\u00a0In addition, data governance and stewardship are now considered vital.\u00a0Those who see governance and security as places of innovation will lead the next-generation of organizations.\u00a0Data regulations and sensitive data will need to be managed, and it all starts at a metadata level in order leverage, yet protect those it intends to serve.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tEnabling visibility and control of metadata that is scattered across the enterprise data landscape is key to success.\u00a0The key is tools and a management process that turns weeks of these manual processes into minutes and enables the business to move faster, allowing self-service capabilities for exploration.\u00a0The process should allow to share and reuse the data and knowledge about the data.\u00a0Metadata scanning can automatically gather from a wide variety of sources, including ETL, databases and reporting tools.\u00a0Secondly, metadata should be stored and managed in a central repository to enable share and reuse data sets, data definitions, metadata and master data.\u00a0Thirdly, metadata understanding, with smart algorithms, should model and index all types to enable the user to quickly locate and understand cross connections across connections.\u00a0Fourthly, a central tool, namely a smart search engine using hundreds of crawlers to enable searches of all metadata to present results in seconds.\u00a0Lastly, a visual mapping tool to create full lineage of the data journey as it flows through multi-vendor systems to assist in the data context to continually improve the management process.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tDavid Stodder\u2019s recommendations in\u00a0Improving Data Preparation for Business Analytics<\/em>\u00a0in TDWI\u2019s Q3 2016 Best Practices Report serves as a constructive guide on the overarching themes to create an environment for more productive users.\u00a0Make shortening time to achieving business insight a data preparation improvement priority by focusing on reducing time how long data preparation takes to deliver valuable data by using innovative technologies and methods to higher levels of repeatability through shared data catalogs, glossaries, and metadata repositories as part of the data preparation process.7\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tWhile all the above-mentioned reasons are strong reasons to productionize metadata strategies with strong tools and thought leadership, the General Data Protection Regulation (GDPR) will come into effect in May of 2018.\u00a0This means that all businesses and organizations that handle EU customer, citizen or employee data must comply with the guidelines imposed by GDPR.8\u00a0The regulation will require consent management, data breach notifications, the potential appointment of a data protection officer, privacy by default processes, privacy impact assessments as well as understanding the rights of data owners.9\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tLet\u2019s look at those regulatory guidelines and what they mean in reality.\u00a0Consent management will require the process data under the GDPR will be governed by the consent of the owner, and metadata enables to register and administer the consent.\u00a0Data breach notifications require metadata to provide information on the creation date of the information, the location and name of the database comprised and when the data breach took place.\u00a0\u00a0If a data protection officer comes into being as a role or a consideration, the metadata repository will be the main source of information for processes and measures needed to be taken for protection of personal information.\u00a0Privacy by default will require that only information that meets specific criteria for the overall purpose of the business, and metadata can assist in enforcing specific data processing are performed according to specific guidelines.10\u00a0Organizations will be required to perform Privacy Impact Assessments (PIA).\u00a0Documentation will be greatly enhanced and accessible for PIAs with metadata and its governance.\u00a0Lastly, the rights of data owners will increasingly become an issue and the time to start for 2018 is now to mitigate any business risk.\u00a0Rising to the level of metadata management and process internally will allow one to deal with the complexity of compliance.\u00a0As anything, if you harness change and that complexity, it can become of source of innovation and market leadership.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tOver the years of discussions with various companies across various verticals, listed below is illustrative of the feedback from thought leaders as well as power users of metadata:\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tClaudia Imhoff, Ph.D., President Intelligent Solutions, Inc. states \u201cA centralized metadata solution is needed in order to be a successful hunter-gatherer in the new complex world of data analytics\u201d.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tBala Venkatraman, BI Expert states a metadata strategy and tool \u201ctransformed metadata from a nice-to-have to a must have\u201d and continues, \u201cdemocratization of the data landscape which in turn yields an overall increased efficiency for any BI team\u2026\u201d.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tRevital Mor, Head of BI for Harel Insurance states \u201ceasy way for mapping data flow from end to end.\u00a0It\u2019s an important and necessary for any organization that values data accuracy\u201d.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tConsider some typical scenarios and tangible outcomes below as a summation and case for action:\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tBusiness Challenge #1:<\/em><\/strong>\u00a0A business user discovered inaccurate information on revenue reports and loses precious time to business waiting for the BI group to respond.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tBI Challenge #1<\/em>:<\/strong>\u00a0To reduce the amount of time and resources required to locate the source of inaccurate reports and correct the problem, while maintaining accuracy.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tHow BI worked prior to Metadata tools and strategic imperative #1:\u00a0In order to identify the source of the problem, BI groups had to manually trace the data to discover and understand all the hops that the data went through to land in a specific report.\u00a0They had to analyze which database tables and ETL processes were involved in and affected by the particular inaccuracy and correlate the metadata to the other reports that were affected \u2013 a very expensive, error-prone and time-consuming process.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tMetadata tools and strategic solution #1:<\/strong>\u00a0Enabling visibility and control of metadata, it can assist the information team to discover all related ETL processes and tables use to produce a report in a matter of seconds.\u00a0It can compare multiple reports to show specific metadata items, fix the problem with all the metadata centralized in one place with a search that is simple, accurate and effective.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tValue to the organization #1<\/em>:<\/strong>\u00a0Cut the metadata search and tracing costs by more than 50% with a faster time-to-market than initially estimated with little to no business disruptions or production issues when fixing the problem.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tBusiness Challenge #2<\/strong>:<\/em>\u00a0New regulation calling for companies for change HR headcount reporting from monthly to daily must be implemented accurately and quickly, without business disruptions.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tBI Challenge #2<\/strong>:<\/em>\u00a0The BI group must quickly and accurately locate the formula calculating the headcount in reports, database tables, ETL processes, views and stored procedures \u2013 without additional manpower.\u00a0The BI group must also understand all possible impacts of making the change.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tHow BI worked prior to Metadata tools and strategic imperative #2: In order to locate the specific calculation, BI had to manually map out the entire BI landscape.\u00a0With metadata scattered all over the place, this is a very time consuming and therefore costly process, which can require multiple development and design cycles before completing the change.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tMetadata tools and strategic solution #1:<\/strong>\u00a0Enable BI to make the change easily in a fraction of the time with cross-platform metadata search engine.\u00a0Easily and quickly map out the entire BI landscape with a detailed visual map showing where the calculation is being made in every system; discover the exact ETL processes and tables that make up a report; show all objects related to the calculation instances throughout the entire landscape on one screen; and implement the change quickly and accurately.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tValue to the organization #2:<\/strong>\u00a0Cut the project costs by more than 50%; complete project faster than originally estimated; elimination of overtime hours or additional staff; zero business disruptions or production issues.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    References<\/strong><\/p>

    http:\/\/businessinsider.com\/facebook-350-million-photos-each-da-2013-9<\/a><\/div>
    https:\/\/static.googleusercontent.com\/media\/research.google.com\/en\/\/archive\/mapreduce-osdi04.pdf<\/div>
    http:\/\/aci.info\/2014\/07\/12\/the-data-explosion-in-2014-minute-by-minute-infographic\/<\/a><\/div>
    http:\/\/aci.info\/2013\/12\/26\/the-amazing-growth-of-google-infographic\/<\/a><\/div>
    https:\/\/www.forbes.com\/sites\/gilpress\/2016\/03\/23\/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says\/#13e179fd6f63<\/a><\/div>
    https:\/\/blogs.informatica.com\/2016\/12\/07\/driving-analytics-data-management-competence-with-metadata\/#fbid=HKTdaIu1s6h<\/a><\/div>
    https:\/\/info.talend.com\/rs\/talend\/images\/WP_EN_DP_Improving_DataPrep_BusinessAnalytics.pdf<\/a><\/div>
    https:\/\/fr.talend.com\/blog\/2017\/06\/01\/12-months-to-gdpr-the-year-of-metadata<\/a><\/div>
    https:\/\/enterprisematters.blogs.xerox.com\/2017\/09\/14\/seven-ways-metadata-helps-gdpr\/#.WcqjpNVSwkI<\/a><\/div>

    \u00a0<\/p>

    \u00a0<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tBy:\u00a0Gal Ziton<\/a>, CTO & Co-Founder,\u00a0Octopai<\/a>\u00a0& Dan Yarmoluk, ATEK Access Technologies, All Things Data Podcast\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

    With Big Data increasing and the time-consuming task of data preparation, the industry is looking for better ways to improve efficiency and speed up the change of data into meaningful information. With that said, when analyzing the data journey and the constituents data serves, metadata is the common recurring theme that enables an organization that replaces “data wrangling” with data discovery. Metadata management must be a core competency, a place of innovation and of strategic importance.  Data regulations and sensitive data will need to be managed, and it all starts at a metadata level in order leverage, yet protect those it intends to serve. <\/p>\n","protected":false},"author":220,"featured_media":3810,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[187],"tags":[95],"ppma_author":[1890],"class_list":["post-712","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bigdata-cloud","tag-big-data-amp-technology"],"authors":[{"term_id":1890,"user_id":220,"is_guest":0,"slug":"dan-yarmoulk","display_name":"Dan Yarmoulk","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g","author_category":"","user_url":"","last_name":"Yarmoulk","first_name":"Dan","job_title":"","description":"Dan Yarmoulk is Director of Business Development (IoT, Data Science) at ATEK Access Technologies, LLC. He is a leader in IoT and Data Science, Digital Transformation, Machine Learning, Artificial Intelligence, and Business Models"}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/712","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/220"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=712"}],"version-history":[{"count":0,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/712\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/3810"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=712"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=712"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=712"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=712"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}