{"id":765,"date":"2018-06-29T03:10:23","date_gmt":"2018-06-29T00:10:23","guid":{"rendered":"http:\/\/kusuaks7\/?p=370"},"modified":"2025-11-28T10:56:00","modified_gmt":"2025-11-28T10:56:00","slug":"five-top-data-challenges-that-are-changing-the-face-of-data-centers","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/five-top-data-challenges-that-are-changing-the-face-of-data-centers\/","title":{"rendered":"Five top data challenges that are changing the face of data centers"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tReady to learn Data Science? Browse\u00a0Data Science Training and Certification<\/a> courses developed by industry thought leaders and Experfy in Harvard Innovation Lab.<\/em><\/strong>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

New data center architectures present new data challenges: how data capture is driving edge-to-core data center architectures.<\/h4><\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tData is clearly not what it used to be! Organizations of all types are finding new uses for data as part of their digital transformations. Examples abound in every industry, from jet engines to grocery stores, for data becoming key to competitive advantage. I call this\u00a0new data<\/em>\u00a0<\/strong>because it is very different from the financial and ERP data that we are most familiar with. That old data was mostly transactional, and privately captured from internal sources, which drove the client\/server revolution.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tNew data<\/em>\u00a0is both transactional and unstructured, publicly available and privately collected, and its value is derived from the ability to aggregate and analyze it. Loosely speaking we can divide this\u00a0new data<\/em>\u00a0into two categories: big data \u2013 large aggregated data sets used for batch analytics \u2013 and fast data \u2013 data collected from many sources that is used to drive immediate decision making. The big data\u2013fast data paradigm is driving a completely new architecture for data centers (both public and private).\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tOver the next series of blogs, I will cover each of the top five data challenges presented by new data center architectures:\n
    \n \t
  1. Data capture is driving edge-to-core data center architectures:\u00a0<\/strong>New data<\/em>is captured at the source. That source might be beneath the ocean, in the case of oil and gas exploration, from satellites in orbit, in the case of weather applications, on your phone, in the case of pictures, video and tweets, or on the set of a movie. The volume of data collected at the source will be several orders of magnitude higher than we are familiar with today.<\/li>\n<\/ol>\n
      \n \t
    1. Data scale is driving data center automation:\u00a0<\/strong>The scale of large cloud providers is already such that they must invest heavily in automation and intelligence for managing their infrastructures. Any manual management is simply cost-prohibitive at the scale that they operate at.<\/li>\n<\/ol>\n
        \n \t
      1. Data mobility is changing global networks:\u00a0<\/strong>If data is everywhere, then it must be moved in order to be aggregated and analyzed. Just when we thought (hoped) that networks were getting faster than internet bandwidth requirements at 40 to 100 Gbps, data movement is likely to increase 100x to 1000x.<\/li>\n<\/ol>\n
          \n \t
        1. Data value is revolutionizing storage:\u00a0<\/strong>In a previous blog entitled,\u00a0\u201cMeasuring the economic value of data,\u201d<\/a>\u00a0I introduced a way of thinking about and measuring data value. There is no question that data is becoming more valuable to organizations and that the usefulness of data over longer periods of time is growing as a result of machine learning and artificial intelligence (AI) based analytics. This means that more data needs to be stored for longer periods of time and that the data must be addressable in aggregate in order for analytics to be effective.<\/li>\n<\/ol>\n
            \n \t
          1. Data analytics is the driver for compute-intensive architectures in the future:<\/strong>\u00a0Organizations are driven to keep more data in order to aggregate it into big data repositories, by the nature of analytics and in particular machine learning. These types of analytics provide better answers when applied against multiple, larger data sources.\u00a0Analytics and machine learning are compute intensive operations. As a result, analytics on large datasets drive large amounts of high speed processing. At the same time, the compute intensive nature of analytics is driving many new ways to store and access data, from in-memory databases to 100 petabyte scale object stores.<\/li>\n<\/ol>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t\t
            \n\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t
            \n\t\t\t\t\t

            Challenge No. 1: Data capture is driving edge-to-core data center architectures<\/h2>\n<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t\t
            \n\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t
            \n\t\t\t\t\t\t\t\t\tNew data<\/em>\u00a0is captured at the source. The volume of data collected at the source will be several orders of magnitude higher than we are familiar with today. For example, an autonomous car will generate\u00a0up to 4 terabytes of data per day<\/a>. Scale that for millions \u2013 or even billions of cars, and we must prepare for a\u00a0new data<\/em>onslaught.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t\t
            \n\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t
            \n\t\t\t\t\t\t\t\t\tIt is clear that we cannot capture all of that data at the source and then try to transmit it over today\u2019s networks to centralized locations for processing and storage. This is driving the development of completely new data centers, with different environments for different types of data characterized by a new \u201cedge computing\u201d environment that is optimized for capturing, storing and partially analyzing large amounts of data prior to transmission to a separate core data center environment.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t\t
            \n\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t
            \n\t\t\t\t\t\t\t\t\tThe new edge computing environments are going to drive fundamental changes in all aspects of computing infrastructures: from CPUs to GPUs and even MPUs (mini-processing units)\u2014to low power, small scale flash storage\u2014to the Internet of Things (IoT) networks and protocols that don\u2019t require what will become precious IP addressing.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t\t
            \n\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t
            \n\t\t\t\t\t\t\t\t\tLet\u2019s consider a different example of data capture. In the bioinformatics space, data is exploding at the source. In the case of mammography, the systems that capture those images are moving from two-dimensional images to three-dimensional images. The 2-D images require about 20MB of capacity for storage, while the 3-D images require as much as 3GB of storage capacity representing a 150x increase in the capacity required to store these images. Unfortunately, most of the digital storage systems in place to store 2-D images are simply not capable of cost-effectively storing 3-D images. They need to be replaced by big data repositories in order for that data to thrive.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t\t
            \n\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t
            \n\t\t\t\t\t\t\t\t\tIn addition, the type of processing that organizations are hoping to perform on these images is machine learning-based, and far more compute-intensive than any type of image processing in the past. Most importantly, in order to perform machine learning, the researchers must assemble a large number of images for processing to be effective. Assembling these images means moving or sharing images across organizations requiring the data to be captured at the source, kept in an accessible form (not on tape), aggregated into large repositories of images, and then made available for large scale machine learning analytics.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t\t
            \n\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t
            \n\t\t\t\t\t\t\t\t\tImages may be stored in their raw form, but metadata is often added at the source.\u00a0 In addition, some processing may be done at the source to maximize \u201csignal-to-noise\u201d ratios. The resulting architecture that can support these images is characterized by: (1) data storage at the source, (2) replication of data to a shared repository (often in a public cloud), (3) processing resources to analyze and process the data from the shared repository, and (4) connectivity so that results can be returned to the individual researchers. This new workflow is driving a data architecture that encompasses multiple storage locations, with data movement as required, and processing in multiple locations.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t\t
            \n\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t
            \n\t\t\t\t\t\t\t\t\tFor manufacturing IoT use cases, this change in data architecture is even more dramatic. For example, at Western Digital, we collect data from all of our manufacturing sites worldwide, and from individual manufacturing machines. That data is sent to a central big data repository that is replicated across three locations, and a subset of the data is pushed into an Apache Hadoop database in Amazon for fast data analytical processing. The results are made available to engineers all over the company for visualization and post-processing. Processing is performed on the data at the source, to improve the signal-to-noise ratio on that data, and to normalize the data. There is additional processing performed on the data as it is collected in an object storage repository in a logically central location as well.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t\t
            \n\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t
            \n\t\t\t\t\t\t\t\t\tSince that data must be protected for the long term, it is erasure-coded and spread across three separate locations. Finally, the data is again processed using analytics once it is pushed into Amazon. The architecture that has evolved to support our manufacturing use case is an edge-to-core architecture with both big data and fast data processing in many locations and components that are purpose-built for the type of processing required at each step in the process.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t\t
            \n\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t
            \n\t\t\t\t\t\t\t\t\tThese use cases require a new approach to data architectures as the concept of centralized data no longer applies. We need to have a logically centralized view of data, while having the flexibility to process data at multiple steps in any workflow. The volume of data is going to be so large, that it will be cost- and time-prohibitive to blindly push 100 percent of data into a central repository. Intelligent architectures need to develop that have an understanding of how to incrementally process the data while taking into account the tradeoffs of data size, transmission costs, and processing requirements.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t\t
            \n\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t
            \n\t\t\t\t\t\t\t\t\tAt Western Digital, we have evolved our internal IoT data architecture to have one authoritative source for data that is \u201cclean.\u201d Data is cleansed and normalized prior to reaching that authoritative source, and once it has reached it, can be pushed to multiple sources for the appropriate analytics and visualization. The authoritative source is responsible for the long term preservation of that data, so to meet our security requirements, it must be on our premises (actually, across three of our hosted internal data centers). As the majority of cleansing is processed at the source, most of the analytics are performed in the cloud to enable us to have maximum agility.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t\t
            \n\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t
            \n\t\t\t\t\t\t\t\t\tFor more information about our internal manufacturing IoT use case, see this\u00a0short video<\/a>\u00a0by our CIO, Steve Philpott.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t\t
            \n\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t
            \n\t\t\t\t\t\t\t\t\tThe bottom line is that organizations need to stop thinking about large datasets as being centrally stored and accessed. Data needs to be stored in environments that are appropriate to its intended use. We call this \u201cenvironments for data to thrive.\u201d Big data sets need to be shared, not only for collaborative processing, but aggregated for machine learning, and also broken up and moved between clouds for computing and analytics. A data center-centric architecture that addresses the big data storage problem is not a good approach. An edge-to-core architecture, combined with a hybrid cloud architecture, is required for getting the most value from big data sets in the future.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t\t
            \n\t\t\t
            \n\t\t\t\t\t\t
            \n\t\t\t\t
            \n\t\t\t\t\t\t\t\t\tThe next blog in this series will discuss data center automation to address the challenge of data scale.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

            Data is clearly not what it used to be! Organizations of all types are finding new uses for data as part of their digital transformations. New data is transactional and unstructured, publicly available and privately collected, and its value is derived from the ability to aggregate and analyze it. We can divide this new data into two categories: big data and fast data. The big data–fast data paradigm is driving a completely new architecture for data centers.I will cover each of the top five data challenges presented by new data center architectures<\/p>\n","protected":false},"author":303,"featured_media":4074,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[187],"tags":[94],"ppma_author":[1935],"class_list":["post-765","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bigdata-cloud","tag-data-science"],"authors":[{"term_id":1935,"user_id":303,"is_guest":0,"slug":"joan-wrabetz","display_name":"Joan Wrabetz","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g","user_url":"","last_name":"Wrabetz","first_name":"Joan","job_title":"","description":"Joan Wrabetz, CEO, Renaissance Consulting Group, is advising a number of early and growth stage companies acting as consultant, advisory board member, or interim executive for technology companies in Big Data, IoT, AI, and DevOPs.  She was vice president of Marketing for the Datacenter Systems Business Unit of Western Digital, and the chief technology officer for QualiSystems and the VP and CTO for the advanced technology division of EMC. She has taught as an adjunct faculty member at the University of St. Thomas, St. Mary's University, and at the Carlson School of Business at the University of Minnesota. She holds patents in load balancing, distributed systems and machine learning classification and analytics."}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/765","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/303"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=765"}],"version-history":[{"count":4,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/765\/revisions"}],"predecessor-version":[{"id":38180,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/765\/revisions\/38180"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/4074"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=765"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=765"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=765"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=765"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}