{"id":772,"date":"2018-07-03T02:17:34","date_gmt":"2018-07-03T02:17:34","guid":{"rendered":"http:\/\/kusuaks7\/?p=377"},"modified":"2025-11-26T13:28:40","modified_gmt":"2025-11-26T13:28:40","slug":"the-challenge-of-data-scale-is-driving-data-center-automation","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/the-challenge-of-data-scale-is-driving-data-center-automation\/","title":{"rendered":"The Challenge of Data Scale is Driving Data Center Automation"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tReady to learn Machine Learning? Ready to learn Data Science? Browse\u00a0Data Science Training and Certification<\/a> courses developed by industry thought leaders and Experfy in Harvard Innovation Lab.<\/em><\/strong>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tData scale is one of the top challenges that is changing the face of data centers<\/span>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tIn the previous blog\u00a05 top data challenges that are changing the face of data centers<\/a><\/strong>, I introduced a set of challenges presented by \u201cNew Data\u201d – data that is \u00a0both transactional and unstructured, publicly available and privately collected, and its value is derived from the ability to aggregate and analyze it. \u00a0Loosely speaking we can divide this\u00a0new data\u00a0into two categories: big data \u2013 large aggregated data sets used for batch analytics \u2013 and fast data \u2013 data collected from many sources that is used to drive immediate decision making. The big data\u2013fast data paradigm is driving a completely new architecture for data centers (both public and private).\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tIn this blog, I would like to focus on\u00a0Challenge #2: \u00a0Data scale is driving data center automation.<\/strong>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tPublic cloud providers already operate at massive scale. \u00a0As smaller organizations move to public cloud, the remaining private datacenters are also getting much larger. \u00a0 A big driver for this scale is data – both the data storage itself, and the requirements for computing scale to analyze that data.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tUnfortunately, at the scale of Petabytes of storage and thousands of compute nodes, any manual management is simply cost-prohibitive. \u00a0This is leading to a completely new set of storage architectures that can operate a large scale and require very little management of the data. \u00a0This means management of moving the data, protecting the data, and making the data available at the right performance level for the analysis that is needed at any point in time – for example big data needs difference performance than fast data.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tA new class of storage vendor has emerged, whose solutions accomplish this goal through a combination of 1) software defined storage 2) commodity building block hardware componentry 3) distributed scalable storage architectures and 4) application awareness. \u00a0This combination ensures that many of the storage management needs of a datacenter are essentially automated as part of the data solution itself, and do not require outside manual interaction. \u00a0Let\u2019s look at each of these solution characteristics and how they make large scale datacenter operations cost effective:\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Software defined<\/strong> storage on commodity building block hardware<\/strong><\/h3>\n<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tBy separating data management functions completely from hardware, new software defined storage solutions make it possible to build storage systems from a small set of common hardware building blocks in a datacenter. \u00a0This dramatically reduces the cost of scaling. \u00a0No pre-planning required, just pop a new compute and\/or storage node into a rack and turn it on. \u00a0New software defined storage solutions will start to use that new capacity and performance immediately and make that capacity available to all of the users in most cases. \u00a0If new computing capabilities become available, there is no \u201clift and shift\u201d of the existing nodes, because the software doesn\u2019t really care about the underlying hardware. \u00a0The software defined storage is now provisioned and managed in the software layer so this can be done once and not every time hardware is added. \u00a0Eliminating hardware planning and management, and separating storage configuration from hardware cuts the management costs associated with large scale storage dramatically.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Distributed scalable storage architectures\u00a0<\/strong><\/h3>\n<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tManaging and provisioning individual storage units at large scale is essentially impossible. \u00a0This has led to two compute and storage architectures that address this issue: \u00a0Hyperconverged and Hyperscale. \u00a0Both architectures provide small building block units of compute and storage that can be scaled out as the datacenter grows. \u00a0Software defined storage on top of those architectures takes advantage of those building blocks to grow capacity and performance. \u00a0A critical component of these solutions is a\u00a0distributed\u00a0storage software architecture. \u00a0This means that storage is\u00a0aggregated\u00a0from across the entire hyperconverged or hyperscale hardware system and is then presented to users and applications as logical units of storage that can be any size. \u00a0The majority of these software defined architectures are also smart enough to take advantage of different types of storage that operate at different performance (such as Flash, hard drives and cold storage). \u00a0The software layers provide data management functions to perform functions like putting data on the right tier, caching data for performance, replicating data for reliability, deduplicating data for efficiency, etc. \u00a0Smart storage software that can take advantage of large scale and non-uniform hardware eliminates another big swath of management cost.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Application Awareness\u00a0<\/strong><\/h3>\n<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tAt the end of the day, data is accessed and used by applications. \u00a0This means that most of the storage is going to be provisioned to individual applications. \u00a0Historically, this was a manual operation performed by smart administrators, like DBAs (Database Administrators). \u00a0However, in the world of large scale data analytics databases like Hadoop and memory centric databases like Spark and Mongodb, manually provisioning storage in units like LUNS to applications is just not feasible. \u00a0Many of these applications also run in some kind of virtual environment, such as VMs or containers. \u00a0New storage architectures often present logical storage from a large aggregated pool\u00a0directly into the application<\/strong>. \u00a0This eliminates a couple of big manual provisioning steps that were historically required for storage. \u00a0In some cases, software defined storage architectures have the capability to give each application logical storage that is specifically created to match their unique performance requirements. \u00a0For example, a MongoDB installation will require all flash storage, while a large Hadoop cluster can operate with hard drives quite effectively. \u00a0The software defined storage solution can manage these choices automatically with some guidance from the application.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe bottom line for large scale datacenters is a requirement to move away from storage \u201cin the box\u201d and toward hyperconverged- and hyperscale-friendly software defined storage architectures that operate at scale with a minimum of manual intervention and management.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tKeep an eye out for the next blog in this series on challenges that are changing the face of datacenter, coming soon.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

As smaller organizations move to public cloud, the remaining private datacenters are also getting much larger.   A big driver for this scale is data leading to a completely new set of storage architectures that can operate a large scale and require very little management of the data. A new class of storage vendor has emerged, whose solutions accomplish this goal through a combination of 1) software defined storage 2) commodity building block hardware componentry 3) distributed scalable storage architectures and 4) application awareness. Let’s look at each of these solution characteristics and how they make large scale datacenter operations cost effective.<\/p>\n","protected":false},"author":303,"featured_media":4110,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[187],"tags":[94],"ppma_author":[1935],"class_list":["post-772","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bigdata-cloud","tag-data-science"],"authors":[{"term_id":1935,"user_id":303,"is_guest":0,"slug":"joan-wrabetz","display_name":"Joan Wrabetz","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g","user_url":"","last_name":"Wrabetz","first_name":"Joan","job_title":"","description":"Joan Wrabetz, CEO, Renaissance Consulting Group, is advising a number of early and growth stage companies acting as consultant, advisory board member, or interim executive for technology companies in Big Data, IoT, AI, and DevOPs.  She was vice president of Marketing for the Datacenter Systems Business Unit of Western Digital, and the chief technology officer for QualiSystems and the VP and CTO for the advanced technology division of EMC. She has taught as an adjunct faculty member at the University of St. Thomas, St. Mary's University, and at the Carlson School of Business at the University of Minnesota. She holds patents in load balancing, distributed systems and machine learning classification and analytics."}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/772","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/303"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=772"}],"version-history":[{"count":4,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/772\/revisions"}],"predecessor-version":[{"id":38149,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/772\/revisions\/38149"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/4110"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=772"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=772"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=772"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=772"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}