{"id":597,"date":"2018-02-05T02:31:42","date_gmt":"2018-02-04T23:31:42","guid":{"rendered":"http:\/\/kusuaks7\/?p=202"},"modified":"2025-05-21T09:46:40","modified_gmt":"2025-05-21T09:46:40","slug":"big-data-hadoop-cloud-tackling-a-chain-of-emerging-challenges","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/big-data-hadoop-cloud-tackling-a-chain-of-emerging-challenges\/","title":{"rendered":"Big Data, Hadoop & Cloud: Tackling a Chain of Emerging Challenges"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tReady to learn Hadoop Big Data?<\/em><\/strong> Browse courses<\/a>\u00a0like\u00a0Adopting Hadoop for the Enterprise: From Strategy to Roadmap<\/a> developed by industry thought leaders and Experfy in Harvard Innovation Lab.<\/em><\/strong>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tData has often been heralded as the new \u201coil\u201d \u2013 a commodity\u00a0more precious than any natural resource<\/a>\u00a0in today\u2019s digital economy. To be fair, oil and data is not an apples-to-apples comparison. Data can \u201cdrive\u201d an autonomous car, but you can\u2019t fill the gas tank with ones and zeros. However, in line with the analogy, data has travailed phases similar to oil exploration and drilling.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tFirst came the data \u201cland grab\u201d phase. About 10 years ago at the advent of big data hype, companies scrambled to ensure they didn\u2019t miss out. Then came the delineation phase, where the industry more tightly defined big data boundaries and applications. We\u2019re now in an efficiency phase. Just like with oil drilling, extracting maximum value from data is all about combining the right expertise with the right technology.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tFor all of big data\u2019s promises, many challenges came to light during the delineation phase and continue today as companies implement big data projects.\u00a0According to Gartner<\/a>, many organizations that have invested in big data projects remain stuck in the pilot stage. So what are the main challenges causing these stalls?\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Big Data Challenges<\/strong><\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tTraditional big data storage and analysis systems have buckled under the weight of large volumes of unstructured data. Due to cost and scalability issues, companies have shifted to more agile, cost-efficient open source solutions like Apache Hadoop and Spark, as well as Lumify, MongoDB, and Elasticsearch and many others. Navigating the sea of big data tools is its own challenge, but let\u2019s focus on Hadoop, a solution at the center of the big data transformation.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tFor all of the difficulties many companies experienced in their Hadoop journey over the years, it has now become mainstream, with significant ROI demonstrated across industries. Financial service and healthcare companies are augmenting, and in some cases completely replacing traditional BI\/DW-based data management systems with large scale Hadoop deployments.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tWhile Hadoop does solve many data problems, it\u2019s opened up new challenges too. Acknowledging Hadoop\u2019s potential, the hard truth is that Hadoop implementation and management (especially on-premise) is difficult and can end up causing more problems than it solves. Hadoop\u2019s learning curve and required level of expertise per industry and use case can challenge a company\u2019s internal data professionals and strain available IT resources.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tAdditionally, scaling Hadoop on premise can be a challenge, requiring more investment in physical infrastructure \u2013 something many companies don\u2019t have resources for. This is why many enterprises are moving to cloud-based Hadoop solutions, including private, public, and hybrid cloud deployments.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Cloud Migration Challenges<\/strong><\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tCloud-based Hadoop solutions allow companies to scale in a more agile fashion as their data needs increase. This can solve the problem of having to add more on-prem infrastructure over time, but as with any solution, migrating big data analytics to a cloud infrastructure begets its own set of challenges.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThese challenges largely revolve around ensuring performance, reliability, accessibility, and scalability of data. With big data and cloud implementations, there is also the looming elephant in the room of data security. This key concern has been put in the spotlight with the numerous recent high-profile enterprise data breaches and an ever-growing list of industry regulations such as HIPAA, PCI, PHI, FERPA, and GDPR.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tSo, what can enterprises do to tackle some of these ongoing challenges?\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

1. Think 5-10 years out<\/strong><\/h3>\n<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tEnterprises adopting Hadoop (or any) big data tools need to think about what\u2019s coming next. This is especially true for companies building their own big data platforms. Flexibility and scalability are essential as emerging technologies like autonomy, AI, virtual reality and IoT will generate new kinds of data faster than ever before. Big data solutions should be as future proofed as possible, as the last thing an enterprise wants to do is implement new big data infrastructure and tools only to have to turn around and do the same thing in another couple of years.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

2. Hire or partner with the right experts<\/strong><\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tWith oil exploration and drilling, the ability to maximize land investment and rig efficiency comes from combining up-to-date technology with the most specific expertise on local geology. In the same way, big data projects are more than just implementing a Hadoop solution. As mentioned earlier, Hadoop implementation and management can be difficult, and having access to experts that understand an individual enterprises\u2019 specific \u201cgeology\u201d is key to success. This means making sure you have the right internal talent, or partnering with the right experts.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

3. Take a data-centric approach to security<\/strong><\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tEnterprises need to remember when migrating big data to cloud infrastructure \u2013 whether public, private, or hybrid \u2013 the onus to secure data is theirs, not the cloud provider\u2019s. No matter what perimeter security measures are taken, data stored in a cloud environment is especially susceptible to breach. Enterprises need to think beyond perimeter security and move to identify sensitive data \u2013 both structured and unstructured, then secure it in Hadoop\/data lake as it\u2019s ingested, and constantly monitor cloud data sources for violations.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Final thoughts<\/strong><\/h3><\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tWhile big data has dropped off the hype cycle, it\u2019s not going away; in fact it will only get \u201cbigger\u201d. Likewise, the Hadoop ecosystem has matured significantly and will continue to do so with all of the big distributions offering data science capabilities. AI is the next frontier for companies with an existing large Hadoop footprint. The desire amongst enterprises to migrate big data to the cloud will continue to increase with managed services gaining momentum. There are many challenges that come with all of these things \u2013 but with the right strategy and foresight, enterprises can truly maximize big data\u2019s value.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

While big data has dropped off the hype cycle, it’s not going away; in fact it will only get “bigger”. Likewise, the Hadoop ecosystem has matured significantly and will continue to do so with all of the big distributions offering data science capabilities. The desire amongst enterprises to migrate big data to the cloud will continue to increase with managed services gaining momentum. There are many challenges that come with all of these things – but with the right strategy and foresight, enterprises can truly maximize big data’s value.<\/p>\n","protected":false},"author":199,"featured_media":3281,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[187],"tags":[122],"ppma_author":[1713],"class_list":["post-597","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bigdata-cloud","tag-big-data"],"authors":[{"term_id":1713,"user_id":199,"is_guest":0,"slug":"chandra-ambadipudi","display_name":"Chandra Ambadipudi","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g","user_url":"","last_name":"Ambadipudi","first_name":"Chandra","job_title":"","description":"Chandra Ambadipudi is Founder and CEO of Clairvoyant<\/a>. He has driven the company to become a leading big data player with multiple Fortune 500 customers today. He also co-founded BlueCanary Data, a predictive analytics product company focused on higher education."}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/597","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/199"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=597"}],"version-history":[{"count":4,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/597\/revisions"}],"predecessor-version":[{"id":37808,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/597\/revisions\/37808"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/3281"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=597"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=597"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=597"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=597"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}