{"id":2249,"date":"2020-02-11T02:15:04","date_gmt":"2020-02-10T23:15:04","guid":{"rendered":"http:\/\/kusuaks7\/?p=1854"},"modified":"2024-01-12T05:45:00","modified_gmt":"2024-01-12T05:45:00","slug":"data-lakes-the-future-of-data-warehousing","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/data-lakes-the-future-of-data-warehousing\/","title":{"rendered":"Data Lakes: The Future of Data Warehousing?"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe term\u00a0Big Data<\/em>\u00a0has been around since 2005, but what does it actually mean? Exactly how\u00a0big\u00a0<\/em>is big? We are creating data every second. It\u2019s generated across all industries and by myriad devices, from computers to industrial sensors to weather balloons and countless other sources. According to a recent study conducted by\u00a0Data Never Sleeps<\/a><\/em>, there are a quintillion bytes of data generated each minute, and the forecast is that our data will only keep growing at an unprecedented rate.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tWe have also come to realize just how important data really is. Some liken its value to something as precious to our existence as water or oil, although those aren\u2019t really valid comparisons. Water supplies can fall and petroleum stores can be depleted, but data isn\u2019t going anywhere. It only continues to grow\u2014not just in volume, but in variety and velocity. Thankfully, over the past decade, data storage has become cheaper, faster and more easily available, and as a result, where to store all this information isn\u2019t the biggest concern anymore. Industries that work in the IoT and faster payments space are now starting to push data through at a very high speed and that data is constantly changing shape.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tIn essence, all this gives rise to a \u201cdata demon.\u201d Our data has become so complex that normal techniques for harnessing it often fail, keeping us from realizing data\u2019s full potential.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tMost organizations currently treat data as a cost center. Each time a data project is spun off, there is an \u201cexpense\u201d attached to it. It\u2019s contradictive\u2014on the one side, we\u2019re proclaiming that data is our most valuable asset, but on the other side, we perceive it as a liability. It\u2019s time to change that perception, especially when it comes to banks. The volumes of data financial institutions have can be used to create tremendous value. Note that I\u2019m not talking about \u201cselling the data,\u201d but leveraging it more effectively to provide crisp analytics that delivers knowledge and drive better business decisions.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tWhat\u2019s stopping people from converting data from an expense to an asset, then?\u00a0 The technology and talent exist, but the thought process is lacking.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tData warehouses have been around for a long time and traditionally were the only way to store large amounts of data that\u2019s used for analytical and reporting purposes. However, a warehouse, as the name suggests, immediately makes one think of a rigid structure that\u2019s limited. In a physical warehouse, you can store products in three dimensions: length, breadth, and height. These dimensions, though, are limited by your warehouse\u2019s architecture. If you want to add more products, you must go through a massive upgrade process. Technically, it\u2019s doable, but not ideal. Similarly, data warehouses present a bit of rigidity when handling constantly changing data elements.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tData\u00a0lakes<\/em>\u00a0are a modern take on big data. When you think of a lake, you cannot define its shape and size, nor can you define what lives in it and how. Lakes just form\u2014even if they are man-made, there is still an element of randomness to them and it\u2019s this randomness that helps us in situations where the future is, well, sort of unpredictable. Lakes expand and contract, they change over periods of time, and they have an ecosystem that\u2019s home to various types of animals and organisms. This lake can be a source of food (such as fish) or freshwater and can even be the locale for water-based adventures. Similarly, a data lake contains a vast body of data and is able to handle that data\u2019s volume, velocity, and variety.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tWhen the mammoth data organizations like Yahoo, Google, Facebook, and LinkedIn started to realize that their data and data usage were drastically different and that it was almost impossible to use traditional methods to analyze it, they had to innovate. This, in turn, gave rise to technologies like document-based databases and big data engines like Hadoop, Spark, HPCC Systems and others. These technologies were designed to allow the flexibility one needs when handling unpredictable data inputs.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
\u201cIf you\u2019re at the earliest stage of maturity, you\u2019re used to asking questions of a SQL or NoSQL database or data warehouse in the form of reports,\u201d said Flavio Villanustre, VP of Technology for HPCC Systems and CISO at LexisNexis Risk Solutions. \u201cIn a modern data lake that has a deep learning capability with anomaly detection, you also get new insights that could have a profound effect on your company or customers, such as the discovery of a security breach or other crimes in progress, the early warning signs of a disease outbreak or fraud.\u201d<\/blockquote>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tJeff Lewis is SVP of Payments at Sutton Bank, a small community bank that\u2019s challenging the status quo for other banks in the payments space. \u201cBanks have to learn to move on from data warehouses to data lakes. The speed, accuracy, and flexibility of information coming out of a data lake is crucial to the increased operational efficiency of employees and to provide better regulatory oversight,\u201d said Lewis. \u201cBankers are no longer old school and are ready to innovate with the FinTechs of the world. A data centric thought process and approach is crucial for success.\u201d\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tData lakes are a natural choice to handle the complexity of such data, and the application of machine learning and AI are also becoming more common, as well. From using AI to clean and augment incoming data, to running complex algorithms to correlate different sources of information to detect complex fraud, there is an algorithm for just about everything. And now, with the help of distributed processing, these algorithms can be run on multiple clusters and the workload can be spread across nodes.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\nOne thing to remember is that you should be building a data lake and not a data swamp. It\u2019s hard to control a swamp. You cannot drink from it, nor can you navigate it easily. So, when you look at creating a data lake, think about what the ecosystem looks like and who your consumers are. Then, embark on a journey to build a lake on your own.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

Data lakes are a modern take on big data. A data lake contains a vast body of data and is able to handle that data’s volume, velocity, and variety. Data lakes are a natural choice to handle the complexity of such data. So, when you look at creating a data lake, think about what the ecosystem looks like and who your consumers are. Then, embark on a journey to build a lake on your own.<\/p>\n","protected":false},"author":725,"featured_media":3635,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[187],"tags":[95],"ppma_author":[3565],"class_list":["post-2249","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bigdata-cloud","tag-big-data-amp-technology"],"authors":[{"term_id":3565,"user_id":725,"is_guest":0,"slug":"adwait-joshi","display_name":"Adwait Joshi","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g","user_url":"","last_name":"Joshi","first_name":"Adwait","job_title":"","description":"Adwait Joshi is CEO of DataSeers<\/a>, a Data Analytics and Visualization company aimed towards creating futuristic solutions involving large amounts of data. He is an expert on big data analytics. "}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2249","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/725"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=2249"}],"version-history":[{"count":4,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2249\/revisions"}],"predecessor-version":[{"id":35488,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2249\/revisions\/35488"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/3635"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=2249"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=2249"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=2249"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=2249"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}