{"id":543,"date":"2017-11-15T02:13:26","date_gmt":"2017-11-14T23:13:26","guid":{"rendered":"http:\/\/kusuaks7\/?p=148"},"modified":"2025-04-02T12:49:50","modified_gmt":"2025-04-02T12:49:50","slug":"why-hasn-t-ai-taken-off-yet-in-monitoring","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/why-hasn-t-ai-taken-off-yet-in-monitoring\/","title":{"rendered":"Why hasn\u2019t AI taken off yet in monitoring?"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tReady to learn Data Science? Browse courses<\/a>\u00a0like\u00a0Data Science Training and Certification<\/a> developed by industry thought leaders and Experfy in Harvard Innovation Lab. <\/em><\/strong>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThere\u2019s a lot of talk about the applicability of artificial intelligence (AI) and deep learning to taming the vast quantities of data that modern Operations teams and their tools deal with. Analyst reports frequently tout AI capabilities, no matter how minor, as a strength of a product, and the lack of them as a weakness. Yet no effective use of AI seems to have emerged and claimed wide adoption in Network Operations or\u00a0Server Monitoring<\/a>. Why not? (Disclaimer: LogicMonitor does not currently have deep learning or other AI capabilities).\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tPart of the issue is that AI is a soft definition. As Rodney Brooks, the director of MIT\u2019s Artificial Intelligence Laboratory,\u00a0says<\/a>, \u201cEvery time we figure out a piece of it, it stops being magical; we say, \u2018Oh, that\u2019s just a computation.\u2019 So by definition we never really reach AI. LogicMonitor has a feature that performs numerical correlation among vast amounts of data, looking for patterns and similarities to an identified oddity (e.g. if disk latency on a database increases, it can identify what other metrics had a similar pattern. Web site requests? Network retransmissions? Database queries from a QA system? This can narrow down the candidates for the root cause). We don\u2019t think of such a system as AI, because it\u2019s\u00a0\u201cjust\u201d<\/strong>\u00a0the application of well understood statistical methods. To an operations person from 20 years ago, however, it would definitely seem like intelligence.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tSome techniques are (for the moment, at least) universally recognized as AI. Machine learning is one of them. Machine learning is finding applicability in all sorts of fields that were thought to be the province of \u201creal\u201d intelligence. It can beat humans at chess, at Go, compose symphonies and haiku as good as those generated by poets. So why not operational troubleshooting?\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tWell, one reason is that supervised machine learning needs to learn, and then follow, rules. It has to be trained on a set of data, such as completed games. From the training set it can generate a model, and use that model it to apply what it has learned to new games or compositions.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\nOne problem with supervised learning in the Ops world is that you can\u2019t tell what rules the AI extracted. It may be getting the correct result, but for the wrong reasons.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThe most fun example of this I know of is\u00a0Clever Hans<\/a>. Hans was a horse that could do arithmetic. Addition, subtraction, division. Ask the horse a question, and he would indicate when a person counting out loud had reached the correct answer. He would do this reliably, and regularly, and repeatedly. The only problem was that the horse wasn\u2019t solving mathematical problems and then listening for the correct number to be said. He was just looking at the people around him, and when their body language indicated the right moment, he\u2019d stomp his foot. He was effectively a neural net that had been trained to give the answer to mathematical puzzles \u2014 but Hans was extracting the answers not in the way that was expected. And not in a way that was useful if you actually wanted to rely on Hans\u2019 computational abilities, and didn\u2019t know the answer in advance.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tSimilarly, AI vision recognition systems can be trained to distinguish leopards from cheetahs from ocelots, very accurately. But they also identify\u00a0couches as leopards. (One could imagine problems if one was relying on these algorithms to defend against leopards\u2026)\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tYou can check the work of an AI when you already know the answer \u2014 but relying on it when you don\u2019t know the answer makes you depend on whatever cues the system deemed applicable from its training data \u2014 and you don\u2019t know what those were. This could be that a device is identified as a likely root cause because it has more than the usual incidence of the letter \u201cP\u201d in its hostname, which may have been an unknown artifact of the training data set.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tOf course, humans are also subject to an inability to explain why they make their decisions or take their actions much of the time. I was unaware, until I read a book about bicycle racing, that in order to initiate a turn to the left, the process is started by a slight steer to the right, to change the center of balance. I\u2019d been riding bikes for decades, and would have denied that I did that. For those that want many more examples of unaware behavior, demonstrating that we often come up with a conscious narrative for why we made a decision after we\u2019ve decided, justifying ex-post-facto why we did something, I highly recommend \u201cThinking, Fast and Slow<\/a>.\u201d Most operational troubleshooting however, is definitely\u00a0slow thinking<\/em>, where we consciously investigate step by step, starting from the symptomatic system, often taking shortcuts from our knowledge of the application, but investigating and testing explicit hypotheses.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tI would also acknowledge that the inability to explain your reasoning doesn\u2019t matter if you get the right answer. Professional bike racers will be able to tell you exactly how fast they can go around a given corner before they lose traction and slide out. So could a physicist. The physicist can show his work, and explain the co-efficient of friction, and the lateral and vertical vectors. The cyclist will not be able to explain how he knows \u2014 but he will know.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tSo the lack of insight into an AI\u2019s processes isn\u2019t necessarily an obstacle to their use in operations. Rather, it\u2019s the fact that an AI is limited by its training set. An AI can write a symphony that we enjoy, because it conforms to our current expectations of what an artistic and pleasant symphony should sound like, but it cannot push the boundaries of creativity, creating symphonies that violate rules and create enough emotional impact that they cause\u00a0riots<\/a> \u2014 then were soon regarded as genteel art.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tIn IT Operations,\u00a0good operational practices dictate that every issue and problem is unique<\/strong>. All prior issues should have been addressed in a way that means they won\u2019t recur, or \u2014 at the very least \u2014 the monitoring has been configured so that it clearly warns of the situation. If neither of those conditions is true, the incident should still be regarded as open. So if the issues are always unique, the training sets will not have covered them, and the insights from an AI are unlikely to be terribly insightful, or helpful. They may in fact be distractions and wild goose chases \u2014 like asking Clever Hans to calculate your tax return.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tOne way around this would be to mine the information across a diverse set of customer operational data \u2014 the fact that I had an issue with Zookeeper that had quorum configuration as the root cause, and I resolved it, may mean that the issue shouldn\u2019t recur for me \u2014 but that knowledge may be useful to other companies running zookeeper. Of course there are problems here not only with data privacy, but also data training. If I identify my zookeeper nodes as z1.prod, with tags #zookeeper and #prod and you call yours n34.lax.us.west with tags #quorum and #live \u2014 how is commonality to be established so that the training lessons can be applied?\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tTechnology can certainly add value to monitoring systems now \u2014 it can cluster alerts from different systems together into single incidents, based on commonalities of time, and (ideally) a knowledge of the topology and dependencies of the system. This can help reduce alert overload, and help simplify root cause detection. It is arguable whether this is AI, however. (Note that the ideal state of \u2018knowing\u2019 dependencies may not be possible for the system, as dependencies can change with load; whether data is cached or not; whenever new code is deployed; or whenever new nodes or containers are added\/removed. Note also that changes in dependencies that cause incidents may not be known by the monitoring system (as the communication mechanisms may themselves be disrupted by the incident) so the monitoring may be misidentifying some alerts as related to the incident based on old data. As the Google team\u00a0puts it<\/a>\u00a0\u201cFew teams at Google maintain complex dependency hierarchies because our infrastructure has a steady rate of continuous refactoring.\u201c)\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tAI is probably better suited to pointing out discrepancies and anomalies in performance over time. AI can alert you to issues caused by releases \u2014 for example, that rendering a page used to take 2 database requests \u2014 now it takes 10. Identifying deviations in performance between versions of code, and whether they are significant, is going to be an increasingly important role of monitoring as development agility increases.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tOf course, progress is continuous in this field. Since I started writing this article, a team from LinkedIn and Stanford have\u00a0published a paper<\/a>\u00a0that shows promise in automating the identification of root cause, using unsupervised machine learning to cluster performance anomalies, as well as snapshots of the call graph of data flows. This shows promise \u2014 however, it is hard to generalize this work to situations where you do not control the code that generates the metrics, thus allowing the automation of the call graph data. (Think routers, commercial software, storage arrays, etc \u2014 all of which may be the root cause of issues, but outside of the call graph.)\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tRegardless, it is heartening to see progress here. Speaking as a former network and systems administrator that was often on call \u2014 the easier we can make life for the people on the front lines of monitoring and alert response, the better for everyone. (And of course, that\u2019s our whole focus at LogicMonitor, so we\u2019ll be paying close attention to this space.)\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tOriginally posted at Breathe Publication<\/a>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

There’s a lot of talk about the applicability of artificial intelligence (AI) and deep learning to taming the vast quantities of data that modern Operations teams and their tools deal with. Analyst reports frequently tout AI capabilities, no matter how minor, as a strength of a product, and the lack of them as a weakness.<\/p>\n","protected":false},"author":84,"featured_media":3032,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[187],"tags":[94],"ppma_author":[1634],"class_list":["post-543","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bigdata-cloud","tag-data-science"],"authors":[{"term_id":1634,"user_id":84,"is_guest":0,"slug":"steve-francis","display_name":"Steve Francis","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/?s=96&d=mm&r=g","user_url":"","last_name":"Francis","first_name":"Steve","job_title":"","description":"Steve is the Founder and Chief Evangelist at LogicMonitor. Since its founding, LogicMonitor has become one of the strongest growing IT software companies in the world, with nearly 1,200 customers. Prior to founding LogicMonitor, Steve was responsible for the datacenter operations of a diverse group of organizations including National Geographic, the University of California, Citrix Online, and Valueclick. "}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/543","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/84"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=543"}],"version-history":[{"count":5,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/543\/revisions"}],"predecessor-version":[{"id":37565,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/543\/revisions\/37565"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/3032"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=543"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=543"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=543"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=543"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}