{"id":2287,"date":"2020-02-27T04:30:17","date_gmt":"2020-02-27T01:30:17","guid":{"rendered":"http:\/\/kusuaks7\/?p=1892"},"modified":"2024-01-03T17:03:55","modified_gmt":"2024-01-03T17:03:55","slug":"confidence-intervals-explained-simply-for-datascientists","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/confidence-intervals-explained-simply-for-datascientists\/","title":{"rendered":"Confidence Intervals Explained Simply for Data Scientists"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tRecently, I got asked about how to explain confidence intervals in simple terms to a layperson. I found that it is hard to do that.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t


Confidence Intervals are always a headache to explain even to someone who knows about them, let alone someone who doesn\u2019t understand statistics.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tI went to Wikipedia to find something and here is the definition:\n
In\u00a0statistics<\/a>, a\u00a0confidence interval<\/strong>\u00a0(CI<\/strong>) is a type of\u00a0estimate<\/a>\u00a0computed from the statistics of the observed data. This proposes a range of plausible values for an unknown\u00a0parameter<\/a>. The interval has an associated\u00a0confidence level<\/strong>\u00a0that the true parameter is in the proposed range. This is more clearly stated as: the confidence level represents the\u00a0probability<\/a>\u00a0that the unknown parameter lies in the stated interval. The level of confidence can be chosen by the investigator. In general terms, a confidence interval for an unknown parameter is based on sampling the\u00a0distribution<\/a>\u00a0of a corresponding\u00a0estimator<\/a>.\u00a0[1]<\/a><\/blockquote>\nAnd my first thought was that might be they have written it like this so that nobody could understand it. The problem here lies with a lot of terminology and language that statisticians enjoy to employ.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThis post is about explaining confidence intervals in an easy to understand way without all that pretentiousness<\/em><\/strong>.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

\n

A Real-Life problem<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tLet\u2019s start by creating a real-life scenario.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\nImagine you want to find the mean height of all the people in a particular US state.<\/em><\/strong>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\nYou could go to each person in that particular state and ask for their height, or you can do the smarter thing by taking a sample of 1000 people in the state.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThen you can use the mean of their heights (Estimated Mean<\/strong>) to estimate the average of heights in the state(True Mean<\/strong>)\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tThis is all well and good, but you being the true data scientist, are not satisfied. The estimated mean is just a single number, and you want to have a range where the true mean could lie.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tWhy do we want a range? Because in real life, we are concerned about the confidence of our estimates.<\/em><\/strong>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tTypically even if I ask you to guess the height of people in the particular US state, you are more inclined to say something like:\u00a0\u201cI believe it is between 6 foot to 6 Foot 2 Inch\u201d rather than a point estimate like \u201cIts 6 foot 2.2345 inches\u201d.<\/em>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tWe humans also like to attach a level of confidence when we give estimates. Have you ever said \u2014 \u201cI am 90% confident\u201d.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tIn this particular example, I can be more confident about the statement-\u00a0\u201cI believe it is between 5 foot to 7 Foot\u201d than \u201cI believe it is between 6 foot to 6 Foot 2 Inch\u201d as the first range is a superset of the second one.<\/em>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\nSo how do we get this range and quantify a confidence value?\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Strategy<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tTo understand how we will calculate the confidence intervals, we need to understand the Central Limit Theorem.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tCentral Limit Theorem:<\/em><\/strong>\u00a0The\u00a0Central Limit Theorem(CLT)<\/strong>\u00a0simply states that if you have a population with mean \u03bc and standard deviation \u03c3, and take random samples from the population, then the\u00a0distribution<\/strong>\u00a0of the\u00a0sample<\/strong>\u00a0means will be approximately normally\u00a0distributed with mean as the population mean<\/em><\/strong>\u00a0and estimated\u00a0standard deviation<\/strong>\u00a0s\/\u221an\u00a0where s is the standard deviation of the sample and n is the number of observations in the sample.<\/em><\/strong>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tSo knowing all this, you become curious. We already have a sample of 1000 people in the US state. Can we apply CLT?\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tWe know that the mean of the sampling distribution is equal to the population mean(which we don\u2019t know and want to estimate)and the sample deviation of the sampling distribution is given by\u00a0\u03c3\/\u221an<\/strong>( i.e., the standard deviation of the sample divided by the number of observations in the sample)\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tNow, you want to find intervals on the X-axis that contains the true population mean.<\/em>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tSo what do we do? We cast a net from the value we know.<\/em><\/strong>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\nTo get such ranges\/intervals, we go 1.96 standard deviations away from Xbar, the sample mean in both directions. And this range is the 95% confidence interval.<\/em><\/strong>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tNow, when I say that I estimate the true mean to be Xbar (The sample Mean) with a confidence interval of [Xbar-1.96SD, Xbar+1.96SD], I am saying that:\n
That this is an interval constructed using a certain procedure. Were this procedure to be repeated on numerous samples, the fraction of calculated confidence intervals (which would differ for each sample) that encompass the true population parameter would tend toward 95%<\/blockquote>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\tWhen you take 99% CI, you essentially increase the proportion and thus cast a wider net with three standard deviations.<\/strong>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
    \n \t
  • Here Xbar is the sample mean(mean of the 1000 heights sample you took).<\/li>\n \t
  • Z is the no of standard deviations away from the sample mean(1.96 for 95%, 2.576 for 99%) \u2014\u00a0level of confidence<\/em><\/strong>\u00a0you want.<\/li>\n \t
  • s is the standard deviation in the sample.<\/li>\n \t
  • n is the size of the sample.<\/li>\n<\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t“Most of the nets we cast in different experiments do contain the true population mean” \t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\nEach line in the figure above is one such experiment where the dot signifies the sample mean, and the line signifies the range. The dotted line in this figure is the true population mean.<\/em>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t

    The Critical Z value<\/h2><\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tAs we said, Z is the no of standard deviations away from the sample mean(1.96 for 95%, 2.576 for 99%) \u2014\u00a0level of confidence<\/em><\/strong>\u00a0you want.\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\tYou can go for any arbitrary level of confidence. Say, for example, you want 90% confidence. You can get that by using the idea that the shaded area inside the normal curve needs to be 0.90.\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t
    import scipy.stats as st\np = 0.9 + (1-0.9)\/2\nZ = st.norm.ppf(p, loc=0, scale=1)\nprint(Z)\n----------------------------------------------------------\n1.6448536269514722\n<\/code><\/pre>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"
    Confidence Intervals are always a headache to explain even to someone who knows about them, let alone someone who doesn’t understand statistics. In statistics, a confidence interval (CI) is a type of estimate computed from the statistics of the observed data. This proposes a range of plausible values for an unknown parameter. The interval has an associated confidence level that the true parameter is in the proposed range. This post is about explaining confidence intervals in an easy to understand way without all that pretentiousness.<\/p>\n","protected":false},"author":653,"featured_media":3821,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[187],"tags":[94],"ppma_author":[3409],"class_list":["post-2287","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bigdata-cloud","tag-data-science"],"authors":[{"term_id":3409,"user_id":653,"is_guest":0,"slug":"rahul-agarwal","display_name":"Rahul Agarwal","avatar_url":"https:\/\/www.experfy.com\/blog\/wp-content\/uploads\/2020\/04\/medium_cc5785b8-8195-44e6-a0de-2e33be05d7cb-150x150.png","user_url":"http:\/\/bit.ly\/384SBYb","last_name":"Agarwal","first_name":"Rahul","job_title":"","description":"Rahul Agarwal is a Data Scientist at Walmart Labs."}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2287","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/653"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=2287"}],"version-history":[{"count":5,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2287\/revisions"}],"predecessor-version":[{"id":35350,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/2287\/revisions\/35350"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/3821"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=2287"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=2287"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=2287"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=2287"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}