{"id":9737,"date":"2020-09-18T07:41:07","date_gmt":"2020-09-18T07:41:07","guid":{"rendered":"https:\/\/www.experfy.com\/blog\/?p=9737"},"modified":"2023-11-06T15:55:36","modified_gmt":"2023-11-06T15:55:36","slug":"getting-started-with-ai-start-here","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/ai-ml\/getting-started-with-ai-start-here\/","title":{"rendered":"Getting Started With AI? Start Here!"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"9737\" class=\"elementor elementor-9737\" data-elementor-post-type=\"post\">\n\t\t\t\t\t\t<section class=\"has_eae_slider elementor-section elementor-top-section elementor-element elementor-element-51e1a04 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"51e1a04\" data-element_type=\"section\" data-e-type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"has_eae_slider elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-27401cd3\" data-id=\"27401cd3\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-abbd961 elementor-widget elementor-widget-heading\" data-id=\"abbd961\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Everything you need to know to dive into your\u00a0project<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8b658df elementor-widget elementor-widget-text-editor\" data-id=\"8b658df\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\n<p>Many teams try to start an applied AI project by\u00a0<a href=\"https:\/\/www.experfy.com\/blog\/the-first-step-in-ai-might-surprise-you\/\" target=\"_blank\" rel=\"noreferrer noopener\">diving into algorithms and data before figuring out desired outputs and objectives.<\/a>\u00a0Unfortunately, that\u2019s like raising a puppy in a New York City apartment for a few years, then being surprised that it can\u2019t herd sheep for you.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p>You can\u2019t expect to get anything useful by asking wizards to sprinkle machine learning magic on your business without some effort from you\u00a0first.<\/p>\n<\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-141a266 elementor-widget elementor-widget-text-editor\" data-id=\"141a266\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\n<p>Instead, the first step is for the\u00a0<em>owner<\/em>\u200a\u2014\u200athat\u2019s you!\u200a\u2014\u200ato form a clear vision of what you want from your dog (or\u00a0<a href=\"http:\/\/bit.ly\/quaesita_ai\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">ML\/AI<\/a>\u00a0system) and how you\u2019ll know you\u2019ve trained it successfully.<\/p>\n\n\n\n<p>My\u00a0<a href=\"http:\/\/bit.ly\/quaesita_first\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">previous article<\/a>\u00a0discussed the\u00a0<a href=\"http:\/\/bit.ly\/quaesita_first\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\"><strong>why<\/strong><\/a>, now it\u2019s time to dive into\u00a0<strong>how<\/strong>\u00a0to do this first step for\u00a0<a href=\"http:\/\/bit.ly\/quaesita_ai\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">ML\/AI<\/a>, with all its gory little sub-steps.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f58b6cd elementor-widget elementor-widget-text-editor\" data-id=\"f58b6cd\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\n<p>This reference guide is densely-packed and long, so feel free to stick to large fonts and headings for a two-minute crash course or head straight to the summary\u00a0<a href=\"http:\/\/bit.ly\/quaesita_realitycheck\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">checklist version<\/a>. Here\u2019s the table of contents:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Figure out who\u2019s in charge<\/strong><\/li>\n<li><strong>Identify the use case<\/strong><\/li>\n<li><strong>Do some reality checks<\/strong><\/li>\n<li><strong>Craft a performance metric wisely<\/strong><\/li>\n<li><strong>Set testing criteria to overcome human biases<\/strong><\/li>\n<\/ul>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a072a08 elementor-widget elementor-widget-text-editor\" data-id=\"a072a08\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p><strong style=\"font-size: 19px;\">Cast of characters:<\/strong><span style=\"font-size: 19px;\">\u00a0decision-maker, ethicist, ML\/AI engineer, analyst, qualitative expert, economist, psychologist, reliability engineer, AI researcher, domain expert, UX specialist, statistician, AI control theorist.<\/span><\/p>\n<p><!-- \/wp:paragraph --><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5eb4c01 elementor-widget elementor-widget-image\" data-id=\"5eb4c01\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/hackernoon.com\/hn-images\/0*VVLwSRkwP37Hx5tw.jpg\" alt=\"\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-eb330bf elementor-widget elementor-widget-heading\" data-id=\"eb330bf\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Figure out who\u2019s in\u00a0charge<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-327b1ba elementor-widget elementor-widget-text-editor\" data-id=\"327b1ba\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>The tasks we\u2019re about to tackle are the responsibility of the project\u2019s responsible adult.\u00a0<a href=\"http:\/\/bit.ly\/quaesita_first\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">That\u2019s whoever calls the shots.<\/a>\u00a0If a PhD researcher is selected for this role, it should be because of that person\u2019s decision skills and deep understanding of your business. If you\u2019re about to put them in this role and then second-guess them later, you\u2019ve chosen the wrong person. The entity we\u2019re calling\u00a0<em>The Decision-Maker<\/em>\u00a0(it could be a person or committee) is the one that\u2019s supposed to get final say. Elect your benevolent dictator wisely.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:quote -->\n<blockquote class=\"wp-block-quote\">\n<p>If the decision-maker is someone you\u2019re planning to second-guess later, you\u2019re doing it\u00a0wrong.<\/p>\n<\/blockquote>\n<!-- \/wp:quote -->\n\n<!-- wp:paragraph -->\n<p>If the decision-maker isn\u2019t well-versed in the art and science of decision-making, there\u2019s a fix: pair them up with a\u00a0<a href=\"http:\/\/bit.ly\/quaesita_roles\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">qualitative expert<\/a>. But if the person in charge doesn\u2019t understand your business, you may as well just flush that cash now.<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a55d3af elementor-widget elementor-widget-image\" data-id=\"a55d3af\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/hackernoon.com\/hn-images\/0*OWzhu8fl5lbqRxrk.jpg\" alt=\"\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1c54a11 elementor-widget elementor-widget-heading\" data-id=\"1c54a11\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Identify the use\u00a0case<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8bd476e elementor-widget elementor-widget-heading\" data-id=\"8bd476e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Focus on the\u00a0outputs<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-346a35c elementor-widget elementor-widget-text-editor\" data-id=\"346a35c\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>The key thing is that\u00a0<a href=\"http:\/\/bit.ly\/quaesita_ai\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">ML\/AI<\/a>\u00a0is not magic and it doesn\u2019t solve every problem. It\u2019s a\u00a0<a href=\"http:\/\/bit.ly\/quaesita_simplest\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">thing-labeler<\/a>\u00a0and it\u2019s up to you to\u00a0<a href=\"http:\/\/bit.ly\/quaesita_island\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">figure out<\/a>\u00a0what you need labeled.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p><a href=\"http:\/\/bit.ly\/quaesita_simplest\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">Thing-labeling<\/a>\u00a0doesn\u2019t only mean classification\u200a\u2014\u200a\u201cIs this photo of a cat or not?\u201d\u200a\u2014\u200athat\u2019s not thinking big enough. By label I mean output. It could be a\u00a0<a href=\"http:\/\/bit.ly\/quaesita_slkid\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">category,<\/a>\u00a0a number, a sentence, waveform, a\u00a0<a href=\"http:\/\/bit.ly\/quaesita_unsupervised\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">group ID<\/a>, a single action, a joystick movement, a sequence of actions, Y\/N on whether something is an anomaly\u2026\u00a0<a href=\"http:\/\/bit.ly\/quaesita_island\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">so many possibilities<\/a>!<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f9e8648 elementor-widget elementor-widget-text-editor\" data-id=\"f9e8648\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:quote -->\n<blockquote class=\"wp-block-quote\">\n<p>Don\u2019t hire that PhD guru before you\u2019ve confirmed you need them. Focus on the outputs\u00a0first.<\/p>\n<\/blockquote>\n<!-- \/wp:quote -->\n\n<!-- wp:paragraph -->\n<p>If you read my\u00a0<a href=\"http:\/\/bit.ly\/quaesita_emperor\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">recent article about how algorithms work<\/a>, you will have noticed that the article took for granted that it\u2019s worth labelling bottles of\u00a0<a href=\"http:\/\/bit.ly\/quaesita_emperor\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">wine<\/a>\u00a0as liked or disliked by Cassie. Who signed off on wasting everyone\u2019s time with that stupid use case?! How does it help the business? Should that classifier even exist? Supposing it can be made to work, is it worth building?<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<section class=\"has_eae_slider elementor-section elementor-top-section elementor-element elementor-element-7b84532 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"7b84532\" data-element_type=\"section\" data-e-type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"has_eae_slider elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-8d5cf95\" data-id=\"8d5cf95\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-22e75d0 elementor-widget elementor-widget-text-editor\" data-id=\"22e75d0\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p><em>That\u2019s<\/em>\u00a0the kind of job you\u2019re doing right now, my friend. It\u2019s the first job.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:quote -->\n<blockquote class=\"wp-block-quote\">\n<p>Just because you can do something, doesn\u2019t mean it\u2019s a good use of anyone\u2019s\u00a0time.<\/p>\n<\/blockquote>\n<!-- \/wp:quote -->\n\n<!-- wp:paragraph -->\n<p>Imagine your\u00a0<a href=\"http:\/\/bit.ly\/quaesita_ai\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">ML\/AI<\/a>\u00a0system is operational and ask yourself if you\u2019re happy you sunk company resources into making it. No? Keep brainstorming. Better to discover no one needs your application\u00a0<em>before<\/em>\u00a0several PhDs waste their lives on it.<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-79d5939 elementor-widget elementor-widget-text-editor\" data-id=\"79d5939\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>This task can be hard because there are so many options, so settle into a comfy couch and meditate. Try my\u00a0<a href=\"http:\/\/bit.ly\/quaesita_island\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">drunk island<\/a>\u00a0exercise if you need a bit of help brainstorming.<\/p>\n<!-- \/wp:paragraph -->\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-9f981b8 elementor-widget elementor-widget-heading\" data-id=\"9f981b8\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Now is not the time for\u00a0inputs<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0aaea9d elementor-widget elementor-widget-text-editor\" data-id=\"0aaea9d\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>Some of you decision folk are gorgeously fluent with data. You can talk inputs and outputs all at once\u2026 and you understand the difference. My advice may surprise you: resist temptation! Just don\u2019t talk about inputs yet. Seriously. I know you can, but don\u2019t. Here are two reasons of many.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p><strong><em>Reason 1: Missed opportunities<\/em><\/strong><\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6c6175f elementor-widget elementor-widget-text-editor\" data-id=\"6c6175f\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>This is the obvious one. Some of your stakeholders are not as fluent as you and they get confused easily. In the early days, you might be pitching your idea to them in the hopes of securing resources for your project and you really don\u2019t want them to misunderstand why your system is worth having. Don\u2019t confuse them! Stay focused. Tell them what it makes, not how it makes it.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:quote -->\n<blockquote class=\"wp-block-quote\">\n<p>Ask yourself, \u201cIs this the end or the means?\u201d If it\u2019s the means, don\u2019t talk about it for\u00a0now.<\/p>\n<\/blockquote>\n<!-- \/wp:quote -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-9e85cbf elementor-widget elementor-widget-text-editor\" data-id=\"9e85cbf\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>The trouble with many fluent folk is that they think everyone shares their fluency. Some utterly brilliant people in tech have surprised me by not having this skill, so now I know not to take it for granted.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>To some people, data is data. It\u2019s all the same. (Dear reader, if you\u2019re not sure whether you\u2019re fluent in this, really force yourself to slow down. Keep asking yourself, \u201cIs this the end or the means?\u201d Make sure you focus on the ends for now.) Stakeholders might just not be able to follow the thread of your argument, which means your pitch will fall flat and you\u2019ll miss a chance to make the world more awesome with AI.<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d75cc24 elementor-widget elementor-widget-image\" data-id=\"d75cc24\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/hackernoon.com\/hn-images\/0*gbA1VnBXp1kMjoRw.jpg\" alt=\"\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-956f521 elementor-widget elementor-widget-text-editor\" data-id=\"956f521\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p><strong><em>Reason 2: Tacit agreement<\/em><\/strong><\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>As an engineer who has been around engineers for a long time, I\u2019ve noticed that our kind loves latching onto details. Big picture be damned, it\u2019s so much fun to kick the stuffing out of every nitty gritty, especially when someone is wrong about something! We love us some technical correctness.<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a803863 elementor-widget elementor-widget-image\" data-id=\"a803863\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/hackernoon.com\/hn-images\/0*6-CDAw2OkvLkyjA7.jpg\" alt=\"\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f55adce elementor-widget elementor-widget-text-editor\" data-id=\"f55adce\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>So here\u2019s the tragicomedy: when you\u2019ve spent the last 6 hours arguing with your buddies about whether or not variable\u00a0<em>x\u00a0<\/em>(raw, standardized, or normalized?) is a good input with suitable logging for predicting output\u00a0<em>y<\/em>, you\u2019ve, ahem, normalized the idea that\u00a0<em>y<\/em>\u00a0is worth pursuing. You stop questioning the point of working on\u00a0<em>y<\/em>\u00a0in the first place and end up building things that don\u2019t need to be built.<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5988658 elementor-widget elementor-widget-heading\" data-id=\"5988658\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">This is about teeming multitudes<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1dd4576 elementor-widget elementor-widget-text-editor\" data-id=\"1dd4576\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>Let\u2019s imagine that automating the Y\/N labelling of\u00a0<a href=\"http:\/\/bit.ly\/quaesita_emperor\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">wine bottles<\/a>\u00a0is the use case you\u2019re going for. (If you prefer tea to wine,\u00a0<a href=\"http:\/\/bit.ly\/quaesita_dmguide2\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">here\u2019s<\/a>\u00a0the alcohol-free version of this text.)<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>Make sure you\u2019re not thinking about labelling just one or two bottles.\u00a0<a href=\"http:\/\/bit.ly\/quaesita_ai\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">ML\/AI<\/a>\u00a0makes sense for automating\u00a0<em>many<\/em>\u00a0repeated decisions. It\u2019s not for one-offs.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:quote -->\n<blockquote class=\"wp-block-quote\">\n<p>ML\/AI is not for one-offs, so make sure your business needs an impressive number of items\u00a0labeled.<\/p>\n<\/blockquote>\n<!-- \/wp:quote -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e8ce315 elementor-widget elementor-widget-text-editor\" data-id=\"e8ce315\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>You\u2019re imaging labelling at least a few thousand of them? And when this thing is live, you\u2019re sure you can\u2019t just look the answers up instead of predicting them? Good. Let\u2019s continue.<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<section class=\"has_eae_slider elementor-section elementor-top-section elementor-element elementor-element-32eeda1 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"32eeda1\" data-element_type=\"section\" data-e-type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"has_eae_slider elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-d7647e0\" data-id=\"d7647e0\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-a654e06 elementor-widget elementor-widget-image\" data-id=\"a654e06\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/hackernoon.com\/hn-images\/1*eBPLxj79_ycHL45L7ZgOXA.png\" alt=\"\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-9baa91f elementor-widget elementor-widget-text-editor\" data-id=\"9baa91f\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>Grab a pen, write down the labels you\u2019ll accept (binary Y\/N in this exercise is easy to write down, but you could have chosen to make them\u00a0<a href=\"http:\/\/bit.ly\/quaesita_island\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">more exotic<\/a>\u00a0if you needed to get creative). Write down how you\u2019d know if the answer is right for one of them. Write down what mistakes would look like. Expect mistakes in machine learning! If you\u2019re expecting flawlessness, it\u2019s best to back away quietly before disappointment crushes your soul.<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-26f684c elementor-widget elementor-widget-heading\" data-id=\"26f684c\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">You might not be ready for machine\u00a0learning<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3acc130 elementor-widget elementor-widget-text-editor\" data-id=\"3acc130\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>Still struggling to find a use case? Consider pausing\u00a0<a href=\"http:\/\/bit.ly\/quaesita_ai\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">ML\/AI<\/a>\u00a0in favor of\u00a0<a href=\"http:\/\/bit.ly\/quaesita_analysts\" target=\"_blank\" rel=\"noreferrer noopener\">analytics<\/a>\u00a0for a while. The\u00a0<a href=\"http:\/\/bit.ly\/quaesita_datasci\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">goal of analytics<\/a>\u00a0is to generate inspiration for the decision-maker. Once you\u2019re inspired, you can come back to\u00a0<a href=\"http:\/\/bit.ly\/quaesita_ai\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">ML\/AI<\/a>\u00a0and start over.\u00a0<a href=\"http:\/\/bit.ly\/quaesita_roles\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">Analytics<\/a>\u00a0(a.k.a. data-mining) is a great idea for every project, whereas\u00a0<a href=\"http:\/\/bit.ly\/quaesita_roles\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">ML\/AI<\/a>\u00a0is only for projects where the goal is to use data to automate thing-labeling. Although the underlying math is often the same, the processes are very different.\u00a0<a href=\"http:\/\/bit.ly\/quaesita_roles\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">Data-mining<\/a>\u00a0is all about\u00a0<a href=\"http:\/\/bit.ly\/quaesita_analysts\" target=\"_blank\" rel=\"noreferrer noopener\">maximizing the speed of discovery<\/a>, while\u00a0<a href=\"http:\/\/bit.ly\/quaesita_datasci\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">ML\/AI<\/a>\u00a0is all about performance in automation. In data-mining, there\u2019s\u00a0<a href=\"https:\/\/hackernoon.com\/what-on-earth-is-data-science-eb1237d8cb37\" target=\"_blank\" rel=\"noreferrer noopener\">only one mistake<\/a>\u00a0your team can make, whereas ML\/AI has an impressive list of potential kablooies. Sign up for those headaches only when you have a use case that\u2019s worth the bother.<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d9b36b8 elementor-widget elementor-widget-heading\" data-id=\"d9b36b8\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Who is it for? Think about your\u00a0users!<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-056c8dc elementor-widget elementor-widget-text-editor\" data-id=\"056c8dc\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>Who is your dazzling invention for? Who benefits? This is a great time to consult with a UX (user experience) specialist and map out your application\u2019s intended users.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:quote -->\n<blockquote class=\"wp-block-quote\">\n<p>Ideating new technology often starts with the what, but it\u2019s important to cover the who before you proceed to the\u00a0how.<\/p>\n<\/blockquote>\n<!-- \/wp:quote -->\n\n<!-- wp:paragraph -->\n<p>Something I\u2019ve learned by spending time with UX designers is that my knee-jerk description of who my users are\u2026 is usually rather na\u00efve. Have I thought about usability for indirect beneficiaries, for society-as-a-whole, for other businesses, for machine systems using the output as input, for engineers engaged in debugging, and so on? To avoid shoddy UX design, please take the time to think about all your myriad user categories before you proceed. User doesn\u2019t just mean customer or end user in the most obvious sense.<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6fa6a56 elementor-widget elementor-widget-heading\" data-id=\"6fa6a56\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Is it ethical to\u00a0proceed?<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-97acd5f elementor-widget elementor-widget-text-editor\" data-id=\"97acd5f\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>What if your idea isn\u2019t uniformly beneficial to everyone? While mapping out your ideal use cases, consider those who might be harmed by the existence of your system. I don\u2019t just mean competitors in your industry. Can any humans be harmed by your application? This is especially important if your technology scales to millions or billions.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:quote -->\n<blockquote class=\"wp-block-quote\">\n<p>Think about the humans your creation impacts! Who benefits and who might be\u00a0harmed?<\/p>\n<\/blockquote>\n<!-- \/wp:quote -->\n\n<!-- wp:paragraph -->\n<p>If you care about developing technology in an ethical and responsible manner, it\u2019s your duty to think about\u00a0<em>all<\/em>\u00a0the people your creation could affect. Thinking about them\u00a0<em>after<\/em>\u00a0you\u2019ve built it is irresponsible. The time for it is\u00a0<em>now<\/em>! An\u00a0<a href=\"http:\/\/bit.ly\/merethics\" target=\"_blank\" rel=\"noreferrer noopener\">ethicist<\/a>\u00a0can help you shoulder some of the burden. Bringing them on board as an advisor is a great idea if your project has the potential to strongly affect human wellbeing. Along with your UX specialist, their participation in the project will help you ensure that groups affected by your creation are given a voice.<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d3489c5 elementor-widget elementor-widget-heading\" data-id=\"d3489c5\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Do some reality\u00a0checks<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4363da0 elementor-widget elementor-widget-text-editor\" data-id=\"4363da0\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>Once you can clearly articulate what labels you\u2019re after, it\u2019s time for a quick reality check: do you have data about this business problem?<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>No access to data means no point in proceeding. You might be able to get what you need online, though\u200a\u2014\u200athere\u2019s a rising trend in making data available for free (for example\u00a0<a href=\"http:\/\/bit.ly\/gcp-publicdata\" target=\"_blank\" rel=\"noreferrer noopener\">here<\/a>).<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-03f9832 elementor-widget elementor-widget-text-editor\" data-id=\"03f9832\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>It still has to be relevant, though. You know you can\u2019t use\u00a0<em>my<\/em>\u00a0<a href=\"http:\/\/bit.ly\/marmitebean\" target=\"_blank\" rel=\"noreferrer noopener\">Marmite<\/a>\u00a0consumption patterns (define: big data) to predict\u00a0<em>your<\/em>\u00a0blood sugar levels. Clearly useless data doesn\u2019t count. You don\u2019t have to analyze the data yet (that\u2019s later in the project) but you should check that you will actually have something to analyze when the time comes. No data means no\u00a0<a href=\"http:\/\/bit.ly\/quaesita_ai\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">ML\/AI<\/a>.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:quote -->\n<blockquote class=\"wp-block-quote\">\n<p>No access to relevant data or no computers to crunch it? There\u2019s no nice way to say\u00a0this\u2026<\/p>\n<\/blockquote>\n<!-- \/wp:quote -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0d191f1 elementor-widget elementor-widget-image\" data-id=\"0d191f1\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/hackernoon.com\/hn-images\/0*UL6LhHAmpo_pBRty\" alt=\"\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-38d510a elementor-widget elementor-widget-text-editor\" data-id=\"38d510a\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>You\u2019ll also need to verify that you have the computing power to process your data. (Have you heard of\u00a0<a href=\"http:\/\/bit.ly\/gcp-hello\" target=\"_blank\" rel=\"noreferrer noopener\">my employer<\/a>? They have plenty and they\u00a0<a href=\"http:\/\/bit.ly\/gcloudpstart\" target=\"_blank\" rel=\"noreferrer noopener\">like to share<\/a>. Just sayin\u2019.)<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e66914b elementor-widget elementor-widget-heading\" data-id=\"e66914b\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Reality checklist<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-422dbb3 elementor-widget elementor-widget-text-editor\" data-id=\"422dbb3\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>Make sure you can answer yes to all of these. Each one of these is likely to get its own guide later, stay tuned. This is just a quick overview of questions for identifying an nonstarter.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:list -->\n<ul>\n<li><strong><em>Appropriate task:<\/em><\/strong>\u00a0Are you automating many decisions\/labels? Where you can\u2019t just look the answer up perfectly each time?<\/li>\n<li><strong><em>Reasonable expectations:\u00a0<\/em><\/strong>Do you understand that your system might be excellent, but it will not be flawless? Can you live with the occasional mistake?<\/li>\n<li><strong><em>Possible in production:<\/em><\/strong>\u00a0Regardless of where those decisions\/labels come from, will you be able to serve them in production? Can you muster the engineering resources to do it at the scale you\u2019re anticipating? You\u2019ll be looking into this question in more detail when you sit down with engineers, but for now the glimmer of a sanity check will suffice.<\/li>\n<li><strong><em>Data to learn from:\u00a0<\/em><\/strong>Do potentially useful inputs exist? Can you gain access to them? (It\u2019s okay if the data don\u2019t exist yet but you have a plan to get them soon.)<\/li>\n<li><strong><em>Enough examples:\u00a0<\/em><\/strong>When you\u2019re sharing a beverage with your\u00a0<a href=\"http:\/\/bit.ly\/quaesita_roles\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">statistician or machine learning engineer<\/a>\u00a0buddy and you casually mention the number of examples available plus the kind of output you\u2019re aiming at, is their expression free of cringes? (I\u2019ll teach you how to develop this intuition yourself in a future article.)<\/li>\n<li><strong><em>Computers:\u00a0<\/em><\/strong>Do you have access to enough processing power to handle your dataset size? (<a href=\"http:\/\/bit.ly\/gcloudpstart\" target=\"_blank\" rel=\"noreferrer noopener\">Cloud technologies<\/a>\u00a0make this is an automatic yes for anyone who\u2019s open to\u00a0<a href=\"http:\/\/bit.ly\/gcloudpstart\" target=\"_blank\" rel=\"noreferrer noopener\">considering using them<\/a>.)<\/li>\n<li><strong><em>Team:\u00a0<\/em><\/strong>Are you confident you can assemble a team with the\u00a0<a href=\"http:\/\/bit.ly\/quaesita_roles\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">necessary skills<\/a>?<\/li>\n<li><strong><em>Ground truth:\u00a0<\/em><\/strong>Unless you\u2019re after\u00a0<a href=\"http:\/\/bit.ly\/quaesita_unsupervised\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">unsupervised learning<\/a>, do you have access to outputs? If not, can you pay humans to make them for you by performing the task?<\/li>\n<li><strong><em>Logging sanity:\u00a0<\/em><\/strong>It\u2019s possible to tell which input goes with which output, right?<\/li>\n<li><strong><em>Logging quality:<\/em>\u00a0<\/strong>Do you trust that the data actually is what its purveyors claim it is? (To learn from examples, you need good examples to learn from.)<\/li>\n<\/ul>\n<!-- \/wp:list -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c58782c elementor-widget elementor-widget-heading\" data-id=\"c58782c\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Assemble your\u00a0team<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e59c22c elementor-widget elementor-widget-text-editor\" data-id=\"e59c22c\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>Once you\u2019ve cleared the reality checklist, it\u2019s time to start recruiting, which you can do in parallel with wading through the rest of this guide. My advice on the roles you\u2019re looking for is\u00a0<a href=\"http:\/\/bit.ly\/quaesita_roles\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">here<\/a>.<\/p>\n<!-- \/wp:paragraph -->\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-aa9b5f1 elementor-widget elementor-widget-image\" data-id=\"aa9b5f1\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/hackernoon.com\/hn-images\/0*cT43WkymE4rsoqn8\" alt=\"\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b8ce7b6 elementor-widget elementor-widget-heading\" data-id=\"b8ce7b6\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">\n<!-- wp:heading {\"level\":3} -->\n<h3>Craft a performance metric\u00a0wisely<\/h3>\n<!-- \/wp:heading -->\n<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2573bd2 elementor-widget elementor-widget-heading\" data-id=\"2573bd2\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Own the tradeoffs<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-cb55f4a elementor-widget elementor-widget-text-editor\" data-id=\"cb55f4a\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>The next bit can get a bit tricky if you\u2019re new to it. You\u2019re in charge of deciding how much each kind of outcome is worth. Is the disgusting bottle of wine that got a Y twice as bad as the delicious bottle we missed out on? 3.4823 times? Up to you!<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>Struggling? Grab someone who likes numbers and have them help you brainstorm.\u00a0<a href=\"http:\/\/bit.ly\/quaesita_roles\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">Qualitative experts<\/a>\u00a0are specially trained for this, but your standard calculator-slinger will do in a pinch. If you want the best helper, utter the formal jargon for it (<em>eliciting indifference curves<\/em>) out loud in a pentagram to summon an economist.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:quote -->\n<blockquote class=\"wp-block-quote\">\n<p>Economists make surprisingly useful advisors for AI projects.<\/p>\n<\/blockquote>\n<!-- \/wp:quote -->\n\n<!-- wp:paragraph -->\n<p>Now that you\u2019ve figured out how to trade off various results on\u00a0<strong>one<\/strong>\u00a0single output, it\u2019s time to think about how you\u2019d like to score a few thousand of them at once. Using the average is an average choice, but you needn\u2019t be average. Again, decision-maker is boss here. The right way to score it depends on what\u2019s right for your business.<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ba5de4f elementor-widget elementor-widget-heading\" data-id=\"ba5de4f\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\"> Expert mode is heavy on simulation<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-cacb054 elementor-widget elementor-widget-text-editor\" data-id=\"cacb054\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>Tricky, complex projects benefit tremendously from\u00a0<em>simulation<\/em>. That\u2019s where an\u00a0<a href=\"http:\/\/bit.ly\/quaesita_analysts\" target=\"_blank\" rel=\"noreferrer noopener\">analyst<\/a>\u00a0specializing in generating fake but plausible data can help you see the potential consequences of the choices you\u2019re making in this article.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:quote -->\n<blockquote class=\"wp-block-quote\">\n<p>A dress rehearsal helps opening night go smoothly.<\/p>\n<\/blockquote>\n<!-- \/wp:quote -->\n\n<!-- wp:paragraph -->\n<p>Simulation gives you a dress rehearsal, which helps you get a bunch of kinks ironed out before you have to start your project for-realsies. Like\u00a0<a href=\"http:\/\/bit.ly\/quaesita_analysts\" target=\"_blank\" rel=\"noreferrer noopener\">analytics<\/a>, it takes some of the burden off the decision-maker\u2019s mandate to\u00a0<em>meditate hard and think of everything.<\/em><\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-741d100 elementor-widget elementor-widget-heading\" data-id=\"741d100\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Make your\u00a0metric<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-00ec497 elementor-widget elementor-widget-text-editor\" data-id=\"00ec497\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>There are many different ways to make a metric. In our wine example, you could go for the really simple one:\u00a0<strong>accuracy\u00a0<\/strong>a.k.a.\u00a0<em>\u201cDon\u2019t be wrong.\u201d<\/em>\u00a0Every mistake is equally bad (0), every correct response is equally good (1), then you take an average (which you\u2019ve been itching to take\u200a\u2014\u200asomeone needs to put a\u00a0<a href=\"http:\/\/bit.ly\/clt_wiki\" target=\"_blank\" rel=\"noreferrer noopener\">central limit<\/a>\u00a0on how appealing people find averages).<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>Hang on, maybe instead you have\u00a0<a href=\"https:\/\/en.wikipedia.org\/wiki\/Fear_of_missing_out\" target=\"_blank\" rel=\"noreferrer noopener\">FOMO<\/a>\u00a0when it comes to good wines and you\u2019re quite happy to have duds along the way? Well, that\u2019s a different metric called\u00a0<strong>recall<\/strong>. Or maybe you don\u2019t want to have your money wasted; you\u2019re in a tight budget. If you\u2019re a starving graduate student (I\u2019ve been there) and you need to be sure that when the systems says Yummy you won\u2019t be wasting your money, but you\u2019re okay with missing out on beautiful bottles. That\u2019s an entirely different metric called\u00a0<strong>precision<\/strong>. We\u2019ll slow down and deep-dive on metrics in another post. For now, nevermind what they\u2019re called, just make one up that reflects what\u2019s right for your business.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>Need help? Has your economist wandered off already? No problem! Perhaps you have a friend who loves designing games? Games geeks have unknowingly been\u00a0<a href=\"http:\/\/bit.ly\/quaesita_emperor\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">training for this<\/a>\u00a0their entire lives! If you don\u2019t hang with that crowd, you can call your\u00a0<a href=\"http:\/\/bit.ly\/quaesita_roles\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">qualitative expert<\/a>\u00a0instead, since it\u2019s their job to help decision-makers clarify their thoughts on this kind of thing.<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<section class=\"has_eae_slider elementor-section elementor-top-section elementor-element elementor-element-8de8202 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"8de8202\" data-element_type=\"section\" data-e-type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"has_eae_slider elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-f1d478a\" data-id=\"f1d478a\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-1811ce4 elementor-widget elementor-widget-heading\" data-id=\"1811ce4\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Ask for an expert\u00a0review<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-994ebf0 elementor-widget elementor-widget-text-editor\" data-id=\"994ebf0\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>In applications where human wellbeing\u00a0is substantially\u00a0on\u00a0the\u00a0line, seek a consultation with a panel of experts to verify that it isn\u2019t possible to get a high score on your metric in some perverse and harmful way.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>Which experts? Have you heard the one where a decision-maker, an\u00a0<a href=\"http:\/\/bit.ly\/merethics\" target=\"_blank\" rel=\"noreferrer noopener\">ethicist<\/a>, an AI control theorist, a statistician, a user experience researcher, a behavioral economist, a domain expert, and a\u00a0<a href=\"http:\/\/bit.ly\/sre_ch1\" target=\"_blank\" rel=\"noreferrer noopener\">reliability engineer<\/a>\u00a0walk into a bar\u2026?<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>Sure, this can be overkill on benign business applications, so your\u00a0<a href=\"http:\/\/bit.ly\/quaesita_roles\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">qualitative expert<\/a>\u2019s introductory-level familiarity with each of these disciplines has you covered and your game designer friend\u2019s instincts are also tapping into a similar vein.<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4c7ab79 elementor-widget elementor-widget-heading\" data-id=\"4c7ab79\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Hello, business performance metric!<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-21d355e elementor-widget elementor-widget-text-editor\" data-id=\"21d355e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>When you\u2019re done, you\u2019ll hear an angelic chorus. You\u2019ve created your business performance metric!<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-54e5a47 elementor-widget elementor-widget-image\" data-id=\"54e5a47\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/hackernoon.com\/hn-images\/0*oEpRW2wqLTC6K0e6.jpg\" alt=\"\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-631a199 elementor-widget elementor-widget-text-editor\" data-id=\"631a199\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>This is not the same thing as a\u00a0<a href=\"http:\/\/bit.ly\/quaesita_emperor\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">loss function<\/a>\u00a0(we\u2019ll talk about those later). When it comes to metrics, the possibilities are endless and it is up to the decision-maker to figure out what\u2019s actually important. In case you\u2019re feeling anxious about tackling this, I\u2019m brewing a few more articles to help you master metric development.<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-90ac9a8 elementor-widget elementor-widget-heading\" data-id=\"90ac9a8\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">(Jargon trigger warning) Something your AI experts should\u00a0know<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4f36737 elementor-widget elementor-widget-text-editor\" data-id=\"4f36737\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>Here\u2019s a nuanced thing you can skip until we dig into it in a later post, but if you know what a\u00a0<a href=\"http:\/\/bit.ly\/quaesita_emperor\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">loss function<\/a>\u00a0is then you\u2019ll realize we\u2019ll have\u00a0<em>two<\/em>\u00a0metrics in play. If that means nothing to you, let\u2019s not worry about it for now, your job here is only to ensure your ML\/AI experts read the next paragraph. Many of them missed this lesson in school.\u00a0<strong><em>Warning:<\/em><\/strong>\u00a0many decision-makers will find that it reads like a jargon-gibberish nightmare\u200a\u2014\u200amy apologies!\u200a\u2014\u200aso just forward it along.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:quote -->\n<blockquote class=\"wp-block-quote\">\n<p>The loss function is for optimization, not for\u00a0testing.<\/p>\n<\/blockquote>\n<!-- \/wp:quote -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-95b1049 elementor-widget elementor-widget-text-editor\" data-id=\"95b1049\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>\u201cIn applied ML\/AI, the\u00a0<a href=\"http:\/\/bit.ly\/quaesita_emperor\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">loss function<\/a>\u00a0is for optimization, not for statistical testing.\u00a0<a href=\"http:\/\/bit.ly\/quaesita_statistics\" target=\"_blank\" rel=\"noreferrer noopener\">Statistical testing<\/a>\u00a0should ask,\u00a0<em>\u201cDoes it perform well enough to build\/launch?\u201d<\/em>\u00a0where\u00a0<em>perform<\/em>\u00a0should be defined by the business problem and its owner. You\u2019re not supposed to alter the business problem statement to suit your convex optimization ambitions. For expediency, you\u2019re free to optimize using a standard loss function that moves in the same direction as the function your leader\u2019s imagination just spawned (perform correlation checks* analytically or with simulation), but please test with\u00a0<em>their<\/em>\u00a0function.** Have you been using loss for all your evaluations? Don\u2019t worry, it\u2019s a common mistake which may have something to do with software defaults, college course format, and decision-maker absenteeism in AI.\u201d<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>** If no standard loss function correlates decently with the performance metric, please alert your decision-maker now that what they\u2019re asking for is very difficult and probably requires investing in optimization researchers.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>** Dissenting AI experts, read\u00a0<a href=\"http:\/\/bit.ly\/quaesita_incomp\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">this<\/a>. Although it\u2019s not about functions, the general ideas about working with decision-makers apply.<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-33a813d elementor-widget elementor-widget-heading\" data-id=\"33a813d\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Set testing\u00a0criteria<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-84219c4 elementor-widget elementor-widget-heading\" data-id=\"84219c4\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Define your population of\u00a0interest<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<section class=\"has_eae_slider elementor-section elementor-top-section elementor-element elementor-element-8e20772 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"8e20772\" data-element_type=\"section\" data-e-type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"has_eae_slider elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-a5c6d75\" data-id=\"a5c6d75\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-354291c elementor-widget elementor-widget-text-editor\" data-id=\"354291c\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>Talking about your system \u201cworking\u201d makes no sense until you specify which\u00a0<a href=\"http:\/\/bit.ly\/quaesita_slkid\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">instances<\/a>\u00a0you\u2019re intending for it to work on. All US summer inputs? All inputs globally? (These are different!)<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>Before you proceed, you\u2019ll need to define your statistical\u00a0<a href=\"http:\/\/bit.ly\/quaesita_popwrong\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">population of interest<\/a>, the broad collection of\u00a0<a href=\"http:\/\/bit.ly\/quaesita_slkid\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">instances<\/a>\u00a0your system needs to demonstrate good performance on before you give it the green light. I\u2019ve got a\u00a0<a href=\"http:\/\/bit.ly\/quaesita_savvy\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">two-part guide<\/a>\u00a0for you in case you need a refresher on that.<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-27a8cb0 elementor-widget elementor-widget-heading\" data-id=\"27a8cb0\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Commit to crushing\u00a0it!<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-565fd28 elementor-widget elementor-widget-text-editor\" data-id=\"565fd28\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>Now that you have your performance metric and population handy, you have one more job to do before you can go put your feet up. Since getting here can take months, your poor feet must be pretty sore.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>Here\u2019s the last task: decide on the minimum performance you\u2019re willing to sign off on, because I\u2019m about to make you promise that you\u2019re not going to let this system take over labeling stuff for you unless it\u2019s good enough.<\/p>\n<!-- \/wp:paragraph -->\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-aaf997b elementor-widget elementor-widget-image\" data-id=\"aaf997b\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/hackernoon.com\/hn-images\/0*uhrqMZ1mcGERnClD\" alt=\"\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-96e5856 elementor-widget elementor-widget-text-editor\" data-id=\"96e5856\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>What does \u201cgood enough\u201d mean? How high should you set the bar? Up to you, but you\u00a0<em>must\u00a0<\/em>commit now.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:quote -->\n<blockquote class=\"wp-block-quote\">\n<p>Setting criteria up front is part of how you keep yourself (and me) safe from horrible machine learning and\u00a0AI.<\/p>\n<\/blockquote>\n<!-- \/wp:quote -->\n\n<!-- wp:paragraph -->\n<p>This criterion is not a guiding star. You can also have a performance level that you tell your team to reach for (purrrrfection?), but that\u2019s not what you will test against. Test against the bare minimum.<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1e354bb elementor-widget elementor-widget-heading\" data-id=\"1e354bb\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">You biased species&nbsp;you<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-154767e elementor-widget elementor-widget-text-editor\" data-id=\"154767e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>Why are you coming up with this cutoff\u00a0<em>now<\/em>, before you\u2019ve even assembled a team for this project? Turns out that as members of the human species, we suffer from a few lovely cognitive biases (don\u2019t say\u00a0<em>sunk cost<\/em>\u00a0or\u00a0<em>endowment effect<\/em>\u00a0or\u00a0<em>confirmation bias<\/em>\u00a0unless you want that economist to appear out of nowhere again, this time with a psychologist in tow) which boil down to this: when humans invest time and effort into something, we fall in love with what we have made\u2026 even if it is a pile of poisonous rubbish. Then we find ourselves bargaining:\u00a0<em>\u201cAwww but the performance is not so bad. I\u2019m kind of proud of 12% accuracy. Maybe we could launch it anyway? Why don\u2019t I test against 10%? See? It passes. It\u2019s\u00a0<\/em><a href=\"http:\/\/bit.ly\/quaesita_statistics\" target=\"_blank\" rel=\"noreferrer noopener\"><em>statistically significantly<\/em><\/a><em>\u00a0good enough.\u201d<\/em><\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>If you want to\u00a0<a href=\"http:\/\/bit.ly\/quaesita_inspired\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">wallow in this sad topic with me for another 6 min<\/a>, who am I to stop you?<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:quote -->\n<blockquote class=\"wp-block-quote\">\n<p>We humans fall in love with what we have poured effort into\u2026 even if it is a pile of poisonous rubbish.<\/p>\n<\/blockquote>\n<!-- \/wp:quote -->\n\n<!-- wp:paragraph -->\n<p>While we are still sober(!) and before we\u2019ve poured much of ourselves into it, we\u2019re going to take a cold, hard look at the business problem and say,<em>\u00a0\u201cIf it does not meet<\/em><strong><em>\u00a0this minimum<\/em><\/strong><em>\u00a0requirement, I\u00a0<\/em>promise<em>\u00a0that it DIES.\u201d<\/em><\/p>\n<!-- \/wp:paragraph -->\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-182d896 elementor-widget elementor-widget-image\" data-id=\"182d896\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/hackernoon.com\/hn-images\/0*NFKzOeaItraDLR73.jpg\" alt=\"\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-12ed70e elementor-widget elementor-widget-heading\" data-id=\"12ed70e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Better than\u00a0human?<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4148120 elementor-widget elementor-widget-text-editor\" data-id=\"4148120\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>I hope you have an inkling now as to why I\u2019m forced to suppress a chuckle when people ask me,\u00a0<em>\u201cIs AI better than humans at stuff?\u201d<\/em><\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:quote -->\n<blockquote class=\"wp-block-quote\">\n<p>Better than human? Tautologies are tautological.<\/p>\n<\/blockquote>\n<!-- \/wp:quote -->\n\n<!-- wp:paragraph -->\n<p>If its maker required it to be better than human and did the testing properly, then if it wasn\u2019t better than human\u2026 they crushed it. If it exists, then yes. Unless they didn\u2019t require it to be better than human, in which case, it probably isn\u2019t. Why are people asking\u00a0<em>me<\/em>? Ask those decision-makers how they set up their criteria and testing. (Speaking of\u00a0<a href=\"http:\/\/bit.ly\/quaesita_incomp\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">populations<\/a>,\u00a0<em>which\u00a0<\/em>humans is it supposed to be better than? That one volunteer?)<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0535c6a elementor-widget elementor-widget-image\" data-id=\"0535c6a\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/hackernoon.com\/hn-images\/0*lQB9P2hRa3Gq5ib1.jpg\" alt=\"\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f45eff1 elementor-widget elementor-widget-text-editor\" data-id=\"f45eff1\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>Also, let\u2019s not obsess so much about whether machines are better than us at stuff. My computer has always been better than me\u2026 at performing multiplication. This bothers me not at all. My bucket is better than me at holding water. What\u2019s the point of a tool if it doesn\u2019t reduce your effort or increase what you can achieve?<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>Instead, focus on whether it\u2019s good enough to be useful.<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c4041c5 elementor-widget elementor-widget-heading\" data-id=\"c4041c5\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Don\u2019t be too demanding<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-37bf2a0 elementor-widget elementor-widget-text-editor\" data-id=\"37bf2a0\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>Always requiring better-than-human performance might make you miss out on profit. It\u2019s a bit like saying that you\u2019ll only hire an Olympic gold medalist to lay bricks for you. Maybe an Olympian is stronger than your\u00a0<a href=\"http:\/\/bit.ly\/clt_wiki\" target=\"_blank\" rel=\"noreferrer noopener\">average<\/a>\u00a0Joe, but having such a strict hiring bar might leave you with no workers at all. Lower your standards to where they make good business sense, but no lower. Setting the testing bar at your bare minimum is\u00a0<em>incentive-compatible,<\/em>\u00a0as economists might say. (If you\u2019re fresh out of B-school and want to nerd out on mechanism design, we\u2019re essentially setting up a\u00a0<a href=\"http:\/\/bit.ly\/wiki_bdm\" target=\"_blank\" rel=\"noreferrer noopener\">BDM auction<\/a>\u00a0for our hypothesis testing procedure.)<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:quote -->\n<blockquote class=\"wp-block-quote\">\n<p>Don\u2019t miss out on a profitable solution by testing against an excessively high\u00a0bar.<\/p>\n<\/blockquote>\n<!-- \/wp:quote -->\n\n<!-- wp:paragraph -->\n<p>Sometimes automation gives lower per-unit quality than handmade goods, but the scale and speed of the machine makes it worth it for business. Is it worth it for\u00a0<em>your<\/em>\u00a0business? Hey, you\u2019re in charge, not me. Good luck!<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>For a short summary for this long text, see the\u00a0<a href=\"http:\/\/bit.ly\/quaesita_realitycheck\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">checklist version<\/a>.<\/p>\n<!-- \/wp:paragraph -->\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f4f6980 elementor-widget elementor-widget-image\" data-id=\"f4f6980\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/hackernoon.com\/hn-images\/0*e19L8qkoXF5UxxER\" alt=\"\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-9582380 elementor-widget elementor-widget-text-editor\" data-id=\"9582380\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<!-- wp:paragraph -->\n<p>That\u2019s step one of\u00a0<a href=\"http:\/\/bit.ly\/quaesita_ai\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">ML\/AI<\/a>\u00a0done! Step two involves data and hardware (and engineers!) so you might like to\u00a0<a href=\"http:\/\/bit.ly\/quaesita_slkid\" target=\"_blank\" rel=\"noreferrer noopener\" class=\"broken_link\">brush up on some vocabulary<\/a>\u00a0in anticipation of forthcoming attractions.<\/p>\n<!-- \/wp:paragraph -->\n\n<!-- wp:paragraph -->\n<p>If you found any of the ideas in this guide worthwhile, please tell them to whoever in your network is most likely to find themselves in the decision role. Let\u2019s build a skilled and responsible new crop of AI leaders\u00a0for\u00a0a bright AI\u00a0future!<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>Many teams try to start an applied AI project by diving into algorithms and data before figuring out desired outputs and objectives. Instead, the first step is for you as the owner to form a clear vision of what you want from your ML\/AI system and how you\u2019ll know you\u2019ve trained it successfully.<\/p>\n","protected":false},"author":335,"featured_media":9738,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[183],"tags":[226,639,563,225],"ppma_author":[2050],"class_list":["post-9737","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-ml","tag-ai","tag-ai-project","tag-algorithms","tag-ml"],"authors":[{"term_id":2050,"user_id":335,"is_guest":0,"slug":"cassie-kozyrkov","display_name":"Cassie Kozyrkov","avatar_url":"https:\/\/www.experfy.com\/blog\/wp-content\/uploads\/2020\/04\/medium_df35f80d-2bff-4fe3-b741-a94d51320e00-150x150.jpg","user_url":"https:\/\/careers.google.com\/?src=Online\/LinkedIn\/linkedin_profilepage&amp;utm_source","last_name":"Kozyrkov","first_name":"Cassie","job_title":"","description":"Cassie Kozyrkov is Chief Decision Scientist at Google, Inc. With a unique combination of deep technical expertise, and world-class public-speaking skills, she has provided guidance on more than 100 projects and designed Google's analytics program, personally training over 15000 Googlers in statistics, decision-making, and machine learning.\u00a0"}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/9737","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/335"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=9737"}],"version-history":[{"count":5,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/9737\/revisions"}],"predecessor-version":[{"id":33912,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/9737\/revisions\/33912"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/9738"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=9737"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=9737"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=9737"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=9737"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}