{"id":1085,"date":"2019-02-15T10:31:58","date_gmt":"2019-02-15T10:31:58","guid":{"rendered":"http:\/\/kusuaks7\/?p=690"},"modified":"2023-07-28T14:06:35","modified_gmt":"2023-07-28T14:06:35","slug":"codify-your-workflow","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/codify-your-workflow\/","title":{"rendered":"Codify your workflow"},"content":{"rendered":"

Ready to learn Data Science? Browse courses<\/a>\u00a0like\u00a0Data Science Training and Certification<\/a> developed by industry thought leaders and Experfy in Harvard Innovation Lab.<\/em><\/strong><\/p>\n

\n

Encoding my workflow saves time and effort over the long-run, but it also requires me to be explicit about what my workflow is in the first\u00a0place.<\/h3>\n
<\/canvas><\/figure>\n

I think I\u2019m like most data scientists in that there are certain things I tend to do repeatedly across projects: there are certain patterns I check for, or certain things I plot, or certain relationships I try to understand. Sometimes it\u2019s because I\u2019ve made a mistake often enough that I find it pays to automatically guard against it. Sometimes it\u2019s because I just find that, personally, certain ways of looking at a problem help me wrap my mind around it. In any case, large parts of my workflow for any given project very often look very similar to large parts of my workflows for previous projects. Not surprisingly, I\u2019ve found it useful to reduce the overhead of these repetitive activities.<\/p>\n

While I\u2019ve understood the value of reducing overhead through automation ever since I learned how to code, I never got much advice on how to do it. It\u2019s just something I picked up. In this post, I try to summarize some of the things I\u2019ve learned, and I\u2019ll use some code from a recently-automated part of my workflow as an example.<\/p>\n

Logical beauty<\/h2>\n

Good design advice has been around at least since Vitruvius wrote\u00a0Ten Books on Architecture<\/a>\u00a0(still a worthwhile read), where he said that good design achieves firmness, usefulness, and delight. It\u2019s relatively easy to translate the first two of those ideals to workflow automation design: whatever we build, it should consistently do the job we want it to do at the scale, speed, and quality that we need. But I think it\u2019s the last ideal \u2014 delight \u2014 that most often makes the difference between a workable design and a good design.\u00a0Fred Brooks argues<\/a>\u00a0that delight, in the context of technical design, should be thought of as \u201clogical beauty\u201d, and argues that\u00a0a logically beautiful product is one that minimizes mental effort<\/em><\/strong>.<\/p>\n

I think minimization of mental effort is an interesting goal for a data science workflow, since many recent embarrassments in the data industry\u00a0seem to have arisen<\/a>\u00a0from practitioners not expending enough mental effort thinking about the potential consequences of their design decisions.\u00a0So we need to carefully select which areas of mental effort to minimize<\/em><\/strong>. That means the first step of a good design is, no matter how trite it sounds, to decide what we want to accomplish.<\/p>\n

Think about driving a car. I drive a car to go from where I\u2019m going as efficiently as possible. A well-designed car should allow me to reserve as much of my attention as possible for those things most directly related to accomplishing my purpose. That means minimizing the mental effort on everything else. In other words, while I\u2019m driving, I should be able to focus on making my turn at the right place, on avoiding things like pedestrians or lampposts, and things like that. I should not be focused on making sure the engine runs, or on shifting gears, or on how the brakes work. If I used the car for another purpose I might focus on those things, but they are not\u00a0proper<\/em>\u00a0to the accomplishment of the particular purpose I have chosen. A good design shouldn\u2019t distract me or prevent me from paying attention to the things that could prevent from from accomplishing my purpose.<\/p>\n

Propriety<\/em>\u00a0is a somewhat old-fashioned way of talking about relevance. Brooks defines propriety in design as \u201cdo not introduce what is immaterial.\u201d I\u2019ve found it useful to think about propriety in terms of the tradeoffs I used in the analogy. If doing something ensures that I am not prevented from accomplishing my purpose, it is proper to that purpose. If doing something just facilitates the the accomplishment of the purpose, it\u2019s not proper to that purpose. Failing to avoiding pedestrians or lampposts will necessarily prevent me from getting where I want to go. That is true if I travel by automobile, foot, skateboard, bike, airplane, or train. Failing to have a combustion engine won\u2019t necessarily prevent me from getting where I want to go; therefore, it\u2019s not proper to the design, no matter how convenient it is a matter of implementation. So after we\u2019ve defined our purpose, define what\u2019s proper to that purpose: as a rule of thumb, if it can move us forward, it may or may not be proper; if it can keep us from moving forward, it\u2019s proper.\u00a0Good design minimizes the mental effort expended on tasks that are not proper to our intended purpose<\/em><\/strong>.<\/em><\/p>\n

Orthogonality<\/em>\u00a0is another component of logical beauty. Brooks summarizes this as \u201cdo not link what is independent\u201d. In other words: my ability to avoid pedestrians and lamp posts partially depends upon being able to see those things, and partially upon being able to maneuver the car to avoid them. A windshield\u2019s functionality and a steering wheel\u2019s functionality should be independent of one another. A design that failed to maintain that independence would be a poor design. So after we define the proper components of our workflow, we minimize dependencies.\u00a0Good design keeps proper components separate from components that aren\u2019t proper so we don\u2019t accidentally minimize the mental effort we expend on the the things that actually need our full attention. Likewise, good design keeps non-proper components from each other so we don\u2019t accidentally minimize mental effort in ways we didn\u2019t anticipate.<\/em><\/strong><\/p>\n

Generality<\/em>\u00a0is the last principle I\u2019ll talk about before moving on to the example. Brooks summarizes it as \u201cdo not restrict what is inherent.\u201d In other words, a tire (or engine or wiring, etc.) that fits one car shouldn\u2019t be designed to fit\u00a0only<\/em>that car without a very good reason.\u00a0Good design demands a reason before restricting the use cases to which a piece of functionality can apply.\u00a0<\/em><\/strong>This makes it easier to accommodate unanticipated needs without requiring us to rebuild the whole product.<\/p>\n

So, to summarize:<\/p>\n

    \n
  1. Decide what purpose we want to accomplish.<\/li>\n
  2. Identify which tasks are proper to the accomplishment of that purpose.<\/li>\n
  3. Minimize dependencies by separating proper tasks from non-proper tasks, and non-proper tasks from one another.<\/li>\n
  4. Plan each task to be as general-purpose as possible while still maintaining propriety or orthogonality.<\/li>\n<\/ol>\n

    Designs that follow the above principles tend to be logically beautiful, in that they minimize our mental effort in all the right places.<\/p>\n

    A minimalist example of workflow\u00a0design<\/h2>\n

    I work a lot with geolocation data, and I often have the need to plot that data. Sometimes I need to plot shapes, and other times I need points. Sometimes I need to see those objects against the background of a satellite image to get some rough context, sometimes I need the more detailed context of roads or labels. I normally have to use some form of this workflow several times a week.\u00a0Sometimes I have to go through it hundreds of times a day<\/a>. At any rate, I have to do it often enough that it\u2019s a good candidate for minimizing mental effort. So I only have one purpose: graphically represent longitude and latitude pairs on a picture that represents the real-world context of those coordinates. That purpose defines propriety: anything that doesn\u2019t actually put shapes on the picture is not proper to my purpose, and therefore mental effort expended on that thing should be minimized. In the case of the current example, defining purpose and propriety were relatively easy. That\u2019s not always the case.<\/p>\n

    Now I need to worry about orthogonality. Imagine speaking to someone with absolutely no technical expertise. I usually picture either one of my earliest managers described everything involving code as \u201cnew-fangled\u201d, or my mother who still refuses to get an email address, or a college freshman majoring in the humanities. You get the picture. My task creates a set of questions that my imaginary novice would see as being both necessary and sufficient to create my desired outcome. If my imaginary novice can say \u201cbut why do you need the answer to X?\u201d, that means I\u2019ve included an unnecessary question. If my imaginary novice can say \u201c but I don\u2019t understand how the answers to X, Y, and Z will lead to the outcome\u201d, that means I\u2019m either missing a question or haven\u2019t asked the right question. The reason I like to use an imaginary novice as a sounding board is because I tend to get too technical too fast when I try to design everything. Because I know how to implement things, my brain jumps to implementation questions sooner than it should. If you have a real flesh-and-blood novice who is willing and able to be your sounding board, use that person instead of an imaginary one.<\/p>\n

    If my goal is to get shapes on a picture, then I have only three questions:<\/p>\n