{"id":24529,"date":"2021-06-02T18:47:27","date_gmt":"2021-06-02T18:47:27","guid":{"rendered":"https:\/\/www.experfy.com\/blog\/?p=24529"},"modified":"2023-08-19T12:37:37","modified_gmt":"2023-08-19T12:37:37","slug":"managing-the-big-data-project-lifecycle-approach-team-composition-pitfalls","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/managing-the-big-data-project-lifecycle-approach-team-composition-pitfalls\/","title":{"rendered":"Managing The Big Data Project \u2013 Lifecycle, Approach, Team Composition, Pitfalls"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Managing the Big Data project<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Everything \u2018expands\u2019 in a Big Data project. There are many more decision points, even before you draw the first entity in your ERD design tool. A typical IT lifecycle, of any type, consists of\u00a0<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
    \n
  1. Analysis (\u201cYou start coding and I\u2019ll go upstairs and see what the business wants\u201d);\u00a0<\/li>\n
  2. Design (\u201cIs this a pattern that has previously been designed?\u201d)<\/li>\n
  3. Coding (\u201cIs this a pattern that has previously been coded?\u201d)<\/li>\n
  4. Testing (\u201cHave we used tests like these, before?\u201d)<\/li>\n
  5. Implementation\u00a0<\/li>\n
  6. Feedback (all positive, right?)<\/li>\n
  7. All of the above happens in Agile development as well as in Waterfall development, but for this discussion, I\u2019ll ignore the details of Agile work sessions.<\/li>\n<\/ol>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t

    Let\u2019s dive in the detailed lifecycle about Big Data projects:<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t

    First, Analysis and Design\u2026<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    Analysis<\/strong> in a Big Data project includes not only the exploration and documentation of what the (Mathematical) Analysts\/ Advanced Business People want and need but also the nature or situation of the data upon which they are going to work.\u00a0<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    You will need to answer: What is or will be estimates of the (growing) size of the data? Will the application<\/a> continue to exponentially collect data; if so, how will it be treated, where will it be stored, and how do the Analysts\/Business People plan to use it? You will need to have conversations, with a \u201cchalkboard\u201d for drawing samples and examples. For example: Suppose we are collecting data, which is growing exponentially over an 18-month period (pick any time frame). Is it important to apply the Analysts\u2019 calculations over the entire 18 months of data, or can we select a sample\u2026; say a time slice of data from a particular 3 month period ?:<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    Additionally, can an amount of data be purged after a certain time period? Can a particular type of data be purged? Do the Managers understand cloud costs of exabyte volumes of data? Does some\/all of the volume of data need to be saved for either historical and\/or regulatory purposes? If we need to save all of the data, can we compress some of it by saving calculation results or trend results?\u00a0<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    You can see from the above questions, that a comprehensive understanding of the source(s), use(s), and short-\/medium-\/long-term disposition of the data must be understood and documented in the Analysis Phase.<\/strong><\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    This is necessary to not only document the requirements but also to be able to estimate project cost, length\/duration of project phases, and the staffing that the project will require. Thus, going forward, if there are departures from this understanding, you have the original documentation which can be acknowledged and amended with Management approval.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    The deliverable of Analysis is a BRD \u2013 Business Requirements Document. <\/strong>At the end of Analysis, it is recommended that a Conceptual Model of the project (block diagrams and\/or subject area circles) for Data and for Processing be included in the BRD.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    Design time and questions also expand in a Big Data project. Design can be addressed in 3 concurrent sections \u2013 Data, Operational Code and Interfacing \u2013 BUT the Data design must itself be addressed in 3 subsections:\u00a0<\/strong><\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t
      \n
    1. What are the business\u2019s\/Analyst\u2019s initial requirements (i.e. What problems do they intend to solve?) This will give us the shape of the initial data model(s).<\/li>\n
    2. What will the requirements\/needs be 1 to 2 years into the operation of the system?<\/li>\n
    3. What will the requirements\/formation of the data be towards the end of this system\u2019s life? That is, if this database or data structure is growing at an exponential rate, at what point (given current technology) does it become unmanageable and require mitigation or a new design?<\/li>\n<\/ol>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t

      Now, the design itself.(depending on the requirements) may be OLTP, a Data Vault, Data Warehouse(s), Data Marts, or practically speaking, a combination of two or more of these underlying designs. You must take one or many purpose(s) of the desired system and select the best design to facilitate processing the Big Data project. I could write buckets on selecting an appropriate design, and also much about not reinventing the wheel, but maybe on another blog topic. Look at the 3 scenarios above. You may recommend different designs for each one, but the usual experience is that an initial design is drawn, and then in subsequent years, the initial design is augmented to support items 2 and 3 above.\u00a0<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t

      For both Design and Coding, please don\u2019t reinvent the wheel<\/strong>. From your experience in previous projects, and from the Internet and a library\u2019s worth of published models there are models and patterns that can be incorporated into your new model. Use them and save some time. For example, in the business world there is essentially one model: \u201cCustomer purchases Product.\u201d Of course there are all sorts of extensions to this, but how many models of \u2018Customer\u2019 and \u2018Product\u2019 are there in the public domain. Maybe 50? Pick the ones which match your requirements and use them.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t

      The deliverable of the Design phase is Design Documentation<\/strong>, which includes the Data Logical Design (at a minimum), an outline of Operational Code and designs for all necessary interfaces \u2013 from source(s) to end user tools. The Design Documentation of the data piece should include the conceptual and detailed Data architecture, ERDs or equivalent detailed models of the data, Data Dictionaries, Data Lineage, and any related information to help the Coding Team produce Physical Data Models and DDL which will satisfy the requirements established in Analysis.\u00a0<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t

      A word about Testing<\/strong> \u2013 The easiest way to develop tests and test plans is to develop and document them in your DESIGN phase\u2026or at least towards the end of your design phase, when you know what the system is going to look like. This is also a good time to estimate what\/which data will be used to test and how long testing will take.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

      \n\t\t\t\t\t\t
      \n\t\t\t\t\t
      \n\t\t\t
      \n\t\t\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t

      Approach and Team Composition in the Big Data project<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
      \n\t\t\t\t\t\t
      \n\t\t\t\t\t
      \n\t\t\t
      \n\t\t\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t

      Identify all of the project team members in advance and get commitment for their time with both the person and their management stream. You will need: an Enterprise Architect, Analysts, Data Modelers, DBAs, Testing Specialists, Interface Engineers, Implementation Engineers, Training Personnel<\/a>, and Technical and\/or Business and Management review personnel at each level. Forgetting or ignoring one of these specialties could put your project in trouble.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t

      Don\u2019t wait until the end of a phase of work to start planning the next phase. The experienced Project Manager will plan overlapping phases of work \u2013 sort of a combo between Agile and Waterfall. A certain amount of Logical\/Physical database design.\u00a0<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

      \n\t\t\t\t\t\t
      \n\t\t\t\t\t
      \n\t\t\t
      \n\t\t\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t

      Pitfalls<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
      \n\t\t\t\t\t\t
      \n\t\t\t\t\t
      \n\t\t\t
      \n\t\t\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t
        \n
      • Little or no CIO backing \u2013 You didn\u2019t go high enough in the chain or else the CIO is not really committed to the concept and implementation<\/li>\n
      • Middle management distracted with multiple projects<\/li>\n
      • Insufficient training for staff<\/li>\n
      • Project Manager (PM) with little or no IT familiarity<\/li>\n<\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
        \n\t\t\t\t
        \n\t\t\t\t\t\t\t\t\t

        There is a tendency, especially among large corporations, to establish a separate department called \u201cProject Management.\u201d This department is staffed with people who are highly trained in leadership, negotiation, scheduling, planning, monitoring of deadlines, PM software, etc. Some PMs have a coding background; some have a testing background. Personally, I have never met one with database development experience. Many do not even know the business very well.\u00a0<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

        \n\t\t\t\t
        \n\t\t\t\t\t\t\t\t\t

        Because complexity can be a factor in\u00a0 Big Data projects, it is essential that the PM be a person who is exceptionally experienced in the business that is being serviced, and also very experienced in the development lifecycles of databases, operational code, APIs, BI, testing, and possibly AI. It will not have to have a PM produce a schedule which, for example, covers every task, but is missing the conceptual, logical and physical modeling of the data. I know it\u2019s hard to find people with these qualifications\u2026too bad. Keep looking. There might even be one in-house who has been overlooked for years while they have quietly gathered credentials.\u00a0<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

        \n\t\t\t\t
        \n\t\t\t\t\t\t\t\t\t
          \n
        • Poor understanding of the scope of a Big Data project even after you assemble the team\u2013 It is essential to have presentations of the concepts at all levels of design. In other words, more presentations, walk-thru’s, and trial examples for people to work through as a team.<\/li>\n<\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

          Managing the Big Data project Everything \u2018expands\u2019 in a Big Data project. There are many more decision points, even before you draw the first entity in your ERD design tool. A typical IT lifecycle, of any type, consists of\u00a0 Analysis (\u201cYou start coding and I\u2019ll go upstairs and see what the business wants\u201d);\u00a0 Design (\u201cIs<\/p>\n","protected":false},"author":1160,"featured_media":24533,"comment_status":"open","ping_status":"open","sticky":false,"template":"single-post-2.php","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[187],"tags":[122,95,1602,1601,432],"ppma_author":[3669],"class_list":["post-24529","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bigdata-cloud","tag-big-data","tag-big-data-amp-technology","tag-big-data-analysis","tag-big-data-project","tag-software-testing"],"authors":[{"term_id":3669,"user_id":1160,"is_guest":0,"slug":"claire-frankel","display_name":"Claire Frankel","avatar_url":"https:\/\/www.experfy.com\/blog\/wp-content\/uploads\/2021\/06\/clfpicdownload-150x150.png","user_url":"","last_name":"Frankel","first_name":"Claire","job_title":"","description":"Claire L Frankel holds a B.S. in Physics\/Mathematics from the State University of New York at Albany. She has been working as a Data Architect since 1985 through several generations of technology, databases and data tools. Her work is primarily for Financial Services firms. She is the author of the chapter on Electronic Messaging Standards in \u201cThe Handbook of Investment Technology,\u201d several White Papers on FAS-157 and SEC144A, and an upcoming blog on Big Data. Ms. Frankel is a member of DAMA, the Data Administrator\u2019s Management Association, the EDM Council (Enterprise Data Management) and WSTA (Wall Street Technology Association.)"}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/24529","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/1160"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=24529"}],"version-history":[{"count":10,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/24529\/revisions"}],"predecessor-version":[{"id":30672,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/24529\/revisions\/30672"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/24533"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=24529"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=24529"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=24529"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=24529"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}