• Educational Analytics
  • Experfy Editor
  • JUN 14, 2014

Exploratory Data Analysis and Big Data Enabled Research

In the big data era, the power of science lies in technology-enabled data discovery

Big data’s biggest challenge, as most will agree, is not data volume. It is the velocity of data flow in applications requiring near real-time analysis with streaming data that poses one major challenge. Another equally formidable big data challenge is the “Complexity” or variety of data that can exist in both, small or big data sets.

A class of tool known as exploratory data analysis (EDA) tools used specifically for scientific research work, can be illustrated to prove the power of data velocity and data variety.

Role of EDA tools in scientific discovery

EDA tools are often used in routine laboratory experiments that do not necessarily involve high-volume data but highly complex data collected from sensor-aided lab instruments or streaming data sources at high velocity. So, the velocity and variety characteristics of big data come into play in scientific data discovery. Scientists frequently use systematic exploration, discovery, and visualization of relationships and correlations in data sourced from such experiments.

The uses of EDA in science experiments

The EDA is broadly used in initial exploratory studies for both data discovery, hypothesis testing, and hypotheses refinement.  The next stage is the detailed analysis of significant parameter-correlations found in initial data discovery among masses of complex, high-dimensional data. In an article published in Nature magazine, the author Andrew Gelman— a Columbia University statistician—seems to indicate that scientists usually start out with small-scale, explorations to collect results; then they proceed to use the collected results to make decisions about the next stages of research to arrive at pre-determined hypotheses. In simple terms, EDA tools combine “both exploratory and confirmatory methodologies” to establish the “repeatability and reproducibility” of such methods in scientific experiments. The ability to be able to reproduce an experiment’s rationale and its results provides immense power to the scientific community.

The important features of an EDA tool

EDA tools rarely make a false attempt to rival the best-of-breed statistical packages like SAS or R, but rather focus on the “correlation discovery” in the complex, scientific data. In that sense, a good EDA tool can complement a statistical package like SPSS or R in the data-discovery process.

Standard EDA tools provide support for the following capabilities:

  • Visual integration of data and detection of  “primary key” across multiple datasets
  • Visual interface for data selection, filtering, and exploration
  • Provision for quick confirmation of hypotheses through a display of findings; also, capability for indicating additional experiments through the generation of new  hypotheses
  • Auto search capability to detect correlations among pairs of parameters
  • Auto search facility for finding correlations between virtual parameters
  • Quantitative analysis of each of the result value
  • Auto sort feature to sort results in a hierarchical manner from strong to weak
  • Optional correlation analyses within multiple sub-segments of each parameter’s range of possible values
  • Visual representation of most significant pair-wise correlations through a network diagram
  • Output from all types of correlation analysis

Benefits of EDA tools in science experiments

EDA tools enable interesting findings in the early stages of research. This provides four clear benefits to the scientist:

  1. The tools can indicates design flaws and suggest necessary improvements.
  2. The EDA tool can validate initial hypotheses.
  3. The EDA tool can generate new, testable hypotheses.
  4. The tools can uncover those high-value data that warrant more statistical analysis.

The above capabilities can together empower the data scientist to tell the “data story” in present in the entire range scientific datasets. Additionally, an EDA tool, used properly, can save hours of useless research, and facilitate unique discoveries that are beyond known correlations and projected results.

Future promises of EDA tools

The Butler Scientifics website reports that EDA-enabled research lead to quick discovery of all the correlations (less than 2 hours) “that the science team identified during their 8-weeks intensive work but also several key correlations that, with a further confirmatory phase, confirmed their original hypothesis. “

Some highly complex correlations such as multi-valued data patterns in multi-dimensional data is still beyond the scope of EDA tools though such features may become a reality in future releases.

Boston city bkg

Made in Boston @

The Harvard Innovation Lab


Matching Providers

Matching providers 2
comments powered by Disqus.