In the big data era, the power of science lies in technology-enabled data discovery
Big data’s biggest challenge, as most will agree, is not data volume. It is the velocity of data flow in applications requiring near real-time analysis with streaming data that poses one major challenge. Another equally formidable big data challenge is the Complexity or variety of data that can exist in both, small or big data sets.
A class of tool known as exploratory data analysis (EDA) tools used specifically for scientific research work, can be illustrated to prove the power of data velocity and data variety.
Role of EDA tools in scientific discovery
EDA tools are often used in routine laboratory experiments that do not necessarily involve high-volume data but highly complex data collected from sensor-aided lab instruments or streaming data sources at high velocity. So, the velocity and variety characteristics of big data come into play in scientific data discovery. Scientists frequently use systematic exploration, discovery, and visualization of relationships and correlations in data sourced from such experiments.
The uses of EDA in science experiments
The EDA is broadly used in initial exploratory studies for both data discovery, hypothesis testing, and hypotheses refinement. The next stage is the detailed analysis of significant parameter-correlations found in initial data discovery among masses of complex, high-dimensional data. In an article published in Nature magazine, the author Andrew Gelman-a Columbia University statisticianseems to indicate that scientists usually start out with small-scale, explorations to collect results; then they proceed to use the collected results to make decisions about the next stages of research to arrive at pre-determined hypotheses. In simple terms, EDA tools combine both exploratory and confirmatory methodologies to establish the repeatability and reproducibility of such methods in scientific experiments. The ability to be able to reproduce an experiment’s rationale and its results provides immense power to the scientific community.
The important features of an EDA tool
EDA tools rarely make a false attempt to rival the best-of-breed statistical packages like SAS or R, but rather focus on the correlation discovery in the complex, scientific data. In that sense, a good EDA tool can complement a statistical package like SPSS or R in the data-discovery process.
Standard EDA tools provide support for the following capabilities:
- Visual integration of data and detection of primary key across multiple datasets
- Visual interface for data selection, filtering, and exploration
- Provision for quick confirmation of hypotheses through a display of findings; also, capability for indicating additional experiments through the generation of new hypotheses
- Auto search capability to detect correlations among pairs of parameters
- Auto search facility for finding correlations between virtual parameters
- Quantitative analysis of each of the result value
- Auto sort feature to sort results in a hierarchical manner from strong to weak
- Optional correlation analyses within multiple sub-segments of each parameters range of possible values
- Visual representation of most significant pair-wise correlations through a network diagram
- Output from all types of correlation analysis
Benefits of EDA tools in science experiments
EDA tools enable interesting findings in the early stages of research. This provides four clear benefits to the scientist:
- The tools can indicates design flaws and suggest necessary improvements.
- The EDA tool can validate initial hypotheses.
- The EDA tool can generate new, testable hypotheses.
- The tools can uncover those high-value data that warrant more statistical analysis.
Future promises of EDA tools
The Butler Scientifics website reports that EDA-enabled research lead to quick discovery of all the correlations (less than 2 hours) that the science team identified during their 8-weeks intensive work but also several key correlations that, with a further confirmatory phase, confirmed their original hypothesis.
Some highly complex correlations such as multi-valued data patterns in multi-dimensional data is still beyond the scope of EDA tools though such features may become a reality in future releases.
