How to Calculate Precision of Data

by Damien Thryn; Updated September 26, 2017

Data precision is one of the most important considerations when conducting scientific or statistical analysis. Commonly confused with the equally important concept of accuracy, the dart board analogy articulated by the University of Hawaii demonstrates the relationship: accurate data points average out to equal expected results, while precise data points cluster closely together, even if they aren't close to anticipated results. According to Dartmouth College, precision is a measurement of the reproducibility of a set of results. Precision in data sets is an important concept even in technology related endeavors, as shown by Kenneth E. Foote and Donald J. Huebner with the University of Texas-Austin in an analysis of Geographic Information Systems. Calculating precision is a fairly simple though somewhat subjective exercise.

Items you will need

  • Graphical representation of a data set
  • Information on the relevant units expressed in the data
  • Minimum allowable margin of error in the experiment
Step 1

Develop a visual representation of data points such as a scatter plot. A very simple visual representation involves plotting the corresponding dependent and independent variable values for each data point on a Cartesian coordinate system.

Step 2

Assess the groupings of data points and look for patterns. Precise data manifests in clusters of data points, indicating that similar input variables correlate to similar output variables.

Step 3

Apply information on the units of measurement used to collect the data to determine the average spacing between data points. A simple ruler measurement can be used to determine the distance between points on the graph, then converted using an arbitrary, convenient scale that corresponds to the units of measurements used to generate the data points. This will allow data points' precision relative to one another to be calculated by taking the average of the distances.

Step 4

Compare the minimum margin of error allowed in the experiment and the average precision of the data points to determine the relative overall precision of the experiment. Different types of experiments will have greater or lesser error tolerance: an engineering project will likely require precision down to very small units, while a social experiment will likely tolerate more variance.


  • Try to assess the likely unit scale before creating the graphic representation of the data points. This will make it easier to assess precision visually in order to identify any areas of especially notable precision or imprecision.

    Clear patterns of data occurring on a visual representation are highly indicative of precision and the repeatability of an experiment. Continued experimentation should add further data points in precise clusters close to those already in existence.


  • Do not confuse accuracy with precision. If the goal of an experiment is to achieve an average output value of give for all inputs, and this is achieved by averaged values ranging from -12 to 14, this is unlikely to be a precise measurement, though it may be accurate. A precise measurement might result in all data points clustering around 17, which would be inaccurate, but precise and therefore predictable.

About the Author

Damien Thryn holds a bachelor's degree in political science from the University of California at Berkeley. Specializing in writing on video games, he has played PC games for more than 20 years and has written about many classics. He is also an avid console gamer.

Photo Credits

  • travail de precision image by margouillat photo from