32 Principals of Data Interpretation

One point that Dr. Bracey makes very certain, is the rule that you should never take statistics at face value.  He lists thirty-two principals that should be considered while interpreting data:

1. Do the arithmetic.
2. Show me the data!
3. Look for and beware of selectivity in the data.
4. When comparing gourps, make sure the groups are comparable. 
5. Be sure the rhetoric and the numbers match.
6. Beware of convenient claims that, whatever the calamity, public schools are to blame.
7. Beware of simple explanations for complex phenomena.
8. Make certain you know what statistic is being used when someone is talking about the "average."
9. Be aware of whether you are dealing with rates or numbers.  Similarly, be aweare of whether you are dealing with rates or scores.
10. When coparing either rates or scores over time, make sure the groups remain comparable as the years go by.  
11. Be aware of whether you are dealing with  ranks or scores.
12. Watch out for Simpson's paradox.
13. Do not confuse statistical significance and practical significance.
14. Make no causal inferences from correlation coefficients.
15. Any two variables can be correlated.  The resultant correlation coefficient might or might not be meaningful.
16. Learn to "see through" graphs to determine what information they actually contain.
17. Make certain that any test aligned with a standard comprehensively tests the material called for by the standard.  
18. On a norm-referenced test, nationally, 50 percent of students are below average, by definition. 
19. A norm-referenced standardized achievement test must test nly material that all children have had an opportunity to learn. 
20. Standardized norm-referenced testes will ignore and obscure anything that is unique about a school.  
21. Scores from standardized tests are meaningful only to the extent that we know that all children have had a chance to learn the material which the test tests. 
22. Any attempt to set a passing score or a cut score on a test will be arbitrary.  Ensure that it is arbitrary in the sense of arbitration, not in the sense of being capricious. 
23. If a situation really is as alleged, ask, "so what?"
24. Achievement and ability tests differ mostly in what we know about how students learned the tested sills. 
25. Rising test scores do not necessarily mean rising achievement.
26. The law of WYTIWYG applies: What you test is what you get.
27. Any tests offered by a publisher should present adequate evidence of both reliability and validity.
28. Make certain that descriptions of data do not include improper statements about the type of scale being used, for example, "the gain in math is twice as large as the the gain in reading."
29. Do not use a test for a purpose other than the the one it was designed for without taking care to ensure it is appropriate for the other purpose. 
30. Do not make important decisions about individuals or groups on basis of a single test.
31. In analyzing test results, make certain that no students were improperly excluded from the testing. 
32. In evaluating a testing program, look for negative or positive outcomes that are not part of the program.  For example, are subjects not tested being neglected? Are scores on other tests showing gains or losses?