Appendix F3 Guidelines for Critically Appraising Quantitative Data Collection Plans

Did the researchers use the best method of capturing study phenomena (i.e., self-reports, observation, biomarkers)?

If self-report methods were used, did the researchers make good decisions about the specific methods used to solicit information (e.g., in-person interviews, Internet questionnaires, and so on)? Were composite scales used? If not, should they have been?

If observational methods were used, did the report adequately describe what the observations entailed and how observations were sampled? Were risks of observational bias addressed? Were biomarkers used in the study, and was this appropriate?

Did the report provide adequate information about data collection procedures (e.g., the training of the data collectors)?

Did the report offer evidence of the reliability of measures? Did the evidence come from the research sample itself, or was it based on other studies? If reliability was reported, which estimation method was used? Was the reliability sufficiently high?

Did the report offer evidence of the validity of the measures? If validity information was reported, which validity approach was used?

If there was no reliability or validity information, what conclusion can you reach about the quality of the data in the study?