Your task for this assignment is to locate a refereed journal article on a program evaluation, read the article, and critique it according to the guidelines listed below. Please note that these are only suggested guidelines that apply to any research articles, not just those dealing with program evaluation. Students are not expected to address every point, but they are expected to seriously consider and address gaps that may exist in the program evaluation's method, etc.. Students are expected, however, to follow APA format.
| |
| * What is the stated purpose of the program evaluation? |
| * Does the literature review provide a context, background, and direction? Does it support the need for the program evaluation? Is the review adequate; if not, what seems to be missing? Explain. |
| * Is a theoretical framework presented? Is it appropriate? Can you think of a different or additional theoretical perspective that might have been useful? |
| * How does the program evaluation intend to contribute to knowledge about the field of study? |
| * What are the research questions, hypotheses, and objectives? If there is a hypothesis, is it directional? What were the independent and dependent variables? Is there a connection between the literature review and the research questions, hypotheses, and objectives? |
| |
| * Is the program evaluation descriptive, causal-comparative, or correlational? Longitudinal? Crosssectional? Qualitative? Quantitative? Explain. |
| * How well is this design suited to the research question or hypothesis? Is the design modified in response to any constraints? Are there threats to the internal validity of the research design? Explain. |
| * What model/theory was used? Was the model used appropriately? Explain. |
| * Is it a formative or summative evaluation? Explain. |
| |
|
|
| * Describe the characteristics of the population studied. |
| * Identify sampling procedures (e.g., simple random sampling), and explain why it was selected. Indicate the size of the sample and explain why the size is sufficient. |
| * What was the sample size? Was it sufficient? |
| * What was the response/participation rate? How did those who responded or participated differ from those who did not? How was this addressed? |
| |
| |
| * What measures were utilized? Were the variables operationalized as needed? |
| * What instruments were used? How were they developed? What did they measure? |
| * What standardized instruments, if any, were used? |
| * Were the instruments valid, reliable, and appropriate? Describe any reliability and validity tests that were conducted. Was this sufficient? |
| * What other type of tests would you have recommended? |
| |
| |
| * What kind of data analysis was conducted? What statistical analysis was used? |
| * Were the statistics appropriate for the type of questions being answered and the variables being used? |
| * What were the units of analysis being used? Were they appropriate? |
| * Was the information discussed and described clearly and accurately? Describe the information presented in each of the tables and figures. |
| * Were any statements made that are open to the ecological fallacy or that suggest reductionist reasoning? Explain. |
| * What were the findings? Do you think the conclusions are valid and reliable? Explain. |
| * Were any causal assertions made or implied in the hypotheses or in subsequent discussion? What approach was used to demonstrate the existence of causal effects? Explain. |
| * Were all four criteria for establishing causal relationships addressed? Explain. |
| * Were any variables controlled to reduce the risk of spurious relationships? Should any other variables have been measured and controlled? How satisfied are you with the internal validity of the conclusions? Explain. |
| * Was the information presented in the figures and tables clear and was it discussed in the text? Explain. |
| |
| |
| * What were the program evaluation's findings? Were they discussed and described clearly and accurately? Were explanations proposed for any anticipated and unanticipated findings? Were the results substantively important? Were conclusions well grounded in the findings? Was any light shed on the theoretical framework used? Explain. |
| * Are any other interpretations possible? Was any further research recommended? What might you recommend? Are there any questions you feel were not addressed or addressed adequately? Explain. |
| * Were there any confounding variables? If so, how might the program evaluation's research design have been improved to reduce interference from confounding variables? |
| * Was any further research recommended? What might you recommend? What additional research questions and hypotheses are suggested by the program evaluation's results? Did the program evaluation yield additional insights? Explain. |
| * Was the program evaluation conducted in an objective fashion? Is there any evidence of bias? Are there limitations to the generalizability (i.e., external validity) of the findings? Explain. |
| |
| |
| * Did the program evaluation seem consistent with current ethical standards? Did it pose any threats to research participants? Were any steps taken to minimize these threats? Explain. |
| * How was cooperation of research participants obtained? Was there informed consent? Explain. |
| * If you were on a Human Subjects Review Committee, what kind of information would you want to know about this program evaluation's design and methodology? |
| * Do you consider this a "good" program evaluation? Explain. |
| |
