Pre-requisites
- Basic knowledge of AI and its functions in information processing
- Familiarity with academic and non-academic sources
- Understanding of bias and misinformation in media
Explanation of activity
Students will work in groups to analyze the effectiveness and reliability of AI in evaluating sources. Each group will be assigned different sources that have been evaluated by AI, and they will:
- Assess AI’s evaluation of the source’s credibility, bias, and reliability.
- Compare AI-generated assessments to their own critical evaluations.
- Discuss AI’s strengths and limitations in source validation.
- Present their findings and recommendations on improving AI’s reliability in source evaluation.
Group Work Activity: Title: AI vs. Human Judgment – Who Evaluates Better?
- Formation of Groups
- Divide students into groups of 3-5 members.
- Assign each group a different AI-evaluated source.
- Analysis Phase
- Groups will critically assess the AI-generated evaluation.
- Compare AI findings with their own manual source evaluation.
- Discussion & Reflection
- Identify strengths and weaknesses of AI in source validation.
- Debate potential AI biases and limitations.
- Presentation of Findings
- Groups will present their analysis and suggestions for improving AI-assisted source evaluation.
- Peer feedback and teacher moderation will guide reflections on AI reliability.
Teacher materials
- Case studies of AI-generated source evaluations
- Guidelines on AI limitations and biases
- Sample prompts to improve AI’s source analysis
Relevant Reading to Understand the Materials:
- Articles on AI in digital literacy
- Research papers on disinformation and AI detection
- Guidelines from media literacy organizations
Sources:
- Sarkar, S., & Ghosh, A. (2024). Leveraging artificial intelligence to enhance media literacy and combat misinformation. JNRID, 2(9), a163-a187. https://doi.org/10.1729/Journal.41684. Available at SSRN: https://ssrn.com/abstract=4991258
- Tiernan, P., Costello, E., Donlon, E., Parysz, M., & Scriney, M. (2023). Information and media literacy in the age of AI: Options for the future. Education Sciences, 13(9), 906. https://doi.org/10.3390/educsci13090906
- Zhou, J., Zhang, Y., Luo, Q., Parker, A. G., & De Choudhury, M. (2023). Synthetic lies: Understanding AI-generated misinformation and evaluating algorithmic and human solutions. CHI ’23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, Hamburg, Germany. https://doi.org/10.1145/3544548.3581318
- Chu-Ke, C., & Dong, Y. (2024). Misinformation and literacies in the era of generative artificial intelligence: A brief overview and a call for future research. Emerging Media, 2(1), 70-85. https://doi.org/10.1177/27523543241240285
Further reading to further develop the activity/understanding of the topic:
- AI ethics and bias in source evaluation
- Comparative studies of AI vs. human judgment in source validation
Sources:
- Hanna, M. G., Pantanowitz, L., Jackson, B., Palmer, O., Visweswaran, S., Pantanowitz, J., Deebajah, M., & Rashidi, H. H. (2025). Ethical and bias considerations in artificial intelligence/machine learning. Modern Pathology, 38(3), 100686. https://doi.org/10.1016/j.modpat.2024.100686
- Ferrara, E. (2024). Fairness and bias in artificial intelligence: A brief survey of sources, impacts, and mitigation strategies. Sci, 6(1). Available at SSRN: https://ssrn.com/abstract=4883478
- Osborne, M. R., & Bailey, E. R. (2025). Me vs. the machine? Subjective evaluations of human- and AI-generated advice. Scientific Reports, 15, 3980. https://doi.org/10.1038/s41598-025-86623-6
- Barrington, S., & Farid, H. (2023). A comparative analysis of human and AI performance in forensic estimation of physical attributes. Scientific Reports, 13, 4784. https://doi.org/10.1038/s41598-023-31821-3
Student Materials
- AI-generated source evaluations for analysis
- Worksheets for recording observations and discussions
- Guidelines on effective AI prompting for source analysis
Analysis of Student Understanding
- Class discussion on AI’s effectiveness and reliability
- Student reflections on AI’s role in research
- Comparative evaluation of AI vs. manual source analysis
Assessing and Evaluation
- Students submit an analysis of AI-evaluated sources
- Group presentations on findings and recommendations
- Peer review discussions on AI effectiveness
Modulations
- Adapting the activity for different academic levels
- Expanding to include AI-assisted writing and fact-checking
- Exploring different AI models and their approaches to evaluation
You may also like
Resources
June 18, 2025
