Critical Evaluation and Digital Literacy

This grove curates the intellectual arc of Ian O'Byrne's doctoral research at the University of Connecticut (2007–2012) — the questions asked, the framework built, the instruments developed, and the threads that run directly into current work on digital literacy, identity, and empowerment.

Useful for: connecting past and present thinking | sharing with doctoral students | understanding where the foundational frameworks came from.

The Core Question

Can adolescents learn to critically evaluate online information by learning to create it?

This was the animating question of five years of doctoral work. The intuition: if students build a convincing-looking but factually wrong website — a hoax — they learn from the inside what makes websites appear credible. They develop a critical stance not from lecture but from the act of construction itself.

The dissertation tested this formally. What it found confirmed the intuition: content creation pedagogy enhances students' ability to recognize credibility and relevance markers, and cultivates the disposition of healthy skepticism toward online information.

The Theoretical Framework

The research drew on three interlocking frameworks, first articulated in the Philosophy of Cognition and Instruction (2009):

New Literacies (Leu, Kinzer, Coiro) — Literacy is deictic: what counts as reading and writing changes as technologies change. Online reading comprehension is not the same as print reading comprehension; it demands new skills, new strategies, new dispositions. The research assumed that what students needed to know was still an open question because the Internet was still changing what "knowing" meant.

Social Constructivism (Vygotsky) — Learning happens through social interaction at the edge of competence (ZPD). The instructional model built in collaborative phases: students analyzed websites together, built websites together, evaluated peers' websites together. Knowledge was constructed socially, not transmitted individually.

Cognitive Apprenticeship (Brown, Collins, Duguid) — Expertise is best developed by doing authentic tasks with expert modeling and scaffolding. The classroom became a design studio: students were apprentice web designers and critical analysts, not passive recipients of information literacy instruction.

The Research Design

A quasi-experimental mixed-methods study conducted in an economically challenged middle school.

Three-phase instructional model:

Analyze — Students examine existing websites for markers of credibility (author, purpose, source, content, accuracy) and relevance (topic fit, currency, usability)
Construct — Students build a website on an assigned topic, including one with deliberately misleading elements (the "hoax" phase)
Evaluate — Students evaluate each other's websites using the same credibility/relevance framework

Two instruments developed:

Online Reading Dispositions — a survey measuring student attitudes and habits of mind when reading online (5 factors: Critical Thinking, Critical Stance, Collaboration, Flexibility, Persistence)
Critical Reader of Online Information — a performance-based instrument measuring ability to evaluate credibility and relevance of real websites (not formally published; embedded in dissertation)

Quantitative strand: Pre/post DORC and CROI scores; odds ratio analyses of instrument factor structures

Qualitative strand: 11 focus group interviews with student website-design teams; interpretive case studies; discussion theme analysis

What the Research Found

Quantitative findings:

The ability to recognize and construct surface-level markers of credibility and relevance can be enhanced through content creation pedagogy
The disposition of critical stance (healthy skepticism toward online sources) can be enhanced — students became more skeptical readers after building fake websites
Critical Thinking dispositions (persistence, metacognition, goal-directed reading) showed improvement

Qualitative findings:

Students reasoned about credibility through the lens of their own design choices — "we made ours look real by adding a copyright, so now I check for that"
Context mattered enormously: classroom environment, teacher relationships, peer dynamics shaped what students were willing to construct and critique
The "hoax" frame was generative — it gave students permission to think like adversaries

The COMPS Foundation

Before the dissertation, the comprehensive exams (2009) forced a crystallization of intellectual commitments that shaped everything that followed.

Philosophy of Cognition and Instruction — An 8-page statement articulating three tenets (Social Constructivism, New Literacies, Cognitive Continuum Theory) and two predictions for education over the next 25 years: democratization of knowledge (good) and deepening digital divides (bad). Written in 2009; both have materialized.

Online Content Creation Annotated Bibliography — A 62-source review mapping the OCC construct across fields: multiliteracies, multimodality, digital storytelling, new media. This bibliography defined the theoretical territory that the dissertation then investigated empirically. The OCC construct — students as makers of digital content, not just consumers — remained a through-line in work for the next decade.

What Came After

The dissertation didn't end the inquiry — it opened it. Several direct lines out of the PhD work:

Publications from the research:

OByrne2009 - Measuring dispositions online — preliminary DORC validation with J. Gregory McVerry, published before the dissertation was complete
OByrne2013 - Facilitating critical evaluation skills through — the dissertation itself, formally published
OByrne2014 - Online content construction Empowering students — translating the OCC framework for broader audiences
OByrne2014 - Empowering learners readerwriter nature digital — the reader/writer duality thesis: online texts are always already sites of both consumption and production

Intellectual threads that continue:

The question of critical evaluation didn't go away — it got harder. Misinformation, AI-generated content, deepfakes. The COIL/DORC framework anticipated a problem that grew significantly worse.
The "learn by making" pedagogy became a broader stance: digital identity work, open scholarship, blogging as thinking. Online Content Creation as empowerment, not just skill-building.
The reader/writer duality — every online reader is a potential writer; every content creator is shaping what others read — became central to work on digital citizenship, online identity, and more recently AI literacy.

Connecting Past to Present

The 2009 COMPS philosophy statement predicted that ICTs would either democratize knowledge or deepen divides, depending on whether people developed the literacies to participate fully. That framing — digital literacy as a civic and equity issue, not just a skills issue — runs through everything from the dissertation to current work on Digital Sovereignty and AI Literacy.

The dissertation studied adolescents in an economically challenged school. That context wasn't incidental — it was the point. Digital literacy instruction matters most where digital access and support are thinnest.

The tools changed. The question didn't.