Evaluate OER Collections

Evaluating an open textbook is similar to evaluating any textbook. To begin, let’s look at the qualities that make a good textbook, whether it’s open or not. By evaluating the following criteria, you can assess an open educational resource repository to identify opportunities to improve the open resources available to your institution:

• Authority: Is it clear who developed and wrote the material? Are their qualifications for creating the material clearly stated?
• Accuracy: Are there errors or omissions visible?
• Objectivity: Is any type of bias present?
• Currency: Is the resource up-to-date and/or is a creation or update date visible?
• Coverage: Does it address the topic at hand sufficiently to add value to the class? Does only a portion of it apply? Do you need to combine it with other resources? Can you align each resource with the learning objectives and weekly lessons on your syllabus in order to identify gaps?
• Accessibility: Is it ADA compliant? (Instructors planning to use OERs in their classrooms should keep in mind that the OERs must comply with federal and state accessibility requirements. Here is a checklist for compliance with Section 508 of the Americans with Disabilities Act.)
• License: Has a Creative Commons License been applied? Can you remix or reuse the item? Who do you have to attribute copyright to, if anyone?
• Persistence: Prior to using an OER in another class, you’ll need to check that the URL is still valid and whether the OER was updated since you last accessed it.

Open Oregon Educational Resources and British Columbia Open Education Librarians (BCOEL) have developed checklists, some designed for a specific user type, to evaluate individual course materials such as open textbooks.

• Faculty Checklist for Evaluating Course Materials
• Administrator Checklist for Evaluating Course Materials
• Student Checklist for Evaluating Course Materials
• BCcampus open textbooks review rubric [Word file]

Assess an OER collection with the Open Education Resource Repository Rubric with the open textbook and OER evaluation process.

What Are The Qualities of a Good Textbook?

You have probably reviewed many textbooks in your career and have your own set of evaluation guidelines. Before we begin looking at the criteria that various open textbook projects are using to evaluate the textbooks in their collections, let’s review some of the qualities and structures that make up a textbook.

Pedagogical Aids for Textbooks

When it comes to textbooks, there is no doubt that the content of the book is important. You want to be sure that the content in the textbook aligns with the learning outcomes of your course and that the content is written at the right level for your learners.

However, a good textbook is more than a collection of facts. Textbooks also contain a number of pedagogical aids, which are features or attributes of the book that help a learner understand the content being presented.

There are a number of common pedagogical aids in textbooks, including:

• Chapter objectives/learning outcomes
• Chapter outlines
• Chapter summaries or reviews
• Checklists
• Headings and subheadings
• Bold and italicized text
• Table of contents
• Table of figures
• Indexes
• Focus questions and practice questions
• Case studies, vignettes, and examples of best practices
• Glossary and key terms
• Demonstrations and simulations
• Maps and timelines
• Illustrations including photos, charts, diagrams, and figures
• Multimedia
• Pronunciation guides

Usefulness of Pedagogical Aids in Textbooks

Of these pedagogical aids, which are the ones that students actually use and find useful? Weiten, Guadagno & Beck (1996) surveyed 134 students asking them how familiar they were with the different pedagogical aids in their textbooks.^[1] They also asked the students the probability they would use each pedagogical aid and their perceived value of each aid. With their research, Weiten, Guadagno, and Beck showed that the top three pedagogical aids that students used in textbooks were boldfaced technical terms, chapter and section summaries, and glossaries.

In his 2003 research Pedagogical Aids and Student Performance, Gurung surveyed more than 200 undergraduate students and asked them to rate the usefulness of 10 pedagogical aids and instructional techniques. ^[2]Gurung’s research wasn’t specific to textbook aids, but did include a number of textbook-specific aids, like outlines, chapter summaries and reviews, boldfaced and italicized terms, and key terms and practice questions found in textbooks. When you eliminate the non-textbook–related aids from Gurung’s research, we find that the top textbook aids used by students were boldfaced terms, italicized terms, and practice questions. Chapter summaries and reviews were a very close fourth.

In terms of helpfulness, students rated boldfaced (92%) and italicized (81%) terms as the most useful pedagogical aids, followed by practice test questions (77%) and chapter summaries and reviews (73%) as being moderately to extremely helpful.

As we can see, it is often the simplest aids like bold and italicized terms, chapter summaries, and outlines that are most often used by students.

Open Textbook–Specific Considerations

So far, we have been looking at textbooks in general and not specifically open textbooks. For the most part, the criteria you would use to judge the effectiveness of an open textbook would be the same as any textbook. However, there are some unique criteria you may want to consider for open textbooks that might not apply to a closed textbook. Some of the additional items you may want to consider when evaluating an open textbook are:

• Should I modify or edit the book to meet my needs, and if so, does the license allow me to edit or modify the book?
• Is the open textbook available in a technical format that I can edit if I want to modify it?
• Is there supplemental material available for the textbook? For some faculty, supplemental resources are important components of a textbook adoption. While some open textbooks do have supplemental materials associated with them, many do not. This may or may not be an issue for you.
• Is there a community to support the open textbook? Often, these communities can help provide support for you in your adoption. If you have a problem or concern or are just looking to connect with others who are using the same resource, a well-developed community can help provide support for your adoption. For example, Stitz-Zeager’s open pre-calculus textbook website has a community with discussion forums and blogs that can help provide some support for your adoption. The Writing Commons is another example of an open textbook with a community associated with it.

Additional Resources

Achieve’s OER rubrics

• Achieve.org has developed eight OER rubrics as well as an evaluation tool to help users determine the degree of alignment of OER to the Common Core State Standards, and aspects of quality of OER.

iRubric: Evaluating OER rubric

• Questions to ask about the OER you are thinking of using. This rubric is developed by Sarah Morehouse with help from Mark McBride, Kathleen Stone, and Beth Burns is licensed under a Creative Commons Attribution 3.0 Unported License.

OER Pros/Cons and Evaluation Methods by University of the Pacific-University Libraries