How to Assess Student Learning in Online Classes

Tips and tools for assessing student learning in online environments

How can student learning be assessed in online environments? Online teaching poses complex challenges and issues that instructors may not have experienced in face-to-face classes.

This page discusses assessment in five aspects: What is student assessment?; how assessment should be conducted; how to utilize assessment tools; how to handle problems; and what students need to know about assessment, with practical tips and tools that can be immediately used. Note that this page introduces general pedagogical knowledge and is not intended to cover all practical examples. Please use your judgment as to what advice to follow, depending on your individual needs.

You can get further information from the video and reference material (KU members only) used in the online workshop, "How Assessment in Online Learning Should Be Understood and Conducted" (June 9, 2020).

What is student assessment?

Assessment of Learning

Assessment of student learning refers to (1) ascertaining the actual state of learning, (2) providing appropriate feedback, and (3) measuring learning progress toward objectives. Assessment is implemented for students' successful learning experiences.

Student assessment is divided into three types: (a) diagnostic assessment, which is conducted before or at the start of a course, (b) formative assessment, which is conducted throughout a course, and (c) summative assessment, conducted at the end of or after a course period.
  Examples of (a) diagnostic assessment: placement tests and assessment tests
  Examples of (b) formative assessment: mini-exams, midterm exams, minute papers, and student reflections
  Examples of (c) summative assessment: end-of-term exams and reports

Differences between In-Person and Online Settings

Unlike face-to-face objective testing, cheating cannot be easily prevented in online objective testing. Meanwhile, there is not much difference between face-to-face and online in other assessment methods.

Comparison of Assessment Methods in Different Learning Formats

assessment_01.png

How assessments should be conducted

Devise Exam Methods and Questions

Devise exam methods

    To prevent cheating in online exams, consider some of the following ideas:
  • Prepare several different question patterns
  • Use random sequences to present questions
  • Break answer time into short chunks (e.g., one slot per question)
  • Use a web camera to monitor students during an exam
    + a second web camera to monitor their hands or the entire test venue

Devise exam questions

Another way to minimize cheating is to reduce the number of memory-based questions and devise questions that cannot be answered easily by referring to reference materials or by internet searches

  • Require students to write answers by showing the process leading to and/or the laws and principles applied to the answers.
      (e.g., for pencil-and-paper tests, ask the students to send in still images of their answers, or videotape the process of answering the questions, and submit the data to the instructor)
  • Ask students to illustrate the relationships between learned concepts and keywords (a concept map) and have them explain the map
  • Ask students to create their own questions and answer them under a given set of rules

The following test formats may be useful besides conducting written and objective tests online as they have been conventionally done in the classroom.

  • Take Home Exam:
    Students are allowed to take a test by referring to textbooks and literature and submit their answers by a specific due date. Take-home exams are typically in essay format.
  • Open Book Exam:
    In this format, students are allowed to bring designated textbooks, lecture notes, and other reference materials into the test venue.

  Sample questions for take-home and open-book exams
  Excerpts from Centre for Teaching and Learning, The University of Newcastle, Australia, "A Guide for Academics--Open Book Exams".
assessment_02.png

Examine Assessment Methods

Consider various assessment options

・Use methods other than objective tests
In addition to mini-tests, essays, and reports done inside and outside the classroom, questionnaires, reflections, and self/peer evaluations may be used as part of the student evaluation process.

・Reconsider the proportions of assessment components
Lower the proportion of test results among the overall assessment components: e.g., 40% for mini-tests, 30% for essays and reports, 20% for student reflections, and 10% for self-assessment.

Incorporate formative assessments

・Give assignments in each class session and calculate grades cumulatively, rather than conducting an exam at the end of the course for summative assessment.

・Attendance alone shall not be assessed. Combine attendance with in-class minute papers (student reflections using ICT) and comprehension tests for assessment.

・Conduct questionnaires if needed, and measure student understanding in parallel with assessment.

Methods to Prevent Cheating in Online Written Exams

・Depending on the subjects, it may be difficult to replace the written exam with another assessment method such as a report. The following website introduces some ideas for conducting exams using web conferencing systems and other methods.
Center for the Promotion of Excellence in Higher Education, Kyoto University, "Educational Assessment".

How to utilize assessment tools

Assessment with PandA(LMS)

Assignment Tool

PandA assists instructors and students in giving, receiving, returning, and grading assignments. It also allows you to set release and due dates for assignments and give students report assignments inside and outside of class.
For more detailed information, see the PandA Instructor's Manual (pp.25-33).

Test/Quiz Tool

The test/quiz tool helps create simple mini-tests in multiple-choice, true/false, and essays. If "marks" are set for each correct answer, the total marks will be automatically displayed. This tool is useful for comprehension tests given inside and outside of class to measure student understanding.
With the test/quiz tool, the following types of questions can be created.
・Fill-in-the-blank questions and questions that ask for numerical answers
・True/false questions
・Short text questions

For an overview of the test/quiz tool on PandA, see the PandA Instructor's Manual (pp.25-33).
For further information on the test/quiz tool on PandA to conduct an online test, refer to Using the Test/Quiz Tool on PandA (in Japanese).
Also, view videos and reference materials (KU members only) provided in the Workshop on PandA for Online Tests and How to Use the Test/Quiz Tool〔2020-23〕.

Zoom-based Assessment

Poll Function

The Poll function enables instructors to create multiple-choice questions to present to students and share the results with them. Questions can be created either in advance or in class when appropriate. If student accounts are registered, the instructor can download the poll results from "Poll Report" and analyze who voted for what.
Polls can be created exclusively by the host.

Chat Function

Using the Chat function, instructors can post questions, receive responses, and distribute files as attachments. Sender names are displayed, and logs are stored. Accordingly, instructors can check the log later to incorporate students' responses into an assessment. Furthermore, the Recording function can be used to evaluate the comments made by students as assessment elements.

You can find information on Zoom here.

Other ICT-based Assessment Tools

Web questionnaires

To facilitate students to reflect on a class session, navigate them to a web questionnaire form (e.g., Google Form and REAS (in Japanese)) to complete during or after class. Web questionnaires are also effective as a means of encouraging students to study outside of class hours.
Questions can primarily include:
・Self-assessment of the degree of understanding of lectures and the level of involvement in lessons
・Reflection on lesson content in students' own words
・Open-ended queries about lessons

Additionally, downloading the compiled results in a CSV file enables the instructor to count them in the assessment and present the responses to provide feedback in the next week's class.

Compile and share comments (to measure the degree of understanding or collect comments in class)

Use Clicker or Mentimeter to ask students to respond to questions prepared in advance or created on the spot. Responses from students can be visualized in real-time. This is a convenient way of tracking the degree of students' understanding and collecting comments during class. Moreover, with this method, the instructor can immediately provide feedback to students.

Rubric-based Assessment

About Rubrics

A rubric is a tabulated set of assessment criteria used to evaluate and provide feedback on submissions, such as essays and reports, as well as student behavior during experiments and presentations.
It consists of three elements: (1) criteria of assessment, (2) rating scale (3- to 5-level rating scales are commonly used), and (3) a detailed description of performance level, as illustrated in the figure below. The instructor is required to develop a rubric that contains information relevant to assignments.

assessment_003.png

Utilize Rubrics

Rubrics offer the following advantages, which enable instructors to conduct instruction and assessment in a unified manner.

  • Rubrics presented to students in advance provide clear grading criteria and facilitate understanding of the quality expected from assignments.
  • Points assigned to assessment scales facilitate scoring.
  • Feedback to students about their position on the assessment scale is useful in offering guidance.
  • Rubrics can be used for self- and peer-assessment conducted by students.

Rubrics need not necessarily be developed from scratch. Existing ones developed by other instructors and those with public access on the internet are available, such as the ones in the Rubric Bank provided by the Japan Association for Educational Development in Higher Education (in Japanese). You can customize these rubrics for your use.

How to handle problems

Prepare for Problems Beforehand

When delivering lectures online, you may often find assessment methods and criteria differ from the initial syllabus. In this case, the instructor is required to inform students about what has changed and how this impacts the assessment. Letting students know early about assessment methods can avoid potential problems beforehand.

Be prepared for network problems

Many network-related problems have been reported in relation to online learning. Listed below are possible ways of preventing network problems from occurring when conducting assessments.

  • Ask students to retain evidence of their submission when they submit an assignment or another type of assessment.
    (e.g., backing up data, capturing a screenshot)
  • Conduct a rehearsal of the assessment/submission process in advance with a TA or other staff.
  • Prepare more than one method of submission and inform students about their options.
  • Inform students in advance about contact information and measures to take when a network problem occurs.

Deal with Misconduct

Develop measures to take upon detection of misconduct. Inform students about the measures

The following measures can be taken to deal with detected misconduct.

  • During an Exam:
    Tell the student to stop taking the exam and ask their ID and make the student leave the room.
  • After an Exam:
    Inform the relevant office on campus about the misconduct, ask the student in question about the fact, and give them guidance.

Deal with plagiarism

ICT-based plagiarism detection tools are available to check for plagiarism in submitted reports.

What students need to know about assessment

Assessment Methods and Criteria

When delivering lectures online, assessment methods and criteria may often differ from the initial syllabus. In this case, the instructor's responsibility is to inform students about what has changed and how it impacts their assessment. Letting students know early about assessment methods can help avoid potential problems.

Communicate the Importance of Academic Integrity

It is crucial for all individuals involved in academia to understand academic integrity, i.e., acting in an academically honorable manner, having a sense of responsibility, and developing the skills to act ethically. Additionally, as a member of an academic community, you should work together with students to promote a culture of academic integrity.

To this end, you are encouraged to create opportunities with your students to discuss what constitutes academically desirable conduct and dishonesty (e.g., plagiarism, inappropriate collaborative work, copying, and cheating) and why academic integrity is important. As an example, the Columbia College Honor Code concerning academic integrity for students is shown below.

Columbia College Honor Code
The Columbia College Student Council, on behalf of the whole student body, has resolved that maintaining academic integrity is the preserve of all members of our intellectual community - including and especially students.

As a consequence, all Columbia College students make the following pledge:

We, the undergraduate students of Columbia University, hereby pledge to value the integrity of our ideas and the ideas of others by honestly presenting our work, respecting authorship, and striving not simply for answers but for understanding in the pursuit of our common scholastic goals. In this way, we seek to build an academic community governed by our collective efforts, diligence, and Code of Honor.

In addition, all Columbia College students are committed to the following honor code:

I affirm that I will not plagiarize, use unauthorized materials, or give or receive illegitimate help on assignments, papers, or examinations. I will also uphold equity and honesty in the evaluation of my work and the work of others. I do so to sustain a community built around this Code of Honor.

Source: https://www.college.columbia.edu/academics/academicintegrity

Efforts to Raising Students' Awareness of Academic Integrity

A way for students to raise awareness of academic integrity is to require them to read a pledge like the one shown below and to choose "I Agree" (if they agree) before they begin an online exam.

Example from UBC Okanagan School of Health and Exercise Sciences, where only course notes/readings can be brought to the exam.
I pledge to uphold the highest standards of ethics and academic integrity as a future health professional. I will abide by the rules my instructor has provided for the final exam and will not violate the UBC Academic Honesty and Standards Policy.

This means that I will only use my course notes/readings and will not use any other outside assistance (no internet or other people including my peers)

Source: https://ctl.ok.ubc.ca/teaching-remotely/final-exams/integrity-pledge/

Specific Elements of Academic Integrity

As stated in the Columbia College Honor Code, the concept of academic integrity is highly abstract. In more concrete terms, what does it mean to establish academic integrity? To answer this question, take a look at the guidelines for establishing academic integrity excerpted from student handbooks issued by universities in western countries, including MIT's handbook on academic integrity for students and a leaflet published by Princeton University.

Instructions for Students "Before Taking an Online Examination/ Signing a Pledge to Follow the Rules"

The following document provides concrete examples of how to inform students about rules on online exams and how to ask them to follow the rules for successful online exams. You can modify the examples according to the method of conducting your online exam and the exam objectives.

Example: Before Taking an Online Examination/ Signing a Pledge to Follow the Rules (KU members only)

    «Exam Rules»
  • About the exam
  • Pre-exam instructions
  • Points to keep in mind during the exam
  • If misconduct is observed
  • «Signing a pledge before taking an online exam»
References
Instructional Methods for Examinations and Grading for Online Learning (Tips), Institute of Liberal Arts and Sciences, Nagoya University (in Japanese)
・ NAKAJIMA Hidehiro (ed.), Instructional Methods for Colleges Series 4: Learning Assessment, Tamagawa University Press, 2018.
"Ten Points to Consider When Assessing Online Learning," Center for Education in Liberal Arts and Sciences (in Japanese)