Chapter 4 Appendices

Appendix A: Assignment Guidelines for the Artificial Intelligence Multiple-Choice Question Assessment

For the ease of assessment adoption, I have provided the assignment documentation that is provided directly to students. This appendix includes the assessment’s learning objectives, description, directions for completion, and grading rubric for instructor use. Since this material is openly licensed, readers can retain, revise, remix, reuse or redistribute this documentation according to this book’s open license specifications.

Learning Objectives

• Construct AI prompts to generate scenario-based multiple-choice questions based on course concepts and theories.
• Evaluate AI-constructed multiple-choice questions for accuracy, structure, and bias.
• Apply theoretical perspectives to hypothetical scenarios when revising AI responses.
• Apply accurate representations of diversity, equity, & inclusion principles through the revision of contextual details and language in questions.
• Justify the accuracy of the AI multiple choice questions utilizing critical thinking skills and peer-reviewed course materials.
• Write logical arguments to support the ranking of incorrect answers.

Assignment Description

For this assignment, you will utilize an artificially intelligent large language model such as ChatGPT or Google’s Gemini to assist you in the construction of scenario-based multiple-choice questions on course concepts and theories. While large language models can write effective multiple-choice questions, they do a poor job of constructing specific questions that ask test takers to apply their knowledge to real-life scenarios. AI also struggles to use inclusive language that represents our multi-faceted, multicultural environment. Therefore, you will be tasked with evaluating, revising, and justifying AI questions for use by students in future classes. The questions will be openly licensed using SoftChalk and embedded in our learning management system for future students to use when studying for course exams.

Directions for Assignment Completion

1. First, you will want to write a “prompt” to type into the generative AI program (Google Gemini, ChatGPT, etc.). A prompt is “any form of text, question, information, or coding that communicates to AI what response you’re looking for” (CoSchedule, 2024.). Depending on how you phrase your prompt, AI could provide varying responses.
2. Next, prompt the artificial intelligence (Google Gemini or ChatGPT) to construct one scenario-based or problem-based multiple-choice question based on <insert a course chapter or specific theories here>. Make sure to include in your prompt that the question should be scenario-based, includes the educational level of your future test takers (ie. undergraduate students), and that you want the AI to include the correct answer to the question in the results.
3. Take a screenshot of the prompt you generated and the AI’s answer. If the terminology in the question is foreign to you, you can generate more than one question to ensure it matches our course materials. Paste the screenshot of your prompt and question into a Microsoft Word or Google document.
4. Next, you will need to investigate if the answer is correct using our peer-reviewed, current course materials (our textbook or lecture PowerPoints). Directly below the screenshot, include a short evaluation of the accuracy of ChatGPT or Gemini’s answer to the question, including a ranking of the question’s answers with your rationale for your ranking.
   1. If the answer AI provided is correct, you will need to identify where you found the correct answer to the question using your textbook. List the page and paragraph number where you found information that justifies why the answer is correct. Explain in a few short sentences why the answer is correct utilizing the information in your textbook.
   2. If you believe that the answer AI provided is incorrect, provide the page and paragraph number proving the answer is incorrect with an explanation of why it is incorrect utilizing the information in your textbook.
5. After confirming the accuracy of the answer, you will now review the question’s remaining incorrect answer choices.
6. After you justify the accuracy of the question’s answer, you will then rank the plausibility of the other answer’s choices. You want to rank the remaining answer options from most correct to least correct. In a few sentences, explain why you ranked the answer choices in the manner that you did.
7. Next, it is time for the fun part! You are now going to revise the problem-based scenario to be more specific by adding context and details to the scenario. For instance, adding biographical information, such as names and ages, provides more context for your reader to answer the question. You should use inclusive names and language in your revision. Your question scenario can be NO longer than four sentences and the details should remain significant to the question. The details should not distract or mislead the reader.
8. Review the answers to the question and make necessary adjustments where needed. All of the answer choices should be plausible, but only one should be the correct answer. Answers should also be written using a similar pattern of wording.
9. Finally, you will need to complete this process for three questions. A sample is provided on our learning management system for your review (Appendix B). Please format your document in the same order and with the same headers.

AI Multiple-Choice Assignment Grading Rubric

Criteria	Exceptional (100-90%)	Above Average/Average (89-70%)	Below Expectations (69-0%)
AI Prompt & Response (5%)	Submission included screenshots of the original artificial intelligence prompts and responses for all 3 questions.	Submission included screenshots of the original artificial intelligence prompts and responses for at least 2-1 questions.	Submission failed to include any screenshots of the artificial intelligence prompts and responses.
Rationale & Ranking (25%)	Demonstrated exceptional critical thinking when justifying the accuracy of the answers to the questions. Accurately ranked answer choices. Provided an exceptional ranking and explanation of the plausibility of the incorrect answer choices.	Demonstrated critical thinking when attempting to justify the accuracy of the answers to the questions. Most answer choices were ranked correctly. Provided a ranking and explanation of the plausibility of the incorrect answer choices. The explanations were unclear or needed additional specificity.	Lacked a clear demonstration of effective critical thinking skills when justifying answer accuracy and ranking answer choices.
Question Reconstruction (25%)	Demonstrated exceptional analytical and critical thinking through the revision of scenario-based multiple-choice questions. Added rich details without confusing student exam responders. Questions were adequately difficult for college students. All questions were no longer than 4 sentences in length.	Demonstrated analytical and critical thinking through the revision of scenario-based multiple-choice questions. Added details, yet had too many details or details confused student exam responder. Some questions were too easy or too difficult for college students. Some questions were longer than 4 sentences in length.	Lacked a clear demonstration of effective analytical and critical thinking skills in the revision of scenario-based multiple-choice questions. Most questions were too easy or too difficult for college students. All sentences were longer than 4 sentences in length.
Connection to Course Materials (15%)	Each answer and ranking rationale is tied to course materials with the inclusion of a page and paragraph number.	Most answer and ranking rationales are tied to course materials with the inclusion of page and paragraph numbers.	Answer and ranking rationales were not tied to course materials and lacked the inclusion of page and paragraph numbers.
Application of Diversity, Equity & Inclusion (25%)	Implements diverse and inclusive language in question scenarios. Names and details in scenarios represented a diverse population.	Attempts to implement diverse or inclusive language in question scenarios. Some names and details in scenarios represented a diverse population.	Lacks inclusive language in question scenarios and contains bias. Didn’t demonstrate any diversity.

References

CoSchedule. (2024). AI prompt. The Ultimate Marketing Dictionary of Terms and Definitions. https://coschedule.com/marketing-terms-definitions

Appendix B: Sample of a Completed Student Assignment

The following is a completed student sample that I provide for students with the assignment directions and the corresponding grading rubric. Figure 1 is an image of a student written prompt entered into Google’s Gemini AI generator. The image also includes the Google Gemini’s response, which is a short multiple-choice question.

Correct Answer Rationale

The correct answer is “Cognitive Behavioral Therapy” because CBT is a type of therapy that is effective in treating a wide range of psychological disorders, including anxiety. CBT focuses on identifying and changing negative thoughts and behaviors that contribute to the client’s anxiety. The other answer choices are also valid approaches to treating anxiety, but they are not typically as effective as CBT for first-line treatment and our textbook advises CBT as the best approach (Course textbook, page 120, paragraphs 3-5).

Ranking Rationale

The next plausible answer to this question would be psychoanalysis. This would be less likely to be as effective or efficient as CBT for a specific anxiety disorder like generalized anxiety disorder. Psychoanalysis is a long-term therapy that focuses on exploring unconscious conflicts, which might be less relevant to Bill’s immediate needs (Course textbook, page 125, paragraph 2).

The third option, humanistic theory, is much less likely to help Bill with his anxiety disorder since this approach focuses on personal growth and self-actualization, which might not directly address Bill’s specific symptoms of anxiety and panic attacks. This may help Bill grow as a person, but his panic attacks are likely to remain (Course textbook, page 130, paragraph 4).

The least correct answer for this question is letter B or psychopharmacology because it states in the question scenario that Bill can’t take SSRIs. Since we do not know more about Bill’s health, we can’t assume or eliminate the possibility that he could take a benzodiazepine. If he could take a benzodiazepine, it is recommended that he not take it for more than 2-4 weeks as it can cause physical dependence. Finally, our textbook author argues that benzodiazepines have been less effective in producing long-term change and CBT appears more effective for producing long-term change (Course textbook, page 135, paragraph 2).

Revised Question

Cara is a clinical psychologist working with a client named Amir who has been diagnosed with generalized anxiety disorder. Amir has been experiencing frequent panic attacks when praying at his local mosque and is struggling to manage their anxiety daily. Although Amir believes in taking medication for illnesses, he can’t take anti-anxiety drugs (SSRIs) because they interfere with the Warfarin medication he is currently taking.

Which of the following approaches should Cara utilize to help Amir manage his anxiety effectively?

Answer choices:

1. Cognitive-behavioral therapy (CBT)
2. Psychopharmacology
3. Psychoanalysis
4. Humanistic Theory

Correct answer:

1. Cognitive-behavioral therapy (CBT)

Appendix C: Sample of Student Survey

The following is a list of the questions I gave students in a Google Form survey to collect and organize student preferences for sharing their work.

Questions	Response Formats
List your first and last name.	Short Answer
May I use your questions for future students? Some questions may be integrated into online digital quizzes that the public can access. Choose yes or no.	Multiple Choice
If your questions are used in a public online quiz, do you want me to include your name, or do you want to remain anonymous?	Multiple Choice Options include: Please include my name to represent my work. I am fine sharing the questions, but leave my name off the quiz. I do not want to share my questions.

Students were also asked the following questions to assess the strengths and weaknesses of the assessment. The students’ survey responses are discussed in the chapter’s debrief section, and assessment revisions were made based on their feedback.

Questions	Response Formats
Did you find the process of revising and improving the AI-generated questions to be challenging?	Multiple Choice Options include: Very challenging Somewhat challenging Neither challenging nor easy Somewhat easy Easy
How did you feel about sharing your revised questions for the benefit of future students?	Multiple Choice Options include: Excited to contribute to future learning Satisfied to be part of the process Neutral Slightly hesitant Not interested in sharing
What were the biggest challenges you faced while completing the assessment?	Long-answer text

Appendix D: Discussion Questions: Reflecting on the AI-Generated Exam Question Assessment

Below is a list of potential student discussion questions:

1. What steps did you take to critically evaluate the AI-generated questions?
2. Based on the AI model’s responses to your prompts, did you identify any bias? Can you provide an example?
3. How can humans ensure that AI-generated questions are culturally sensitive and inclusive? What strategies can be used to avoid bias and stereotypes?
4. How did you improve the quality of the AI-generated questions? What specific changes did you make to your questions to represent your life or the lives of others?
5. In your future career, how can you ensure your work is diverse and inclusive? What strategies can you use to avoid bias and stereotypes in your future career?