Ed Discussions’s Customizable Class Bot
What we tried
One question we’re working through this semester is how AI use affects student learning. We hope this course prepares students for research and the workplace, including learning to be savvy and discerning with AI. The course encourages students to use AI for coding while also learning to explain their code and understand the underlying concepts. Closed-book exams and an oral exam assess students’ own understanding.
Shiro implemented a custom AI bot on Ed Discussion to help answer students’ questions. The teaching team tested the bot on practice questions. While it was able to answer them successfully, Shiro pointed out that some answers were too long and used more notation than students would need for the exam. The course offers the Ed Discussion bot as a resource for students.
recode_values() and case_when().What we noticed
Despite having access to the bot, students do not appear to be using it.
I understand the hesitation—in fact, students’ choice not to use it makes a lot of sense to me. The course allows other AI tools, and students were encouraged to get a coding agent subscription at the beginning of the semester, so the Ed bot is competing with tools students already use.
For me, the ease and familiarity of my own chatbot outweigh the reasons to try the class bot, especially when its benefits aren’t clear. When I attempted the first problem set, I chose the AI I was most familiar with. It’s hard for the Ed Discussion bot to compete with my existing routine. While I can’t speak for all students, this is my intuition about what some may be thinking.
What we would change
If we continue offering the Ed Discussion bot, I think we need to show students when and why it would be useful, including what advantages it offers over their own AI tools. A short demonstration using an actual assignment question could give students a reason to try it.
For future iterations of the course, I would also like to explore requiring students to show how they are engaging with the AI as it feels inevitable that many students are using AI tools to assist with their work, if not completing the entire assignment. One way to address this is making students submit their chatbot history, which would allow the teaching team to see which parts of the pset the students would consult AI as well as what types of prompts or questions students have. (Though, I can’t imagine the teaching team having enough time to go through it manually but using AI to summarize what happened in the chat session could streamline this). This way, AI is still allowed to be used, but rather than uploading the whole assignment for an AI to complete, the students have to break the assignment into parts and go through the effort of showing they are engaging with the AI to get their answer. A less demanding approach is to ask students to respond to a reflection question at the end of the assignment, asking students to identify one exchange, include the relevant prompt and response, and explain how they thoughtfully interacted with the AI: whether they used, changed, or rejected its response, and how they checked the result. The chat session history approach at this moment cannot be faked or generated as easily as the reflection question, which AI can answer for the students as well.