Pre-class homework completion rates in my first-year courses

In my mind it is hard to get students to do pre-class homework (“pre-reading”) with much more than an 80% completion rate when averaged out over the term. It usually starts higher than this, but there is a slow trend toward less completion as the term wears on. After taking a more careful look at the five introductory courses in which I used pre-class assignment I have discovered that I was able to do much better than 80% in some of the later courses and want to share my data.

Descriptions of the five courses

The table below summarizes some of the key differences between each of the five introductory physics courses in which I used pre-class assignments. It may also be important to note that the majority of the students in Jan 2010 were the same students from Sep 2009, but not much more than half of the Jan 2013 students took my Sep 2012 course. For Jan 2013 only two of the students had previously taken a course with me.

Course Textbook Contribution to overall course grade Median completion rate (the numbers in brackets show the 1st and 3rd quartiles)
Sep 2009 (Mechanics) Young & Freedman – University Physics 11e Worth 8%, but drop 3 worst assignments. No opportunities for late submission or earning back lost marks. 0.73 (0.62,0.79)
Jan 2010 (E&M) Young & Freedman – University Physics 11e Worth 10%, but drop 2 worst assignments. No opportunities for late submission or earning back lost marks. 0.78 (0.74,0.89)
Sep 2011 (Mechanics) smartPhysics Worth 8%. Did not drop any assignments, but they could (re)submit at any point up until the final exam and earn half marks. 0.98 (0.96,0.98)
Jan 2012 (E&M) smartPhysics Worth 8%. Did not drop any assignments, but they could (re)submit at any point up until the final exam and earn half marks. 0.94 (0.93,0.98)
Jan 2013 (E&M) Halliday, Resnick & Walker – Fundamentals of Physics 9e & smartPhysics multimedia presentations Worth 10%. Did not drop any assignments, but they could (re)submit at any point up until the final exam and earn half marks. 0.93 (0.87,0.97)

Overall the style of question used was the same for each course, with the most common type of question being a fairly straight-forward clicker question (I discuss the resources and assignments a bit more in the next paragraph). I have not crunched the numbers, but scanning through results from the Jan 2013 course shows that students are answering the questions correctly somewhere in the 65-90% range and the questions used in that course were a mishmash of the Jan 2010 and Jan 2012 courses. Every question would have an “explain your answer” part. These assignments were graded on completion only, but their explanation had to show a reasonable level of effort to earn these completion marks. Depending on class size, I did not always read their explanations in detail, but always scanned every answer. For the first couple of assignments I always made sure to send some feedback to each student which would include an explanation of the correct answer if they answered incorrectly. Each question would also be discussed in class.

A rundown of how the resources and assignments varied by class:

  • For Sept 2009 and Jan 2010 I used a Blackboard assignment to give them the three questions each week and told them which sections of the textbook to read, and I didn’t do much to tell them to skip passages or examples that weren’t directly relevant.
  • For Sept 2010 and Jan 2012 I used smartPhysics (link to UIUC PER group page, where they were developed). These consist of multimedia presentations for each chapter/major topic, which have embedded conceptual questions (no student explanations required for these). After they are done the multimedia presentation, they then answer the actual pre-class questions, which are different from those embedded in the multimedia presentation. For the most part, the questions in their pre-class assignments were similar to the ones I was previously using except for the smartPhysics ones were often more difficult. Additionally, my one major criticism of smartPhysics is that I don’t feel they are pitched at the appropriate level for a student encountering the material for  the first time. For more on this, have a look at the second bullet in the “Random Notes” section of this post I did on pre-class assignments (link). One of the very nice things about smartphysics is that everything (the regular homework, the pre-class assignments and the multimedia presentations) all used the same web system.
  • For January 2013, I was back on assigning the pre-class assignments through Blackboard. The preamble for each of the pre-class assignments pointed them toward a smartPhysics multimedia presentation and the relevant sections of the textbook we were using. Students could use one, the other or both of these resources as they felt fit. I don’t think I ever surveyed them on their use of one over the other, but anecdotally I had the sense that way more were using the multimedia presentations.

The data

I present two graphs showing the same data from different perspectives. Figure 1 shows how the fraction of the class completing a given pre-class assignment varies over the course of the term. There is a noticeable downward trend in each course. Figure 2 shows the fraction of assignments completed by each student in each class.

This is the caption

Figure 1. For the first five graphs, the x-axis represents time going from the start of the course (left side) to the end of the course (right side). The box and whiskers plot compares the five courses according to the previously used colours, where the line in the boxes shows the median and the boxes show the 1st and 3rd quartiles. The whiskers are the matplotlib default; they extend to the most extreme data point within the 1.5*(75%-25%) range.

This is the caption

Figure 2. Histograms showing the fraction of assignments completed by each student. An ambiguous looking artifact appears in the 0 bin for Sept 2011, but all students in that course completed 70% or more of the pre-class assignments

Discussion

There is clearly a large difference between the first two courses and the final three in terms of the rates at which students were completing these pre-class assignments. The fact that I saw 98% of these assignments completed one term is still shocking to me. I’m not sure how much each of the following factors contributed to the changes, but here are some of the potential factors…

  1. Multimedia presentations – students seem to find these easier to consume than reading the textbook. There is a study [Phys. Rev. ST Physics Ed. Research 8, 010103 (2012)] from
    Homeyra Sadaghiani at California State Polytechnic University where she did a controlled study comparing the multimedia presentations to readings in textbooks, and used the same pre-class assignments for both. In addition to finding that the multimedia presentation group did slightly better on the exams, she also found that the students had a favorable attitude toward the usefulness of the multimedia presentations, but that the textbook group had an unfavorable attitude toward the textbook reading assignments. But she also mentions that the multimedia group had a more favorable attitude toward clicker questions than the textbook section, and this alone could explain the difference in test performance as opposed to it having to do with the learning which takes place as part of the pre-class assignments. If the students in one section are buying into how the course is being run more than another, they are going to do a better job of engaging with all of the learning opportunities and as a result should be learning more. There are a variety of reasons why reading the textbook may be preferred to have them watching a video or multimedia presentation, but you can’t argue with the participation results.
  2. Generating buy-in – I have certainly found that, as time wears on, I have gotten better at generating buy-in for the way that my courses are run. I have gotten better at following up on the pre-class assignments in class and weaving the trends from their submissions into the class. However, for the Sep 2009 and Jan 2010 courses, that was the most personal feedback I have ever sent to students in an intro course on their pre-class assignments so I might have expected that getting better at generating buy-in might cancel out the decreased personal feedback.
  3. Changes in grading system – This may be a very large one and is tied to generating buy-in. For the first two courses I allowed them to drop their worst 3 or 2 pre-class assignments from their overall grade. In the later courses, I changed the system to being one where they could even submit the assignments late for half credit, but were not allowed to drop any. In the latter method I am clearly communicating to the students I think it is worth their time to complete all of the assignments.

In poking around through the UIUC papers and those from Sadaghiani, that 98% completion rate from my Sept 2011 course is really high, but is going to be an overestimate of how many people actually engaged with the pre-class material as opposed to trying to bluff their way through it. The smartPhysics system also gave them credit for completing the questions embedded in the multimedia presentations and I’m not presenting those numbers here, but when I scan the gradebooks, those that received credit for doing their pre-class assignments also always received credit for completing the embedded questions in the multimedia presentations. But, it is possible to skip slides to get to those so that doesn’t mean they actually fully consumed those presentations. Based on reviewing their explanations each week (with different degrees of thoroughness) and then docking grades accordingly, I would estimate that maybe 1 or 2 students managed to bluff their way through each week without actually consuming the presentation. That translates to 2-3% of these pre-class assignments.

Sadaghiani reported “78% of the MLM students completed 75% or more of the MLMs”, where MLM is what I have been calling the multimedia presentations. Colleagues of mine at UBC found (link to poster) that students self-reported to read their textbooks regularly in a course that used a quiz-based pre-class assignment (meaning that students were given marks for being correct as opposed to just participating, and in this case were not asked to explain their reasoning). 97% of the students actually took the weekly quizzes, but there is a discrepancy in numbers between those that took the quizzes and those that actually did the preparation.

With everything I have discussed here in mind, it seems that 80% or better is a good rule of thumb number for buy-in for pre-class activities, and that one can do even better than that with some additional effort.

About these ads

9 Comments on “Pre-class homework completion rates in my first-year courses”

  1. Andy "SuperFly" Rundquist says:

    This kind of data and analysis is very useful, Joss, thanks! I have tried all kinds of ways to get students to engage, but I always stop short of making it worth any credit. It probably doesn’t surprise you that I get pretty crappy completion rates. I really like the notion of asking students for summaries and questions instead of quizzing them, but I’ve never really been fully happy with the results.

    This semester I’m trying the “flip the flipped” model, where I expect basically nothing before class and a lot afterwards. The plus sides are 1) everyone feels fully prepared for class (because I don’t expect anything :) ), 2) they have to really engage in class with brand new ideas, and 3) they get to ask for exactly what resources they’d like to have outside of class to get prepared for assessments (quizzes and screencasts). The down sides are 1) we don’t get very far in class, 2) they seem to be doing less work (if the semester ended right now many would fail), and 3) I’m sure they’re consuming the resources with specific quizzes in mind instead of trying to learn broadly.

    Here’s the biggest thing, though: I’m less crabby. That’s huge, and it might be big enough to have me try this approach in a sophomore majors class next semester.

    So, my questions for you (since that’s one of the rules of #NaBloCoMo ): Is the point where the work crosses from busy work for points to useful work for learning clear to students? Is it roughly similar for the majority of the students? Can you tell if you’re close to that magic point?

    Thanks again for the very cool post. Happy National Blog Comment Month!

    • Joss Ives says:

      And a Happy #NaBloCoMo to you to Andy!

      To answer your questions. Judging from student feedback that I did (see http://learnification.wordpress.com/2012/03/23/student-opinions-on-what-contributes-to-their-learning-in-my-intro-em-course/), the majority do see it as somewhat or completely busy-workish. I asked them how much they felt the pre-class assignments contributed to their learning and 30% said a lot, 55% said a little and 11% said not at all. But I’m pretty careful to make sure that, in class, I build on the pre-class assignments instead of simply re-teaching what was there.

      I have a slow build of a post that I have been tossing around in my brain related to how, in many ways, I want to do away with pre-class assignments in my own courses and embrace something more similar to your flip of the flip model. There are two main reasons. First, when a student chooses not to do a pre-class assignment, they are not only making themselves fall behind because they skipped some work, but they are also making it so that there time spent in class is a lot less productive. If you go the other way, and they are skipping the post-class work, they still have the full advantage of the learning that took place in class. Second, since so much of the work that they do in my courses is group work of some sort, those that choose to not be prepared also end up hurting the members of their group because their group spends so much time trying to get the unprepared people up to speed. I have seen it happening a number of times and find it to be a very frustrating situation. I have come close, but never pulled the trigger on forcing all the unprepared students to be in groups with each other.

      Anyway, the magic point involves not only their preparation, but also what you do in class with the preparation they have done. Despite my last paragraph, I am quite happy with my current implementation of the system given its purpose. In an ideal world, I would like to see the smartPhysics presentations boiled down to something more suitable for first-contact with the material, where they currently want to cover all the challenging math and derivations within the presentation. I think I must be close to the magic point because the pre-class assignment completion rates are at least as good as the homework ones, and many of those that don’t feel like doing the homework are resourceful enough to get the homework marks without actually doing the work to complete those homework assignments. So there is a little more than just busy-work for marks going on. Unfortunately, some of the extra buy-in that I was generating with the personal feedback has to go since my sections have increased in size by an order of magnitude now that I am at UBC. I’m going to try out the local system and see how it suits.

  2. Jared Stang says:

    Can you share what you’ve learned about generating buy-in for how your course is run? Is it mostly about explicitly connecting to the pre-reading during the class or is that you constantly remind them that doing the pre-reading is good for them? If the latter, do you opt to show them research that backs up your claims?

    • Joss Ives says:

      Hi Jared. In addition to what I wrote in my reply for Andy, it’s not just about generating buy-in for the pre-class assignments, but generating buy in for how the entire course runs. One of the pieces that I got better at was presenting the entire course as a cohesive whole, where all the pieces fit together nicely to help them learn physics. I pick my battles with reminding them what is good for them, but I estimate that for the first half of the course I will present something quick once per week that is meant to continue to generate buy in or meant to maintain buy in. If I have data from my own previous courses, they seem to that more convincing than presenting something from the literature (according to my anecdotal evidence of number of people coming up to me after class in a panic because my correlation graph of thing x and final exam grade showed that they where setting themselves up to fail). If I don’t have data from my own courses, I will then use the literature to come up with something convincing. I try to be really opportunistic with these buy in reinforcement messages. If there is a week when a lot less people than normal complete the pre-class assignment, that is when I would specifically talk about that. After the third weekly quiz I usually show them how success on the weekly quizzes correlates very highly with final exam scores.

  3. bretbenesh says:

    This data is awesome. Thanks for sharing.

    Here is what has worked best for me (keeping in mind that my classes are all under 35 students): don’t grade it, but send students personal emails if they do not do the reading assignment. This creates a sort of social pressure to do the pre-class work.

    • Joss Ives says:

      Your method of sending the personal emails might be even more effective in a really large class where they assume that they are anonymous, but that email communicates the exact opposite. Given how high the completion rates are for the reading quizzes at UBC, it wouldn’t even be a terribly large number of emails that I would have to send. I will have to keep that in my back pocket.

      After approximately 4 weeks, I have typically emailed all the students that I have identified as at risk and share with them the places where their behavours are suggesting that they are at risk. Completing very few pre-class or end-of-week homework assignments is way up there on my list of things to look for.

  4. This is a really interesting study. I find it really encouraging that in all the scenarios the rates of completion are really quite high (although as you say a few may have bluffed their way). Interesting comments as well as the data. This article has made me want to learn more, and I am planning to look through the literature on multimedia vs traditional reading, and on different reward systems. Thanks so much!


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 34 other followers