Summer 2012 Research, Part 1a: Bonus content for immediate feedback during an exam bonus content

This is a quick follow-up to my previous post on my research related to the effect of immediate feedback during exams.

I love it when I’m going through my “to read” pile of papers and realize that there is something in there related to one of my own research questions. There is a paper from last year (Phys. Rev. ST Physics Ed. Research 7, 010107, 2011) by Fakcharoenphol, Potter and Stelzer from University of Illinois at Urbana-Champaign that looked at how students did on matched pairs of questions as part of preparing for an exam.

There’s a lot of interesting stuff in this paper, but the result which is most relevant to my own research is the following. They developed a web-based (voluntary) exam preparation tool where students would do a question, receive feedback as just the answer or as a solution, then do a matched question where they received the other type of feedback. They divided the students into four groups so that every student had equal access to answer vs. solution feedback on the questions.  For each matched question pair (let’s call them questions A and B), the grouping of the students allowed half of the students to answer question A first and the other half to answer question B first. Within each of those groups, half of the students received answer only feedback for their first question and the other half received solution feedback for their first question.

They called the first question of a pair answered by the students their baseline and the students scored 58.8%±0.2% on those questions. Keeping in mind that they had many pairs of questions, the average performance of the students on the follow questions was 63.5±0.3% when only the answer was supplied after answering the first question and 66.0±0.3% when the solution was  provided after answering the first question. There are statistically significant differences between all of these numbers, but the gains from receiving the feedback are not overly impressive. More on this in a moment.

Back to my own research. During an exam, I used matched pairs of questions and gave the students feedback on their first question (in the form of just the answer) before they answered the second question. I saw a statistically significant improvement from the first question (65.3±6.8%) to the second one (77.5±6.0%), but due to low statistics there was not much to conclude other than it was worth pursuing this research study further. The results from the UIUC folks set the magnitude scale for the effect I will see once I am able to improve my statistics (58.8%±0.2% to 63.5±0.3% due to solutions only feedback).

I’m really not certain if I expect to see less, equal or more improvement for my “during an exam feedback” design as their “preparing for an exam feedback” design. In their design, the level of preparation of their students when using their study tool is all over the map (they look at this in more detail in their paper) so it is not known if the learning effect due to the feedback also depends on when during their overall study plan they were using the tool (e.g. as a starting point for their studying vs. to check their understanding after having done a bunch of studying).  Since both our designs use multiple-choice questions (but preparation vs assessment conditions) I am not certain how guessing would play into everything.

I have to admit that if my future research into the effect of feedback during an exam finds that I am getting only a 5% gain (like UIUC did for their solution only feedback) from this intervention that I doubt that I would continue with the practice.

Tier 2 Canada Research Chair – Teaching and Learning at University of the Fraser Valley

I’m looking for feedback on my computational physics course

Overview

This will be my first course in which I use some flavour of standards-based grading. This will also be the first time I have taught a computational physics course (or more accurately a computational thinking for physicists course). I do have some experience helping students develop some computational thinking skills from my Advanced Lab course, but I have never taught a full course dedicated to this topic. Fun!

The idea is that I will be getting them to work in Python and Mathematica in parallel. My main purpose behind this is to help the students see the platform-agnostic computational thinking which underlies their their computational tasks. A side benefit is that I can help Python and Mathematica put aside their differences and work toward a common goal.

This course will be based around a weekly cycle of

• Basic tasks, to be completed in both the Mathematica and Python computing environments; and
• Intermediate and advanced tasks, to be completed using either of the computing environments.

The grading part of SBG

Students will earn marks for each of the content standards and their final grade will come from points earned by demonstrating appropriate mastery of these standards. I will give the students example tasks that correlate to the standards, but the students will always have the option of performing any computational task that they wish that shows appropriate mastery of the standards.

Approximately 2/3rds of the way through the course, the students will start working on a project which models a complex physical system. I have two standards associated with this project (a physics one and a communication one) and I am planning to either weight these standards more heavily than the others or make them into a set of more fine-grained standards. I really haven’t decided on this.

Assessing an individual standard

I plan to assess each standard on a 0-4 scale. I’m not in love with my use of the term “completed” below when discussing tasks, but the idea is that the most important thing to me is that their programs do exactly what they are meant to do.

I think that I will be allowing students to partner up which means that I need a mechanism to assess the individual beyond just a working (and properly commented) program. I think that the individual assessment will be for them to orally run me through their properly working program. Students will be given the option of submitting screencasts to do this, but not all students will have access to a computer that has both a mic and Mathematica so I will need to keep the in-person explanations as a method.

Here’s a rough outline of how I plan to assess an individual standard, where each

• 4 – Exceeds expectations (student must make the case that they have exceeded expectations or have successfully completed an advanced task)
• 3 – Meets expectations (student has completed the relevant intermediate tasks or otherwise demonstrated intermediate-level mastery in either environment)
• 2 – Approaches expectations (student has completed the relevant basic tasks or otherwise demonstrated basic-level mastery in both environments)
• 1 – Doesn’t meet expectations (student has completed the relevant basic task or otherwise demonstrated basic-level mastery in one of the environments)
• 0 – Not yet assessed

Very loose weekly plans

We will spend roughly one week on each of the following broad themes.

1. Introduction to the environment (functions, variable types)
2. Iteration basics and animation (introductory modeling: similar to the early Matter and Interactions VPython stuff)
3. File input/output, basics array manipulation and case structures
4. Advanced list/array operations and manipulation
5. Data visualization and plotting (histograms, scatter-plots, bar charts, error bars, etc)
6. Data analysis (basic statistical analyses, fitting)
7. Solving complex algebraic equations, integration and differentiation
8. Solving differential equations

After the students have started working on their projects we will spend less than half of our class time working on non-project topics.

9. Introduction to the physics modeling project
10. Monte Carlo methods
11. Numerical methods
12. Linear algebra
13. FFTs

The standards

These standards are based on a collection of computational physics learning goals that were put together by Andy Rundquist, Danny Caballero and Phil Wagner. I then went around to all the faculty in my own department and asked them what skills they would like to see the students develop as part of this course and folded the common ones in.

There are some standards marked as “[ungraded]“. The idea with these is that they are things which do not seem to be worth assessing for one reason or another, but are things that I still want to highlight for the students as being important.

Onto the standards…

1. Environment fundamentals
   1. [ungraded] I can use online and built-in help resources
2. Computer algebra fundamentals
   1. I can represent and perform common operations with complex numbers and vectors.
   2. I can use built-in integration and differentiation functions. This includes using assumptions for integration.
   3. I can access mathematical and physical constants or define these constants globally when not available.
   4. I can solve algebraic equations. This includes simplifying algebraic results and using a root finder. I can use graphical or other techniques to set appropriate neighborhoods for the root finder.
3. Programming fundamentals
   1. [ungraded] I can assign and clear variables.
   2. I can write and use robust functions. The important characteristics of a robust function are (1) that they need no access to parameters outside of those which they were passed; (2) they are written such that they can easily be copied into any script or notebook and be used as intended; and (3) they can return information in the form of single parameters, vectors, arrays or other useful objects.
   3. I can use at least two different iteration methods to accomplish the same task.
   4. I can use case structures.
4. Arrays and lists
   1. I can manipulate and slice arrays/lists. Slicing an array means to pick out columns or rows from larger arrays. Manipulation of an array includes transposing the array, replacing elements, and adding columns/rows to existing arrays
   2. [The wording needs work on this one] I can operate on entire arrays/lists instead of having to operate on the individual elements of the array/list.
5. Numerical techniques
   1. I can write my own code (not call existing functions) to perform numerical integration with varying levels of precision (Trapezoidal rule, Simpson rule)
6. Solving differential equations
   1. I can solve ODEs (1st Order, 2nd Order and Coupled) analytically. I can determine if an analytic solution exists.
   2. I can solve ODEs numerically. I can set initial conditions and the domain of the solution.
   3. I can solve a Partial Differential Equation and specify the boundary conditions appropriately.
7. Data manipulation, input and output
   1. I can import data from a text file which has a standard format (e.g., comma-separated or tab-separated values).
   2. I can export data to a text file which has a standard format.
   3. I can filter and sort data.
8. Plotting data, quantities and functions
   1. I can plot 1D and 2D continuous functions and discrete data. This includes being able to superimpose multiple plots on the same canvas (e.g., visual comparisons between data and models).
   2. I can use graphical solutions to solve problems, such as simultaneous equations problems.
   3. I can modify the important parameters needed to make a graph “nice”. This includes setting axis limits, adding axis labels, changing line or point styles, making semi-log and log-log plots, plotting error bars.
   4. I can create and interpret 2D, 3D, and density/image/false-colour plots.
   5. I can create vector-field plots.
   6. I can plot solutions to ODEs and PDEs.
9. Data Analysis
   1. I can compute the average, standard deviation, median, etc., of a data set.
   2. I can fit a model (function) to data using weighted and unweighted chi-square minimization. I can extract the best fit parameters and their errors. I can use reduced chi-square and the scatter of the residuals to evaluate the “goodness of fit”.
   3. I can perform a Fast Fourier Transform (FFT). This includes being able to account for the Nyquist frequency, normalization of the FFT, converting a FFT into a power spectrum or spectral density and performing an inverse FFT.
10. Monte-Carlo methods
    1. I can use Monte-Carlo methods to model systems which involve random processes.
    2. I can use Monte-Carlo methods to perform error propagation.
    3. I can use Monte-Carlo methods to perform integration
11. Animation
    1. I can animate a physical system.
12. Linear algebra
    1. I can perform typical matrix operations such as addition, multiplication, transposition, etc.
    2. I can find eigenvalues and eigenvectors.
    3. I can perform matrix decomposition and reduction
13. Mathematica-specific standards
    1. [ungraded] The “/.” command.
    2. I can use the Manipulate command.
    3. I can use patterns as part of recursion and case.
14. Documentation (Portfolio-based: the student must choose and submit their three best examples for each documentation standard)
    1. I can document the use of a function. This is specific to only the details of what goes in and out of the function, not the nuts and bolts of what the function does.
    2. I can use sectioning and big-picture documentation to communicate the overall use of a program as well highlighting and describing the purpose of the major sections of the program.
    3. I can use documentation to clearly explain how a complex chunk of code (such as a function) works.
15. Project [these are to be expanded into multiple or more heavily weighted standards]
    1. I can model a complicated physical system.
    2. I can write an effective project report using LaTeX.

Some issues with the above standards

1. Some of the linear algebra stuff overlaps with some of the computer algebra stuff (vectors and matrices).
2. I would like the standards to be similar in scope to each other, but they are nowhere near that right now.

The feedback I am looking for

I would love to hear any and all feedback you might have, either in the comments below, or you can comment on a google-docs version of this post that I made publicly commentable. Some specific things I have in mind are…

What are the most important or least important standards on the list? If you have any opinions on which ones seem less essential or which ones seem absolutely essential, I would love to hear about it.

How can I improve my proposed assessment of the standards? Do you have any suggestions for alternate ways of assessing the standards at the individual level for work done by a group in this context?

Not this round

I have a very heavy teaching load in the fall (3 new-to-me upper-division courses) so I am trying to figure out appropriately sized chunks of ambition for each. I have lots of ideas for each course, but there is no way that I am going to find the time to do a decent job of everything I have in mind. With that in mind, you will notice that there is no emphasis on sense-making or interpretation in the standards or in the way the standards are assessed. I think Danny Caballero and colleagues are doing some fantastic work with this at CU, and I really want to fold these thing into the course in the future, but for this round these are going to have to be things that I bring up repeatedly in class, but don’t explicitly build into the course. Baby steps Ives. Baby steps.

My busy summer should provide plenty of blog fodder

The world seems to have conspired to keep me delightfully busy this summer. Fortunately most of the activities that will be keeping me busy are well-suited to posting about on this blog. I will have to make a concentrated effort to find the time for blogging, but it does seem to always be worth the time even when I’m insanely busy (I must remember that!).

Fall courses

First up is that I will be teaching three third-year courses in the fall, all of which are new courses to me. These courses are

• A computational physics course where I will be getting the students to learn the basics of python and Mathematica, but letting them choose which tool they want to use for the more advanced tasks.
• A digital electronics course (lecture + lab = 2 courses) where I will be using Field Programmable Gate Arrays (FPGAs) throughout the entire lab portion of the course to build upon the hand-wiring of individual components. This will be my first time using FPGAs.

I’m planning to run the lecture part of the course in the same way I run my usual “lecture” courses: pre-class assignments, whiteboarding and clickering during class time, weekly group quizzes, etc. I’m also trying to figure out how much I can integrate the lecture and lab into a more cohesive whole. The lecture and lab are co-requisite courses where the students receive separate final grades for each part so integrating them will prove to be a bit finicky.

I’m also gearing up to use some flavor of SBG in both the computational physics course and the digital lab. I will definitely be posting about the development for both of those in hopes of getting some feedback.

Conference presentations

I am delighted to have been invited to give a talk at FFPER-PS 2012. The FFPER conferences are my absolute favorite (I affectionately refer to them as Physics camp) so giving an invited talk there is extra special to me. I’m going to present on my work looking at the effectiveness of group quizzes. The main focus of my current analysis is trying to tease out the evidence or lack of evidence for learning that takes place on the group quizzes translating to success on similar questions at a later time, such as the final exam.  One of my summer students and I are trying to get through the analysis on this one as quickly as possible since my presentation is only a couple of weeks away.

I am also giving a short contributed talk at the summer AAPT meeting on the “kinder, gentler oral exams” that I used in my Advanced Lab course this past year. I will also be bookending the AAPT meeting with PERC and the Conference on Laboratory Instruction Beyond the First Year of College, but I decided that my plate was too full this summer to try to present anything at those despite the fact that I have some great stuff that I would like to contribute.

Stay tuned.

Learning Before Class Strategies Part 2.5: Types of Video Lectures

A screen capture of a smartPhysics multimedia lesson

This is part 2.5 (the second half of part 2) of a series on pre-class learning strategies, which can be used as part of the flipped or inverted class. In this post I will discuss the different types of video lectures.

• Part 1 focused on some common types of assignments/assessments that you can use.
• Part 2 discussed the various types of resources you can provide your students for use in learning before class strategies.
• Part 2.5 is this post
• Part 3 will discuss some tips and some issues that I have come across trying to implement learning before class strategies.
• I also had a quick update post that pointed to some recent articles/blog posts by others on the subject of pre-reading assignments.

End

Learning Before Class Strategies Part 2: Types of Resources to Provide Your Students

This is part 2 of a 3 part series on pre-class learning strategies, which can be used as part of the flipped or inverted class. In this post I will discuss some of the types of resources that you can provide your students to do their learning before class.

• Part 1 focused on some common types of assignments/assessments that you can use.
• Part 2 is this post!
• Part 2.5 will discuss the types of video lectures in a bit more detail (this post was getting long)
• Part 3 will discuss some tips and some issues that I have come across trying to implement learning before class strategies.
• I also had a quick update post that pointed to some recent articles/blog posts by others on the subject of pre-reading assignments.

Photo by Kevin Dooley via Lifehacker

What I want out of these learning resources

Common types of learning resources to provide your students

These are types of learning before class resources that I have tried and I think is a fairly comprehensive list. Some of these are more well-suited to specific types of assignments (see Part 1) than others.

1. Textbook - Despite everything I say above about textbooks (thus far) not being something that does a good job of being a resource for both first contact and reference, I do like Knight‘s intro physics textbook for being something that is quite readable for the student, even for first contact between the student and a new topic. Of course there are still many text elements (paragraphs, sections and examples) that I would ask the students to skip due to the amount of “mastering the basics” in class that is needed before this skipped content would be meaningful to any student other than the rare one that was determined to make sense of everything they they encountered in the textbook.
2. Video lecture – These come in multiple flavours. Screencasting is becoming increasingly common, but there is also pencasting, highly-produced multimedia presentations like those found in smartPhysics, and video-recording a regular lecture. I will discuss each of these a little bit more in Part 2.5 of this series, but just comment on them as a group here. One of the strengths of the video lecture format is that it doesn’t take too much time to produce something coherent and of high enough quality that the students will find it useful. Anecdotally, I have found that students are much more willing to wrestle with slightly more challenging content in the video format than if I had just asked them to read very well presented notes on the same topic. Of course this is exactly why many (most?) students  are happy to attend traditional lectures, but would never think of reading the textbook covering the exact same content.
3. Simulations - Inspired by Noah Podolefsky’s Global Physics Department talk on PhET simulations I have started basing some of my learning before class assignments on simulations. What I usually do is ask them to play with the simulation for 5-10 minutes and then send me 3 questions they had, things that they discovered or things that they found interesting in that time. After trying this type of simulation-based assignment out a few times I am finding that the students tend to generate questions that touch on most of the important points you would want to touch on, but instead frames the classroom discussions in terms of their curiosity instead of you telling them what’s important (even though it is the same actual content). The feedback I got from them indicated that they really seemed to like this type of pre-class assignment. If you don’t want to just let them play and tell you about it, you can try to focus their attention on specific things by asking them investigation types of questions like “what parameters affect X?”.
4. Other targeted written resources - This is a grab bag category much like the video lecture one. This category includes
   • Your own written work meant to present the content at a level similar to a screencast or pencast that you would produce. A good example is the series of blog posts that Rhett Allain turned into Just Enough Physics;
   • Sections of textbooks that are targeted at a lower level than the given course. I often discovered when teaching Quantum that pointing my students to Knight’s discussions on the same topic would have been a goof “first encounter with a topic” resource or for intro physics I pointed my students toward physics for future presidents as an additional resource (before the book was published, all the chapter PDFs were available on that webpage).

   In many ways I prefer written resources to screencasts because (a) it is much easier to make small edits, and (b) I find it easier to piece together a few of them into a cohesive narrative. On the flip side, it is a lot less work to produce a screencast of acceptable quality than getting bogged down in writing something of acceptable quality. Or as Andy always say “I speak faster than I write.”

5. Materials meant specifically to generate interest –  I have only tried it out once, but I was happy with how it worked out. You can give them a popular science or journal article, a chapter from a popular science book, a video, etc.  This one goes really well with the student generated question type of pre-class assignment (see Post 1 in this series) and then functions quite similar to the simulation-based pre-class assignments by letting the student questions frame the classroom discussion.

Part 2.5 so very soon.

Learning Before Class Strategies – Quick Update

I am just about finished writing Part 2 of me Learning Before Class Strategies series of posts (Part 1 here), but I thought I would post links to a couple of recent posts on the subject of pre-reading assignments and an article from a recent UBC CWSEI Physics and Astronomy newsletter.

But first, here is a look at the relationship between exam averages and the marks my students earned for completing their smartPhysics Prelecture and Checkpoint assignments (marked for effort only) in my Fall 2011 introductory mechanics course. Not terribly compelling in terms of trying to sell my intro E&M students on the pre-class assignments. Fortunately Peter (see link below) gave me a bunch of good motivational ammo to pass on to them.

Sigh, the person that completed the least pre-class assignments had over an 80% average on exams.

Recent posts/articles of note on pre-reading assignments

Which courses are essential in a Physics degree?

My department’s physics majors degree is very minimally prescribed compared to most places. Our students take the standard Mechanics and E&M in the intro sequence and I am putting my question out there for everything beyond that being up for grabs. Which topics, skills or courses are the ones that you think a student absolutely should have if they are to receive a piece of paper saying that they have a college physics degree.

This is my personal list and it is meant to cover either experimental or theoretical interests so there are no real experimental requirements and the theory is as much as an experimentalist would need.

Must have at least intro textbook level

These are topics that the really thick (“with modern physics”) intro textbooks cover at a sufficiently high level that they prevent the students from having severe gaps in their general physics knowledge. These topics show up in 2nd year courses in most programs if they were not part of the intro sequence.

• Mechanics (including intermediate topics such as forced and damped oscillation, but these are covered in the thick intro texts)
• Circuits
• Geometric and wave optics
• Thermodynamics
• Relativity
• Wave-particle duality
• Nature of the atom

Must have at least one upper-year course

Every person with a recent physics degree should take at least one upper-year course in these topics or the majority of physicists would consider this person to have severe gaps in their physics knowledge. These are in addition to the very important math topics of vector calculus, and ordinary and partial differential equations.

• Quantum Mechanics
• Electricity and Magnetism

It seems crazy that a physicist might not have this course (or skill), but I guess they don’t HAVE TO have it

• Classical Mechanics
• Solid state physics
• Statistical Mechanics
• Standard Model
• Skills: computational modeling, experimental design

So my list of must have upper-year courses is only two. It was hard to move stuff like solid state and stat-mech to the “should” from the “must” list, but I did.

What did I miss? What did I put on there that shouldn’t be?

Update September 1, 2011 – Chad Orzel has posted a poll on this very topic using example textbooks to demonstrate the level of the course.