Once I started writing this it got pretty long so I will call this part 1 and work on part 2 another day.
A month ago I took a couple of my students to a local coffee shop, filled them full of treats, and poked their brains for a couple of hours about my Advanced Laboratory course that ended in April, 2012. I’m summarizing their feedback here to make sure I have a good record of their feedback and, of course, to share. For any given piece of feedback from the students I will try my best to explain the context and by the end you should have a pretty good idea of how the course looked and where it will be headed in the future.
Let’s start with my definition of an Advanced Laboratory course from an earlier post:
This type of course, a standard course in most physics departments, is a standalone lab course without any associated lecture course. There is an amazing amount of variability from one Advanced Lab course to the next and they range in format from one experiment per week with everything already set up and cookbook procedures ready to be followed, to a single student-developed project over the entire term (or year!).
In my specific incarnation, we spend the first month doing some introductory activities to build up some foundational skills which are mostly related to data analysis and presentation. For the rest of the course pairs of students work on two month-long experimental physics projects. The students are guided to work on projects that can be viewed as being part of a larger research line, where they build on the work of previous students and future students will build on their work. Thus no two groups will ever perform identical experiments.
Onto the feedback!
Weekly research group meetings
Each week we had a research group meeting where each group was asked to post a couple of figures or tables to the course wiki and quickly bring us up to speed on what they had done the previous week as well as what they planned on doing the next week. A very small part of their grade was based on how much they contributed to these research group meetings. My expectation was that, averaged over multiple meetings, each student would ask at least one meaningful question to the presenters per meeting or contribute in some other way to the conversations surrounding the projects of the other groups. I had twelve students enrolled so I split the class into two larger research groups so each research group consisted of myself and three pairs of students.
I was quite happy with how these meetings worked and felt it was really valuable for the presenters to have to frame things so that people other than me and the presenters actually understood what they were up to. Anecdotally it felt like students spent more of their initial time learning about the basics (experiment and theory) of their projects than in the past because they were going to have to explain things to somebody else.
Interview feedback: the meetings felt too long. Their initial suggestion was to make the meetings every other week or to time-limit them somehow. A research group meeting took somewhere between a half-hour and an hour each week. The actual presentations were reasonably concise, but there were always lots of questions from the other groups and feedback from me. This was also the place where we hashed out a lot of the gory details of what they should try to do in the coming week. But the thing was that the other groups often contributed to these discussions on what to tackle in the coming week so I felt like the whole process was extremely valuable for all parties. But it could probably be tightened up.
Future plan: Partners will alternate being the main presenter each week (previously they were both expected to present each week) and will be asked to present 1 or 2 tables of figures. The feedback was that it sometimes felt like a stretch to find that 2nd table or figure to present. The actual presentation will be limited to 5 minutes for a total of 10 minutes per project group between the presentation and the discussion afterwards. I won’t be strict on the time limits, but will be mindful of the clock to help prioritize which discussions to have at that moment and which ones can be saved for private discussions later. One of the students also suggested that having time limits on their presentation time would serve as good practice for their formal oral presentations later in the course and these did have strict time limits.
Twice during the intro sequence I tried to have a number of small groups working on different things and then had them report out to the class. The second time we did this I called them “parallel investigations” and sent them off to study goodness of fit, Monte-Carlo methods for fitting or Monte-Carlo methods for error analysis. In addition to orally reporting out their findings, I asked one partner to write their findings up in LaTeX and the other on the course wiki. These two write-ups were allowed to be identical and the reason that I used two formats was because one of the students was going to write up the background and theory for their first project on the wiki and the other was going to write up the analysis and results as a LaTeX “tech note”. Thus I wanted them to have some practice using these writing formats. Note that on the second project they were to switch who wrote on the wiki and who wrote the LaTeX “tech note”.
Interview feedback: this might not have been the best use of their time. Yup, I agree. In the end, for the investigations that they did not perform, each group was simply ont he receiving end of a mediocre lecture on the topic and never got a chance to actively engage with the ideas. This is the exact opposite of how I try to run my courses.
Future plan: I’m planning on completely restructuring how the first-month introductory stuff works and will talk about that a bit more later, but I think the parallel investigations idea as it existed is officially dead.
Introduction to LaTeX
This is a place where I have not offered my students a ton of support. I wrote a tutorial so that they can install miktex and texniccenter on their windows machines and gave them a couple of sample LaTeX documents that cover all the basics, but that’s it.
Interview feedback: they would like some coherent instruction and resources. For most of them this is their first time ever dealing with a markup language and the learning curve seems to be steeper than I have been admitting to myself.
Future plan: It looks like a crew of us on twitter are going to put together a LaTeX workshop for the Summer 2013 AAPT meeting and I am hoping that as part of this process we will have put together a straight forward introduction to LaTeX for physicists package that I can drop on my students like a big ol’ pile of awesomeness.
From Student-Generated Scientific Inquiry (Leslie Atkins and Irene Salter) I used their lab notebook activity which uses pages taken from different famous scientists actual lab notebooks. The students are asked, in small groups, to take some notes on how these famous scientists took notes and organized their information. As a large group we then built a rubric for their lab notebooks based on their observations of the pages from the notebooks of the famous scientists. The students were highly engaged in this activity and seemed to be supportive of the rubric that we developed from this activity.
Interview feedback: they thought this activity was great. But they didn’t find it in the way that I expected. The two students I interviewed both had some previous experience with lab notebooks in research labs and had, in the past, put way too much emphasis on maintaining an immaculate lab notebook. This activity had let them know that it was OK to have rough notes in their lab notebook.
Future plan: I hate marking lab notebooks. It is the worst. And with so much of the work they do being digital these days it is really hard to find a solution that fits into their work flow and doesn’t involve pasting umpteen print-outs into their lab notebook. I’m actually planning on backing off of trying to get them to keep a really good lab notebook and emphasize getting them to report at the beginning and end of the day what they planned on doing and what they actually accomplished (science fair style!). I will be checking it every class period and it will be graded as complete or incomplete. Once I feel that I can get a group of students doing a consistently good job of this, I will consider the next step to take.
I have three new-to-me courses that I am teaching this fall: comp-phys, digital electronics lecture and digital electronics lab. I am sharing the syllabus for my digital electronics “lecture” course below, but have removed a few things which are only relevant internally.
UFV Physics 362 – Digital Electronics and Computer Interfacing Syllabus (V1) – Section AB1, Fall 2012
About this course
In addition to learning about digital electronics, one of the main goals of this course is to help you develop as a lifelong independent learner. Robert Talbert puts it much better than I ever could (http://goo.gl/ZIh0R):
“As you move through your degree and eventually into your career and your adult post-college life, your main value to the rest of the world and to the people you love is your ability to learn and grow without needing other people around to make it happen. There are many times in life where you MUST learn something, and you can’t wait for the next semester at the local college to come around for you to sign up for a course. You have to take charge. You have to learn on your own.”
This course is structured around the idea that you will do some initial learning on your own before you come to class and then in class you will work with your peers to deepen your understanding. You will be doing the heavy lifting in class instead of just watching me do examples and derivations on the board (do you remember how proficient you became at sword-fighting by watching the Princess Bride?). Some students find this very disorienting and some of you will find that this course structure will take you out of your normal comfort zone. The best thing you can do is come into the course with a positive attitude and be prepared to tweak your normal recipes for success to be able to get the most out of this course.
Please note that this course has a corequisite lab (Physics 372) which will focus on the hands-on aspects of digital electronics as well as the interplay between theory and hands-on applications.
Course Description (from the official outline)
This course emphasizes elementary digital electronics and interfaces. Topics include gates and Boolean algebra, Karnaugh maps, flip flops, registers, counters and memories, digital components, microprocessor functions and architecture, instruction sets, D/A and A/D converters, and waveshaping. PHYS 372, the laboratory portion of this course, must be taken concurrently. This course is designed to provide practical experience with the basic digital logic chips and how digital circuits can be interfaced with microprocessors.
Note that we will co-construct a proper set of detailed learning goals as we proceed through this course and those detailed learning goals will define what sort of questions can be asked on the quizzes and the final exam. The learning goals listed below, which are from the official course outline, are meant to be very broad and as such only provide a very rough framework in which we will fit all the fun that is digital electronics.
Learning goals from the official course outline: This course is designed to provide students with:
- the theory needed to understand the purpose and how digital devices function;
- an understanding and an appreciation of how a digital computer functions;
- the ability to design, construct and test simple digital logic circuits;
- an ability to program the common microprocessors;
- how information can be transferred to and from computers.
Tony R. Kuphaldt, Lesson in Circuits: Volume IV – Digital, http://openbookproject.net/electricCircuits/Digital/index.html
In addition to this online textbook, I will leave a nice big pile of electronics textbooks in A353 for your use. As a group we can sort out a reasonable scheme for lending out these books while making sure that they are still available to everybody.
Pre-class Assignments: The engine that drives this course is the collection of Socratic Electronics worksheets. For each worksheet, you will be assigned to research and answer a subset of the questions. In class you will present your findings in small and large groups. The goal is for you to learn how to locate information, problem-solve, collaborate, and clearly articulate your thoughts while learning about digital electronics. The answers to all the questions will be provided with the worksheets, so it is the solutions in which I am most interested and for which you are responsible in your preparation.
Class Periods: I run each class period under the assumption that you have completed the relevant pre-class-assignment and have made a genuine effort to make some sense of the material before showing up to class. We will use class time to help you clarify your understanding of the material and to build on the core ideas that you wrestled with in your pre-class assignments. In class you will mostly be working in small groups. Not all members of a group will have been assigned the exact same pre-class questions, so the first thing that you will do is present your own findings and come to group consensus on the solutions. In class I will also ask you to work on additional questions from the worksheets as well as other additional questions which I will provide. At appropriate times, I will provide mini-lectures to clarify ideas or to plant the initial seed for an idea which you will be studying on an upcoming pre-class assignment.
Peer and instructor assessment of pre-class and in-class work: Each class period you will be given a number of contribution points to spread among the rest of your group (not including yourself) based on how much their pre-class preparation and in-class work contributed to your group’s overall learning in class. The exact number is 8*(N-1)+1, where N is the number of students in your group. You can give any individual student up to 10 points and do not have to give out all of your points. I will average the points assigned to you by the rest of your group for each class period. If needed, I may adjust this average up or down by up to a couple of points if I feel that your class period average is a very poor match to my own observations of your pre-class preparation and in-class contributions. I prefer not to have to make any adjustments this way and will very clearly spell out for you what factors I have considered when adjusting this daily class period average. I will drop your five worst daily class period averages when calculating your final mark for this category.
Homework: Nope, but I will make sure that you have sufficient resources for quiz and exam preparation.
Quizzes: Roughly every two weeks we will use the entire class period to have a quiz, for a total of 5 quizzes over the course of the term. The quiz will be split into two pieces: a solo quiz and a group quiz. You will first write the solo quiz and then approximately 2/3rds of the way through the class period I will collect the solo quizzes and then get you to write the group quiz, typically in groups of 3 or 4. The group quiz will mostly be the same as the solo quiz, but will often have some extra questions. If you score higher on the solo quiz than the group quiz, I will use your solo quiz mark when calculating your overall group-quiz average.
Quiz Averages: I will use your best 3 of 5 group quiz scores when calculating your overall group-quiz average. Things are a little more complicated for your overall solo-quiz average. In addition to the three-hour final exam, I will be creating five different half-hour-long re-tests, one for the material covered on each of the five quizzes during the term. You can choose to write two of these re-tests as part of the final exam and your mark from each of those re-tests can replace your earlier mark on the corresponding quiz (including if you missed the earlier quiz completely). The catch here is that I will only allow you to write a given re-test if you demonstrate to me that you have put in a reasonable amount of effort to learn that material. I will expect you to make your case by presenting me with the specific things that you did to learn the material and that you did to learn from your mistakes on the initial quiz.
|Peer and instructor assessment of pre-class and in-class work:||
Tentative Course Schedule
The numbers Sxx indicate the worksheet number for that day’s worksheet. The worksheets can be found at http://www.ibiblio.org/kuphaldt/socratic/doc/topical.html
|Sept. 3||D01 – Numeration Systems (S04)||No class||Classes begin Sept. 4.|
|Sept. 10||D02 – Binary Arithmetic (S05)||D03 – Digital Codes (S06)||D04 – Basic Logic Gates (S03)|
|Sept. 17||D05 – TTL Logic Gates (S07)||D06 – CMOS Logic Gates (S08)||No class|
|Sept. 24||Quiz 1||D07 – Trouble Gates (S09)||D08 – Boolean Algebra (S13)|
|Oct. 1||D09 – Sum-of-Products and Product-of-Sums Expressions (S14)||D10 – Karnaugh Mapping (S15)||No class|
|Oct. 8||Thanksgiving. No classes.||D11 – Binary Math Circuits (S16)||Quiz 2||Wednesday Oct. 10 is last day to withdraw without W appearing on transcript.|
|Oct. 15||D12 – Encoders and Decoders (S17)||D13 – Multiplexers and Demultiplexers (S18)||No class|
|Oct. 22||D14 – Latch Circuits (S21)||D15 – Timer Circuits (S22)||Quiz 3|
|Oct. 29||D16 – Flip-flop Circuits (S23)||D17 – Counters (S26)||No class|
|Nov. 5||D18 – Shift Registers (S28)||Quiz 4||Remembrance day. No classes.|
|Nov. 12||D19 – Digital-to-Analog Conversion (S30)||D20 – Analog-to-Digital Conversion (S31)||No class||Tuesday Nov. 13 is the final day to withdraw from courses.|
|Nov. 19||D21 – Memory Devices (S34)||D22 – Optional Topics (see notes)||D23 – – Optional Topics|
|Nov. 26||Quiz 5||D24 – Optional Topics||No class||Potential topics include digital communication, micro-controllers, state machines and electro-mechanical relays.|
|Dec. 3||D25 – Optional Topics||Monday Dec. 3 is the last day of classes|
My 6 (nearly 7) year-old son and I made up a new game today. It’s a good math skills game. The game is based on putting tiles down so that the numbers on the tiles add up to multiples of 5.
Skills your child needs to have to play this game
- Adding 2-3 small double-digit numbers
These are the rules that we found made a game that didn’t drag on, but also meant that the winner wasn’t simply the first person to play. The winner is the first player to get rid of all of their dominoes.
- Place all the dominoes on the table face down.
- Each player draws 9 dominoes. Draw a single domino and put it face up in the middle of the table (the 10/10 in my example picture). The youngest player goes first.
- To play a tile, there are two options.
- The first option is to play a single side on a single side already played where the sum of these two sides needs to be a multiple of 5. Examples in the picture include the 0 played next to the 10 on the right-hand side or the 2 played next to the 8 on the middle of the left-hand side of the picture.
- The second option is to play both sides of a new domino (“sideways”) on a single side already played, where the sum of all three sides needs to be a multiple of 5. There are two examples in the picture: the 7/8 played on the 10 on the left-hand side and the 7/11 played on the 2 at the top.
- After a player plays a tile, it becomes the other player’s turn.
- If a player can’t play a tile on their turn, they draw a single tile from the face-down tiles and then it is the other players turn.
- A sideways domino cannot be played on a sideways domino. For example, a 2/3 tile could not be played sideways on the 7/8 tile at the bottom left. A 3, 8 or 13 could have been played on the 7, or a 2, 7 or 12 could have been played on the 8 as was done.
- The winner is the first player to play all of their tiles.
- Beginner rules – Get rid of all rules that involve sideways placements. That way you are only ever adding two numbers together
- Sums other than multiples of 5 – Instead of multiples of 5, try multiples of 7 or some other base that is worth practicing with your child.
- Double sideways - Allow a sideways tile to be played on a sideways tile so that the multiple of 5 comes from summing all 4 sides.
One of my brief studies, based on data from a recent introductory calculus-based course, was to look at the effect of immediate feedback in an exam situation. The results show that, after being provided with immediate feedback on their answer to the first of two questions which tested the same concept, students had a statistically significant improvement in performance on the second question.
Although I used immediate feedback for multiple questions on both the term test and final exam in the course, I only set up the experimental conditions discussed below for one question.
The question I used (Figure 1) asked about the sign of the electric potential at two different points. A common student difficulty is to confuse the procedures of finding electric potential (a scalar quantity) and electric field (a vector quantity) for a given charge distrubution. The interested reader might wish to read a study by Sayre and Heckler (link to journal, publication page with direction link to pdf).
Experimental design and results
There were three versions of the exam, with one version of this question appearing on two exams (Condition 1, 33 students) and the other version of this question appearing on the third exam (Condition 2, 16 students). For each condition, they were asked to answer the first question (Q1), using an IFAT scratch card for one of the points (Condition 1 = point A; Condition 2 = point B). With the scratch cards, they scratch their chosen answer and if they chose correctly they will see a star. If they were incorrect, they could choose a different answer and if they were correct on their second try, they received half the points. If they had to scratch a third time to find the correct answer, they received no marks. No matter how they did on the first question, they will have learned the correct answer to that question before moving on to the second question, which asked for the potential at the other point (Cond1 = point B; Cond2 = point A). The results for each condition and question are shown in Table 1.
|Q1 (scratch card question)||Q2 (follow-up question)|
|Condition 1||Point A: 24/33 correct = 72.7±7.8%||Point B: 28/33 correct = 84.8±6.2%|
|Condition 2||Point B: 8/16 correct = 50.0±12.5%||Point A: 10/16 correct = 62.5±12.1%|
Table 1: Results are shown for each of the conditions. In condition 1, they answered the question for point A and received feedback, using the IFAT scratch card, before moving on to answer the question for point B. In condition 2, they first answered the question for point B using the scratch card and then moved on to answering the question for point A.
So that I can look at the improvement from all students when going from the scratch card question (Q1) to the follow-up question (Q2), I need to show that there is no statistically significant difference between how the students answered the question for point A and point B. Figure 2 shows that a two-tailed repeated-measures t-test fails to reject the null hypothesis, that the mean performance for point A and B are the same. Thus we have no evidence that these questions are different, which means we can move on to comparing how the students performed on the the follow-up question (Q2) as compared to the scratch card question (Q1).
Figure 3 shows a 12.2% improvement from the scratch card question (Q1) to the follow-up question (Q2). Using a one-tailed repeated-measures t-test (it was assumed that performance on Q2 would be better than Q1), the null-hypothesis is rejected at a level of p = 0.0064. Since I have made two comparisons using these same data, a Bonferroni correction should be applied. The result of this correction is there were statistically significant differences at the p = 0.05/2 = 0.025 level, which means improvement from Q1 to Q2 was statistically significant.
In additional to reproducing these results using multiple questions, I would also like to examine if these results hold true for some different conditions. Additional factors which could be examined include difference disciplines, upper-division vs. introductory courses and questions which target different levels of Bloom’s taxonomy.
Note: I found a paper that looks at the effect of feedback on follow-up questions as part of exam preparation and discuss it in more detail in this follow-up post.
Let me start off by saying that, as a student, I found oral exams to be very intimidating and frustrating. I could see their value as assessment tools, but found that in practice they were simply a source of personal dread. Enter 2012 where I am using oral assessments with my own students, but what I have done is try to minimize what I found intimidating and frustrating about oral exams. I have made my oral assessments kinder and gentler.
The strengths of oral assessments
In my opinion, the strengths of oral assessments are a result of their interactive nature.
If a student is stuck on a minor point, or even a major one, you can give them a hint or use some leading questions to help them along. Compare this to what happens if a student gets stuck on a written exam question and you can see how the oral assessment provides you with a much better assessment of student understanding than an incomplete or nonsensical written response.
Another strength is that no ambiguity need be left unturned. If some sort of ambiguous statement comes out of a student’s mouth, you can ask them to clarify or expand on what they have said instead of dealing with the common grader’s dilemma of sitting in front of a written response trying to make judgement calls related to ambiguous student work.
Some other benefits are that marking is a breeze (I will discuss my specific marking scheme later) and I have also found that I can generate “good” oral exam questions much more quickly than I can written ones.
My perception of the weaknesses of traditional oral assessments
The following are common, but not universal characteristics of oral assessments.
Public –Looking dumb in front of me may not be fun, but it is far more comfortable than looking dumb in front of a room full of your peers or discipline experts. Having spent some time on both sides of the desk, I don’t feel that my students ever “look dumb”, but as a student I remember feeling dumb on many occasions (here I will also include comprehensive exams, dissertation defences and question periods after oral presentations in my definition of oral assessments). I guess I’m saying that it feels worse than it looks, but doing it in public makes it feel even worse.
A lack of time to think – This is actually my biggest beef with oral assessments. In a written assessment you can read the question, collect your thoughts, brain-storm, make some mistakes, try multiple paths, and then finally try to put together a cohesive answer. I realize that you can do all these things in an oral assessment as well, but there is a certain time pressure which hangs over your head during an oral assessment. And there is a difference between privately pursuing different paths before coming to a desired one and having people scrutinize your every step while you do this.
Inauthentic – By inauthentic, I mean that oral exams (and for the most part, written ones too) isolate you from resources and come with some sort of urgent time pressure. If we are confronted with a challenging problem or question in the real world, we usually have access to the internet, textbooks, journals and even experts. We are able to use those resources to help build or clarify our understanding before having to present our solution. On the flip side, we can also consider the question period after a presentation as a real-world assessment and we are usually expected to have answers at our fingertips without consulting any resources so arguments can be made for and against the authenticity of an oral assessment.
Context (Advanced Lab)
Before I break down my kinder, gentler oral exams, I want to discuss the course in which I was using them. This course was my Advanced Lab (see an earlier post) where students work in pairs on roughly month-long experimental physics projects. One students is asked to be in charge of writing about the background and theory and the other the experimental details, and then on the second project they switch. For their oral assessments I used the same set of questions for both partners, but the actual questions (see below) were very project-specific. My hope was that using the same questions for both partners would have forced them to pay much closer attention to what the other had written.
It took at most a total of 2 hours to come up with the 6 sets of questions (12 students total in the course) and then 9 hours of actual oral exams which comes out to less than an hour per student. I would say that this is roughly equivalent to the time I would have spent creating and marking that many different written exams, but this was much more pleasant for me than all that marking.
Kinder, gentler oral exams
I will describe the format that I use and then highlight some of the key changes that I made to improve on what I perceive to be the weaknesses of traditional oral exams.
I book a 45-minute time slot for each student and they come to my office one at a time. When they show up in my office I have 3 questions for them. They have 10 minutes to gather their thoughts and use whatever resources that they brought (including using the internet, but not consulting with somebody) to help formulate some coherent answers. I also give them a nice big whiteboard to use how they see fit. Once their 10 minutes are up (it is not uncommon for them to take a couple extra minutes if they want that little bit of extra time), they are asked to answer the questions in whatever order would please them. For each question I try, but not always successfully, to let them get their answer out before I start asking clarification, leading or follow-up questions. If they are on the completely wrong track or get stuck I will step in much earlier. If the leading questions do not help them get to the correct answer, we will discuss the question on the spot until I feel like the student “gets” the answer. Sometimes these discussions would immediately follow the question and sometimes I would wait until after they have had a chance to answer all three questions. After they have answered all three questions and we have discussed the correct answers, I pull out the rubric (see below) and we try to come to consensus on their grade for each question. They leave my office with a grade and knowledge of the correct answer to all three questions.
The key changes:
- Private – I have them come to my office and do the assessment one-on-one instead of in front of the whole class.
- 10 minutes to collect their thoughts and consult resources – It is similar to the perceived safety blanket offered by an open book exam. Students that were well-prepared rarely used the entire time and students that were not well-prepared tried to cram but did not do very well since I would always ask some clarification or follow-up questions. I have some post-course feedback interviews planned to learn more about the student perspective on this, but my perception is that the preparation time was helpful, even for the well-prepared students. It gave them a chance to build some confidence in their answers and I often delighted in how well they were able to answer their questions. I think that time also offered an opportunity to get some minor details straight, which is beneficial in terms of confidence building and improving the quality of their answers. And finally, knowing that they had that 10 minutes of live prep time seemed to reduce their pre-test stress.
- Immediate feedback – Discussing the correct answer with the student immediately after they have answered a question is a potential confidence killer. I suspect that the students would prefer to wait until after they have answered all the questions before discussing the correct answers, and I am interested to see what I will learn in my feedback interviews.
- Grading done as collaborative process with the student – In practice I would usually suggest a grade for a question and mention some examples from their answer (including how much help they needed from me) and then ask them if they thought that was fair. If they felt they should have earned a higher grade, they were invited to give examples of how their answer fell in the higher rubric category and there were many occasions where those students received higher grades. However, the problem is that this is a squeaky wheel situation and it is hard to figure out if it is entirely fair to all students. For cases where I asked students to tell me what grade they thought they earned before saying anything myself, students were far more likely to self-assess lower than I would have assessed them than to self-assess higher than I would have assessed them.
The grading rubric used was as follows:
|100%||Greatly exceeds expectations||The students displayed an understanding which went far beyond the scope of the question.|
|90%||Exceeds expectations||Everything was explained correctly without leading questions.|
|75%||Meets expectations||The major points were all explained correctly, but some leading questions were needed to help get there. There may have been a minor point which was not explained correctly.|
|60%||Approaching expectations||There was a major point or many minor points which were not explained correctly. The student was able to communicate an overall understanding which is correct.|
|45%||Below expections||Some of the major points were explained correctly, but the overall explanation was mostly incorrect.|
|30%||Far below expectations||Some of the minor points were explained correctly, but the overall explanation was mostly incorrect.|
Some example questions
- I would pull a figure from their lab report and ask them to explain the underlying physics or experimental details that led to a specific detail in the figure.
- “Run me through the physics of how you were able to get a current into the superconducting loop. Why did you have to have the magnet in place before the superconducting transition?”
- “Describe the physics behind how the Hall gave a voltage output which is proportional (when zeroed) to the external field. How do the external magnetic field and the hall sensor need to be oriented with respect to each other?”
- “Explain superconductivity to me in a way which a student, just finishing up first-year science, would understand.”
Electron-spin resonance experiment
- “Discuss how the relative alignment between your experiment and the Earth’s magnetic field might affect your results.”
- “In what ways did your detector resolution not agree with what was expected according to the lab manual? What are some reasonable steps that you could take to TRY to improve this agreement?”
Some other directions to take oral assessments
A couple of my blogger buddies have also been writing about using oral assessments and really like what they are up to as well.
Andy Rundquist has written quite a bit about oral assessments (one example) because they are quite central to his Standards-Based Grading implements. One of the things that he has been doing lately is giving a student a question ahead of time and asking them to prepare a page-length solution to the question to bring to class. In class the student projects their solution via doc-cam, Andy studies it a bit, and then he starts asking the student questions. To my mind this is most similar to the question period after a presentation. The student has had some time, in isolation, to put together the pieces to answer the question, and the questions are used to see how well they understood all the pieces required to put together the solution. Another thing that Andy does is gets the whole class to publicly participate in determining the student’s overall grade on that assessment. I love that idea, but feel like I have some work to do in terms of creating an appropriate classroom environment to do that.
Bret Benesh wrote a couple of posts (1, 2) discussing his use of oral exams. His format it closer to mine than it is to Andy’s, but Bret’s experience was that even if they knew the exam question ahead of time, he could easily tell the difference between students that understood their answers and those that did not. I really want to try giving them the questions ahead of time now.
One final note
I am giving a short AAPT talk on my kinder, gentler oral exams, so any feedback that will help with my presentation will be greatly appreciated. Are there certain points which were not, but should have been emphasized?
The successful applicant will hold a doctoral degree (obtained in the last ten years) and will be an outstanding emerging scholar who has demonstrated innovation and a proven ability to cultivate multidisciplinary, collaborative partnerships in local, national, and international research networks. The candidate must possess an original and independent research program in the general area of teaching and learning, the use of new teaching technologies and innovative pedagogical approaches relevant to the post-secondary education level.
The goals of the CRC program (www.chairs-chaires.gc.ca) are to promote leading edge research and the training of highly qualified personnel at universities.
This will be my first course in which I use some flavour of standards-based grading. This will also be the first time I have taught a computational physics course (or more accurately a computational thinking for physicists course). I do have some experience helping students develop some computational thinking skills from my Advanced Lab course, but I have never taught a full course dedicated to this topic. Fun!
The idea is that I will be getting them to work in Python and Mathematica in parallel. My main purpose behind this is to help the students see the platform-agnostic computational thinking which underlies their their computational tasks. A side benefit is that I can help Python and Mathematica put aside their differences and work toward a common goal.
This course will be based around a weekly cycle of
- Basic tasks, to be completed in both the Mathematica and Python computing environments; and
- Intermediate and advanced tasks, to be completed using either of the computing environments.
The grading part of SBG
Students will earn marks for each of the content standards and their final grade will come from points earned by demonstrating appropriate mastery of these standards. I will give the students example tasks that correlate to the standards, but the students will always have the option of performing any computational task that they wish that shows appropriate mastery of the standards.
Approximately 2/3rds of the way through the course, the students will start working on a project which models a complex physical system. I have two standards associated with this project (a physics one and a communication one) and I am planning to either weight these standards more heavily than the others or make them into a set of more fine-grained standards. I really haven’t decided on this.
Assessing an individual standard
I plan to assess each standard on a 0-4 scale. I’m not in love with my use of the term “completed” below when discussing tasks, but the idea is that the most important thing to me is that their programs do exactly what they are meant to do.
I think that I will be allowing students to partner up which means that I need a mechanism to assess the individual beyond just a working (and properly commented) program. I think that the individual assessment will be for them to orally run me through their properly working program. Students will be given the option of submitting screencasts to do this, but not all students will have access to a computer that has both a mic and Mathematica so I will need to keep the in-person explanations as a method.
Here’s a rough outline of how I plan to assess an individual standard, where each
- 4 – Exceeds expectations (student must make the case that they have exceeded expectations or have successfully completed an advanced task)
- 3 – Meets expectations (student has completed the relevant intermediate tasks or otherwise demonstrated intermediate-level mastery in either environment)
- 2 – Approaches expectations (student has completed the relevant basic tasks or otherwise demonstrated basic-level mastery in both environments)
- 1 – Doesn’t meet expectations (student has completed the relevant basic task or otherwise demonstrated basic-level mastery in one of the environments)
- 0 – Not yet assessed
- Environment is the general term to describe either Mathematica or Python.
- Program is the general term to describe a Python program/script or a Mathematica notebook.
- To get a 3 or 4 you need to have shown proficiency at the 2 level (basic) for both environments. If basic proficiency is only shown for one environment, the score on that standard is reduced by 1.
- To be eligible for reassessments, a 1 must be earned on a given standard within the first two weeks of the standard being opened.
- Right now the connections to physics are not built in, but that is in the long-term goals for the course.
Very loose weekly plans
We will spend roughly one week on each of the following broad themes.
- Introduction to the environment (functions, variable types)
- Iteration basics and animation (introductory modeling: similar to the early Matter and Interactions VPython stuff)
- File input/output, basics array manipulation and case structures
- Advanced list/array operations and manipulation
- Data visualization and plotting (histograms, scatter-plots, bar charts, error bars, etc)
- Data analysis (basic statistical analyses, fitting)
- Solving complex algebraic equations, integration and differentiation
- Solving differential equations
After the students have started working on their projects we will spend less than half of our class time working on non-project topics.
- Introduction to the physics modeling project
- Monte Carlo methods
- Numerical methods
- Linear algebra
These standards are based on a collection of computational physics learning goals that were put together by Andy Rundquist, Danny Caballero and Phil Wagner. I then went around to all the faculty in my own department and asked them what skills they would like to see the students develop as part of this course and folded the common ones in.
There are some standards marked as “[ungraded]“. The idea with these is that they are things which do not seem to be worth assessing for one reason or another, but are things that I still want to highlight for the students as being important.
Onto the standards…
- Environment fundamentals
- [ungraded] I can use online and built-in help resources
- Computer algebra fundamentals
- I can represent and perform common operations with complex numbers and vectors.
- I can use built-in integration and differentiation functions. This includes using assumptions for integration.
- I can access mathematical and physical constants or define these constants globally when not available.
- I can solve algebraic equations. This includes simplifying algebraic results and using a root finder. I can use graphical or other techniques to set appropriate neighborhoods for the root finder.
- Programming fundamentals
- [ungraded] I can assign and clear variables.
- I can write and use robust functions. The important characteristics of a robust function are (1) that they need no access to parameters outside of those which they were passed; (2) they are written such that they can easily be copied into any script or notebook and be used as intended; and (3) they can return information in the form of single parameters, vectors, arrays or other useful objects.
- I can use at least two different iteration methods to accomplish the same task.
- I can use case structures.
- Arrays and lists
- I can manipulate and slice arrays/lists. Slicing an array means to pick out columns or rows from larger arrays. Manipulation of an array includes transposing the array, replacing elements, and adding columns/rows to existing arrays
- [The wording needs work on this one] I can operate on entire arrays/lists instead of having to operate on the individual elements of the array/list.
- Numerical techniques
- I can write my own code (not call existing functions) to perform numerical integration with varying levels of precision (Trapezoidal rule, Simpson rule)
- Solving differential equations
- I can solve ODEs (1st Order, 2nd Order and Coupled) analytically. I can determine if an analytic solution exists.
- I can solve ODEs numerically. I can set initial conditions and the domain of the solution.
- I can solve a Partial Differential Equation and specify the boundary conditions appropriately.
- Data manipulation, input and output
- I can import data from a text file which has a standard format (e.g., comma-separated or tab-separated values).
- I can export data to a text file which has a standard format.
- I can filter and sort data.
- Plotting data, quantities and functions
- I can plot 1D and 2D continuous functions and discrete data. This includes being able to superimpose multiple plots on the same canvas (e.g., visual comparisons between data and models).
- I can use graphical solutions to solve problems, such as simultaneous equations problems.
- I can modify the important parameters needed to make a graph “nice”. This includes setting axis limits, adding axis labels, changing line or point styles, making semi-log and log-log plots, plotting error bars.
- I can create and interpret 2D, 3D, and density/image/false-colour plots.
- I can create vector-field plots.
- I can plot solutions to ODEs and PDEs.
- Data Analysis
- I can compute the average, standard deviation, median, etc., of a data set.
- I can fit a model (function) to data using weighted and unweighted chi-square minimization. I can extract the best fit parameters and their errors. I can use reduced chi-square and the scatter of the residuals to evaluate the “goodness of fit”.
- I can perform a Fast Fourier Transform (FFT). This includes being able to account for the Nyquist frequency, normalization of the FFT, converting a FFT into a power spectrum or spectral density and performing an inverse FFT.
- Monte-Carlo methods
- I can use Monte-Carlo methods to model systems which involve random processes.
- I can use Monte-Carlo methods to perform error propagation.
- I can use Monte-Carlo methods to perform integration
- I can animate a physical system.
- Linear algebra
- I can perform typical matrix operations such as addition, multiplication, transposition, etc.
- I can find eigenvalues and eigenvectors.
- I can perform matrix decomposition and reduction
- Mathematica-specific standards
- [ungraded] The “/.” command.
- I can use the Manipulate command.
- I can use patterns as part of recursion and case.
- Documentation (Portfolio-based: the student must choose and submit their three best examples for each documentation standard)
- I can document the use of a function. This is specific to only the details of what goes in and out of the function, not the nuts and bolts of what the function does.
- I can use sectioning and big-picture documentation to communicate the overall use of a program as well highlighting and describing the purpose of the major sections of the program.
- I can use documentation to clearly explain how a complex chunk of code (such as a function) works.
- Project [these are to be expanded into multiple or more heavily weighted standards]
- I can model a complicated physical system.
- I can write an effective project report using LaTeX.
Some issues with the above standards
- Some of the linear algebra stuff overlaps with some of the computer algebra stuff (vectors and matrices).
- I would like the standards to be similar in scope to each other, but they are nowhere near that right now.
The feedback I am looking for
I would love to hear any and all feedback you might have, either in the comments below, or you can comment on a google-docs version of this post that I made publicly commentable. Some specific things I have in mind are…
What are the most important or least important standards on the list? If you have any opinions on which ones seem less essential or which ones seem absolutely essential, I would love to hear about it.
How can I improve my proposed assessment of the standards? Do you have any suggestions for alternate ways of assessing the standards at the individual level for work done by a group in this context?
Not this round
I have a very heavy teaching load in the fall (3 new-to-me upper-division courses) so I am trying to figure out appropriately sized chunks of ambition for each. I have lots of ideas for each course, but there is no way that I am going to find the time to do a decent job of everything I have in mind. With that in mind, you will notice that there is no emphasis on sense-making or interpretation in the standards or in the way the standards are assessed. I think Danny Caballero and colleagues are doing some fantastic work with this at CU, and I really want to fold these thing into the course in the future, but for this round these are going to have to be things that I bring up repeatedly in class, but don’t explicitly build into the course. Baby steps Ives. Baby steps.