I'm writing this post for Canek as a first draft for a Design, and Learning project we're doing together. The idea is that I'll take (via this blog post) a first pass at creating a 2-page summary of the paper on reflection that we've chosen to cover, and that he'll pick up on the content, add his own thoughts, and convert it into the final format. I am running flat-out on my RAT and my exhaustion is painfully, painfully obvious in this post -- I'm so glad I've got him to work with on this, because I can focus on just thoughts/summary/reading/content without having to simultaneously worry about polish. Thank you, Canek.

Anyway.

We're looking at one (of many) patterns that expert designers employ: reflecting on process. (Other groups in the class are covering other patterns such as idea generation, troubleshooting, iteration, and a host more from a giant 114-page paper that Robin co-wrote that's winding its way towards print right now.) Since Canek and I are both engineering education PhD students, we want to unpack what "reflecting on process" looks like for undergraduate engineering students. How can we understand the way this pattern instantiates itself in that particular context?

Fortunately, Jennifer Turns, Brook Sattler, and Deborah Kilgore have done it for us already -- or at least they've written a paper that gives us a piece of the puzzle, and we're going to summarize it. So... ENTER THE CITATION!

Turns, J., Sattler, B., & Kilgore, D. (2010). Disciplinary knowledge, identity, and navigation: The contributions of portfolio construction. Proceedings of the International Society of the Learning Sciences (pp. 818-825). Chicago, June 30.

These folks took 69 undergrads from a variety of engineering disciplines and put them in portfolio studios. The 69 undergrads were a gender-balanced subject pool, with over 50% self-identifying as belonging to an underrepresented group -- which makes me wonder how these folks were recruited, because... wow! If the experience was voluntary, was there something about portfolio creation that tends to be appealing to underrepresented groups -- perhaps the knowledge that, as minorities, they'll be "under closer scrutiny" and thus need to pay a lot of attention to developing good self-presentation skills?

In any case, these engineering undergrads went through short porfolio workshops run in a studio style, meaning that they spent class time both creating their own portfolios and peer-reviewing each others'. For consistency and to ensure maximum open-endedness, all the workshops were facilitated by the same individual, whose role was highly scripted, offering "little or no information about the criteria for choosing portfolio content or possible future uses for the portfolio itself," deferring "what should I do?" questions to group discussion. The workshop framed portfolios as the ""construction of an argument about the ways in which one is prepared to contribute to future engineering practice, specifically the realm of engineering practice that is of personal interest." While the end portfolios may have been seen by their creators as positivistic demonstrations of their knowledge and abilities, the more important part was the constructivist process of self-discovery and self-authorship the creators went through while making them.

At the last workshop session, each student filled out a 35-45 minute online questionnaire about their experience constructing the portfolios. Turns and Sattler independently coded the written responses, ending up with high cross-rater reliability (an average of 0.83 for Cohen's kappa, which means that even before they talked about their answers, they had mostly arrived at the same ones). So what does this look like? What do engineering students write about when they reflect on building artifacts they'll use to professionally present themselves?

Disciplinary knowledge. Students wrote about a heightened sense of accountability for their acquisition of engineering knowledge, a better sense of knowing what it was they did know, and a realization that they could look at many different sorts of experiences (jobs, etc. as well as classes) as giving them a wide variety of different sorts of knowledge.

Identity. Students wrote about their "changed personal sense of identification with engineering" and how important it was to be able to explain their preparedness for practice. Making portfolios helped them realize that they could -- and should -- shape their future learning actions so that they could "grow up" to become the future-engineering-selves they wanted to be, and so that they could later prove the nature (and awesomeness) of those future-engineering-selves to future employers/schools.

Experiences (the authors called this "navigation"). Students wrote about how being forced to go back and revisit things they'd done in the past made them appreciate and understand them more broadly; they saw the big picture, and the vision of this map made them think more about how they wanted to move forward with increased intentionality, perhaps filling in some gaps in their map they were able to perceive.

Meta-process, or the process of creating the portfolio itself (the authors called this "portfolio"). Students wrote about the fact that they learned the portfolio-making process in the first place, and that they appreciated having this artifact of their identity that they could now show to external people.

So what?

Well, portfolios seem to be a good thing for reflection -- and a good thing because they force students to reflect. They seem to encourage students to become self-directed learners by recognizing gaps and past triumphs when their cognitive map of "what I have done" clicks into place as a whole, rather than the class-sized pieces they'd previously been divided into. For this reason, the authors framed portfolios and their portfolio workshops as educational interventions, almost like a medicine whose effect would be all of these positive things.

The paper also noted that the causality link was not definite nor unique -- students could (and perhaps did) get all these positive effects through other experiences, and it's possible the portfolio workshop was a redundant "medicine" because the students would have had a later learning experience that would have given them the same insights eventually. I think it would be interesting to investigate this question further -- did the portfolio workshops accelerate these effects, cause them to happen with greater intensity to some students that would otherwise not have developed their reflection skills as strongly, etc? That's only possible with a control group, but this study did not have a control group.

Canek brought up a thought in class today that I will (mostly out of exhaustion at the moment) leave to him to elaborate upon. He came up with this great phase to describe portfolios -- they are an "artifact of your identity," and so you end up with some interesting extension possibilities. A traditional portfolio, the kind examined in this study, is an artifact of your present identity, built out of your actual past. But you can also make a portfolio for a future identity you hope to have -- the things you want to say about yourself 5 or 10 years from now -- and then figure out how to grow into it. You can make a portfolio for an alternate past self: what if you had made different decisions, gone a different route? Portfolios can support many different types of reflection.

(You can tell I'm getting exhausted; my thoughts are winding down, slogging as I write. I'm just trying to make it to the finish of this post and I am going to take a nap when I get out of class.)

Final note: in this blog post, I've totally ignored that the 69 students were split across two types of portfolio workshops -- one type focused on a portfolio for a single class, the other on a portfolio across multiple classes. Canek might pick up on this more, but I didn't feel it made a huge enough difference to cram into our 2-page summary.