Thursday, July 25, 2019

The What, How and Why of Multiple-Choice Assessment Questions

https://stocksnap.io/photo/YYH82Y0CS7
I am currently participating in a 3-Week Course by Patti Shank on how to "Write Learning Assessments" focusing on writing multiple choice questions (MCQs) that are valid and reliable. The course has a great group of people who have lots of insights to share! If you don't already know or follow Patti, you need to change that right now! She is a workplace-learning expert, instructional designer, researcher, and author who shares evidence-based practices that promote deeper learning. 

During Week 1 of the course, one of the things that came up in the group discussion was how to distinguish between lower-level assessment questions and higher-level assessment questions as they map to lower-level and higher-level cognitive skills.

Now, I have written my fair share of MCQs! In my current work, I lead and facilitate competence development projects, which includes designing competency-based assessment exams for trades/ apprenticeship programs. I work across many sectors and have developed assessment/certification exams for Power line Technicians, Residential Construction Workers, Shipbuilding Workers, Saw Filers, Hairstylists, Servers, etc. Currently, I am working on a Recognition of Prior Learning (RPL) assessment pathway for Early Childhood Educators. Across all of these projects, I may have written well over 2000 provincial certification exam questions (summative assessments) and reviewed more than 10,000 questions written by technical writers.

A quick way that I use to distinguish between a lower-level and a higher-level assessment question is to look at whether it is a what, how or why question.

  • A "what" question will typically ask to recall, identify, define, describe, etc. which are all lower-level learning objectives. It is important to note that "what" questions may be phrased as what, which of the following, when, how much, how frequently, who, where, etc. 
  • A "how" question will tend to focus on step-by-step procedures or steps or phases in a process. For example, how will you do something, what will you do next, how will you apply a principle or a guideline, how will you calculate something, etc.
  • A "why" question will focus on analyzing, problem-solving or troubleshooting-type objectives. For example, justify, compare and contrast, evaluate, categorize or rationalize, etc.
Needless to say, there is more to mapping cognitive learning objective levels to appropriate learning assessment questions. But generally speaking, as we move from "what" to "how" to "why" questions, we are moving from assessing lower-level cognitive skills to higher-level cognitive skills.  

Wednesday, May 15, 2019

Planning for Transfer

Stocksnap
When I reflect about the goal of all learning and education, it is primarily to cause individual, organizational or community-level change. However, it is this aspect of planning for change using specific transfer plans where we often tend to fail in our role as program planners. I guess, the greatest disservice to planning programs is to plan and execute programs that stay on budget, meet all stakeholder requirements, are designed and delivered smoothly BUT fail to transfer any learning to the job. In that sense, I view planning for transfer as an integral responsibility of my role as a program planner.

As I reflect about my own work experience, I can differentiate the programs I have planned for folks in an IT organization trying to sell new software versus foremen at the waterfront working with containers and gantry cranes all day. There is so much that's different about the culture of selling in a global organization versus the culture of safety at the waterfront especially in a unionized environment. While my IT adult learners did short bursts of 'training' on their mobile phones, the foremen were coached and mentored one-on-one, in an intense program, on the dock. The transfer context of a sales pitch versus a ship to be loaded or unloaded on time is so strikingly different that program planning including evaluation and transfer approaches for the two cannot be the same.

For me, the key insight is to be aware of the learners' context because without it, there can be no learning or transfer. As a program planner, I tend to immerse myself in 'a day in the life' of my audience to make these key decisions. I like how Connie (@elearningcoach) describes the need for participating in such a discovery before the analysis. I love her concept of 'customer safaris' as a discovery tool and in my work at the waterfront, such a safari also includes climbing container vessels! I engage in needs analysis and participate in job work shadowing/observation to understand more about the audience. I continue to educate myself and my clients including key program sponsors and other stakeholders to systematically think about evaluation and transfer as key components of planning. All along the way, I ask myself two key questions:
  • How will the participants apply what they are learning when they get back to their workplace?
  • How will we know if we met the desired goals of individual learning leading to enhanced organizational performance?

Saturday, April 6, 2019

Designing Blended Learning Experiences


If I'd ask you to select your preferred method of learning among all possible options, what is the first thing that comes to your mind? I bet you, classroom training isn't it! 

When it comes to designing learning experiences for others, as L&D we tend to ignore or not pay attention to how we like to learn ourselves. 

If given a choice, we don't always enjoy learning in only one way. We usually like to learn by exploring (reading, google search, YouTube videos), talking to others (asking questions), working with others (learning by observation), hands-on (by applying), by practicing (and making mistakes), etc. We certainly don't like to learn (and we don't learn) simply by sitting in a classroom with one-way information flow (or dump) from an instructor. But when it comes to designing learning experiences for others, I wonder why we don't design more blended experiences that are more natural

Blended learning is:
  • Not a new thing. 
  • Not a radical concept. 
  • Not a new-age way of thinking about learning. 

As Elliott Masie puts it, “We are, as a species, blended learners.” 

Designing more blended experiences speaks to the diversity in the room because not everyone likes to or can learn in the same way. And I am not talking about the cliche of learning styles aka visual, auditory, reading/writing, kinesthetic styles. That is a myth. What is true is that: "Cognitive science has identified a number of methods to enhance knowledge acquisition, and these techniques have fairly universal benefit. Students are more successful when they experience the material in multiple modalities." https://www.scientificamerican.com/article/the-problem-with-learning-styles/

Blended learning experiences put the participants in control making them feel more engaged and responsible for their own learning. Designing blended learning experiences also means being respectful of adult learners and adult learning. It fosters the acknowledgment of the knowledge and experience that participants bring to the table by giving them the flexibility to choose how and what they'd like to learn. Based on my experience, the more autonomous and flexible the program, the more successful the learning experience and its application. Autonomy is also a critical success factor in promoting self-directed learning and helping people become continuous, lifelong learners. And that's a key skill that we desperately need for the future of work.