We’re all enjoying the Rio Olympics right now, and witnessing incredible athletic feats. Some sports achievements are very easy to measure. Take track and field, for example. In a race, the runner who crosses the finish line first wins. In soccer or water polo, the team that scores the most points wins the game.
In other Olympic sports, however, the winner may not be so nearly clear-cut. In gymnastics, an athlete prepares a program to the best of her (or his) ability. The judges score her on many elements, and the highest score among the athletes determines the winner. The gymnast will not know immediately following her program whether or not she won, as her competitors follow her.
Similarly, instructional designers don’t face an immediate, clear-cut path to knowing if their training “stuck” — and if the learners are able to apply what they’ve learned to their jobs. However, with evaluations, instructional designers can learn a great deal about their training programs’ success.
The Kirkpatrick evaluation levels assess learner reaction (level 1), learning (level 2), application of skills to the job (level 3), and business results (level 4). Challenges may arise with executing each evaluation level. Let’s look at a few of the challenges in levels 1-3 and see how instructional designers might mitigate them.
Measuring learner reaction, level 1, can be done by a course evaluation form, show of learners’ hands, verbal check-ins, or via a “dashboard” that measures the pace of the course and learner energy levels. The trainer notes learner reactions during the course and can use these to amend the course both “on the fly” and after. Clients may, however, read too much into level 1 results and want to change the learning solution without further assessment. It is important to explain to clients exactly what level 1 assessments measure — solely learner reaction — and to use other evaluation tools as well.
Level 2 evaluations, or measurements of learning, frequently take the form of final written tests or simulations. However, writing assessment questions can often be a challenge in itself. If the questions are too difficult for learners to understand, this could impact the validity of the test. Similarly, it can be hard to write appropriate “distractor” questions that are not obviously incorrect. After all, some people pass a learning assessment with flying colors — without ever taking the class or having done the job for which they are training.
The other issue for instructional designers is that there is not necessarily a direct correlation between level 2 “test” success and on-the-job achievement. Extensive knowledge may not lead to professional efficacy. To test for skills, ask assessment questions that explore the best way to execute certain job tasks or make on-the-job decisions, rather than questions seeking rote knowledge.
Level 3 evaluations measure application of skills to the job, often via learner or manager surveys, interviews with learners or managers, or on-the-job observation. An important consideration when measuring at level 3 is to ensure appropriate resources and tools — coaching, performance support, etc. — are in place to reinforce learning and support behavioral change.
The “success case method” created by Robert Brinkerhoff also may reveal successful on-the-job behavioral changes. With this method, stakeholders interview a sample of learners — typically high performers — to determine which factors have made them successful — and then corroborate this success with independent evidence. Because the strong learners share which influences contributed to their success, their interview responses can yield clues to help the less successful learners.
The success/case method can be very effective, giving stakeholders qualitative feedback beyond data, says Paula Spizziri, an EnVision consultant. “Sometimes you don’t know what’s behind the curtain…this gives you that,” she said. However, an organization using the success/case method needs to allocate time and resources for the interviews and independent data collection.
While all evaluation methods have some challenges, many of these hurdles can be mitigated. Because learning success, like that of a gymnastics program, may not be immediate, the evaluation process strengthens your training program.
I happen to be a big music lover, and enjoy both listening and performing as a flutist in the Sharon Concert Band. At last week’s band rehearsal our conductor, Steve, was doing his usual great job of keeping us on tempo, signaling when each section needs to come in, and gesturing to show we should play more piano (Steve often needs to remind us to play more softly). As I was playing, it occurred to me that a symphonic performance resembles the process of training.
In both, people assume distinct roles to accomplish the goal — whether it be performing the piece or executing the training program. And both require a specific process, anchored by practice, to reach those goals.
A symphony (or other musical piece) begins in the mind of a composer, who creates the music to be played. After learning the piece of music, the conductor rehearses the musicians in preparation of performing the symphony.
An instructional designer, similar to a composer, authors the learning solution. A trainer (“conductor”) then takes over, using the leader’s guide (music “score”) and teaches the learners (“musicians”) the material they need to know. A trainer may use visual aids or workbooks to guide the learner, just like a conductor uses a baton to lead musicians and keep the beat.
A symphony may feature a few soloists throughout the piece. A soloist shines in an orchestral performance, much like an active participant stands out in the classroom. A section leader in each instrument group organizes the section, much like a facilitator may help “conduct” a breakout group in classroom instruction.
While musicians and learners “rehearse” all together, each “musician” must put in individual practice time to learn the material. During a learning session, the trainer may break the learners into groups, where each group can work on a directed activity and then share the results with the entire classroom. Likewise, a conductor rehearses one group of musicians at a time, such as the flutes, to work on a specific part. Afterward, the entire orchestra can play together.
A successful symphonic performance depends both on the musicians rehearsing together (as in classroom instruction) and each player practicing (or learning) on his or her own. Facilitation by an experienced conductor, or trainer, brings out the best in the musicians or learners. Each person has his or her “part” to play, and each role must be “performed” to the best of the performer’s abilities to achieve a winning final product.
Have you ever gone on a treasure hunt, or maybe seen one in the movies? The seeker searches for a treasure chest—usually attending to challenges along the way. Imagine large rolling rocks and a few poison darts, all difficult to control! Eventually the seeker locates the elusive treasure chest, but not all its contents are gleaming. There is some culling to do, and some polishing, before being ready to present the gems to the funders of the treasure hunt.
Back in the real world, learning & development professionals are given a mission to add value to the business, often through a request for “training.” On the L&D professionals’ “treasure search” they often attend to challenges along the way. Imagine the multiple competing priorities, regulatory controls, and insufficient resources that can impact successful learning.
Once the treasure is “found” (the learners have learned something!), what culling and polishing needs to be done so you can clearly see your treasure, before you present your findings to the funders and other stakeholders in your organization? Sometimes it can be challenging to find those precious gems and polish them so we can see how they glitter!
Earlier this month, I presented a workshop at the regional ATD conference. The topic, Polishing Your Gems, focused on creating an evaluation plan for a learning solution. Joined by client Matt Matosic from the Boston Public Health Commission (BPHC), we worked through a case study summarizing BPHC’s course on Hospital-Based Patient Decontamination and discussed options for evaluating its success. Participants then had an opportunity to begin developing their own evaluation plans, using an Evaluation and Measurement Planning Checklist we provided.
But where to start? Here are the core questions we suggested in our workshop to help folks determine which treasures to seek, as they begin their own evaluation planning.
Business issues questions
- Why is this training (or other intervention) being requested?
- What is driving the request or identification of this issue?
- How will solving this problem support the organization’s “business” goals?
- What do you need to be able to show as an outcome of your efforts?
- Who will see the results? (Consider: L&D management, business line management, senior management, learners themselves, instructional designer, instructor, funders, and other stakeholders)
- What will each stakeholder do with the information?
You might recognize that many of these questions get at Kirkpatrick’s level 4 (business impact).
Job performance questions
- What changes in job performance are needed to support meeting the business goals?
- What changes in job performance can reasonably be addressed via a learning solution?
- What workplace supports (such as managers, mentors, or peers; job aids or other performance support tools; periodic updates or web meetings) are realistically available to support the change in job performance?
These questions relate to Kirkpatrick’s level 3 (behavior change on the job).
Answers to these questions will help you define the learning objectives, set up workplace support systems to ensure learning continues and is reinforced on the job, and develop evaluation tools at all the Kirkpatrick levels.
For more information on our Evaluation and Measurement Planning Checklist, contact email@example.com.
Practicing a classroom course polishes and improves it. Typically, practice comes with a pilot, during which instructional designers and trainers iron out classroom kinks.
Two pilots (an abbreviated pre-pilot and a full pilot) were employed in a class entitled Operationalizing Emergency Plans: Incident Command in Action, on which EnVision consulted for a public agency. The example shows how a pilot helps the design process when it works well.
This four-hour, immersive simulation was designed for nurses, administrators, and emergency medical technicians in healthcare environments such as long-term care facilities, hospitals, community health centers, and public health commissions. During the simulation, the learners make decisions to help their organization transition from day-to-day operations to a true-to-life emergency situation; a fictitious, extreme heat wave hits the area with no relief at night, producing health and medical complications for the learners to handle.
The learners are divided into teams of various medical and healthcare agencies seated at separate tables, with each learner playing a specific Incident Command System role. The scenario begins three days before the heat wave hits and progresses from the early stages to mid- and late-stages when serious repercussions occur, including power loss, a medical surge, and mass fatalities. The simulation ends with action reporting and improvement planning.
In the class, each team determines which actions to take. EnVision’s client utilized facilitators from different community agencies to help shape discussion at each table. In addition, each table had “injects” sporadically arrive with information that could impact each table’s decisions.
“During the class, the learners explore what it takes for organizations to be ‘operationally ready’ for emergencies, and to experience the Incident Command System in action,” explained Marilyn Kobus, an EnVision team member who worked on the project. Kobus supported the client subject matter expert/course designer-developer by offering design consultation, providing project management services, and contributing to the course instructor guide.
Following the pilot, the learners shared positive feedback. First and foremost, they felt more prepared for a large-scale emergency. The immersive nature of the class enabled the learners to envision what professionals in the different job roles might actually be thinking during an emergency. One learner called the class “a complete revelation.”
In addition, the learners liked the choice of a heat wave as a learning scenario versus a more typical New England event, like a blizzard. Their main challenge was playing “catch-up” following the fast-paced first module.
The instructional designers and trainers learned also, especially in the pre-pilot. Delivered to students in the Public Health Program at Tufts University School of Medicine, the pre-pilot uncovered necessary changes in the course timing, content, and organization of materials. The team also decided to add pre-work for the pilot session.
From the design team’s perspective, the pilot largely hit the bull’s eye. The team planned for minor changes for the actual course, such as having a participant assume the role of team scribe and modifying the pre-work.
Kobus was pleased with the pilot’s success, and believed the design team’s focus on gathering input from community stakeholders to create the scenario played a big role in it. “I have not seen a pilot with so many moving parts run as smoothly as this one did. Despite the fact that there were multiple players, detailed scenarios for the simulation, and unique “injects”…despite that complexity, I thought it was an outstanding pilot. Learners were fully engaged from the start with high energy, and said they gained great skills and information to take back to the job.”
Too often, we go through our workdays head down, nose to the grindstone, and office door shut (or cubicle door metaphorically so). A coworker may need help with something, but we get lost in a deadline, meeting prep, client appointment and are oblivious. Perhaps once in a while, we stop to touch base with our colleagues. Yet in this electronic age, so much of our networking gets done online, rather than face-to-face. And our coworker still has her question that never gets answered.
At EnVision, we value teamwork, communication, and the figurative open office doors very highly. The “EnVisioning our Future” (EOF) sessions we offer are natural extensions of this philosophy.
Twice each year, we hold a day-long EOF for the EnVision team. The subject matter runs the gamut from a new business book’s theory to examples of elearning . Our most recent EOF focused on software application tips and shortcuts. The day always provides a great opportunity to learn from and help each other.
At our “Sharing Application Tips” EOF, several EnVision members presented all kinds of software tips — from embedding objects in a PowerPoint slide to assessing our document’s reading level — all designed to help each of us be more productive and efficient. As each team member presented her software tips, those tips were enthusiastically appreciated by the team.
”Sometimes, the simplest things make a huge difference in the way we work,” said Ginny Maglio, an EnVision team member. “The EOF professional development day validates that we never stop learning and can always learn from each other.”
At this EOF, as with past sessions, our team members realized three benefits beyond the subject matter knowledge they acquired. Team sessions, like our EOF days, can empower your team members to:
1. Help each other while also gaining knowledge in another area. In our recent EOF, one team member demonstrated how to create an info graphic using PowerPoint, while another showed us how to create a table of contents in Word.
2. Learn about—and from—other team members’ projects. Because we each work independently or in small teams, we may not know about other team members’ projects and competencies. The EOF sessions introduce team members to each other’s skills and capabilities, and also show participants how different parts of the organization work to achieve clients’ objectives.
3. Grow relationships with each other. Networking, sharing ideas, and just plain “kicking back” with your coworkers facilitates bonding, making you more comfortable in your role and more likely to approach a coworker with a question. And who knows? You may make a new friend out of it, too.
So, in the rush to meet deadlines, don’t forget to hold regular learning sessions. This results in not only a better-informed team, but also a collaborative, harmonious one.
Change is hard and causes unease, anxiety, and at times, frustration. Every parent who has dropped a child off at kindergarten (or college) knows this!
While change can be overwhelming or even scary, it is often necessary. Professional change brings its own set of challenges. How can you best navigate change at work?
EnVision needed to adapt when we were required to move our SharePoint site over to a new hosting company. We were notified by the “old” hosting company via e-mail and given less than a month to move over our files, as our original service would then be discontinued.
I was a bit taken aback after learning of the tight timetable. After taking a few deep breaths, I notified Tom, our IT guy, and Danielle, our technical/administrative whiz. I couldn’t do this without them.
Tom managed the functionality of the server – back-up and hosting. Danielle’s responsibilities, which required about 8-10 hours of work, included revising the new site’s look to mimic the old SharePoint; setting up appropriate user access for each team member; creating usernames and passwords; and providing crackerjack technical support.
Danielle met some challenges initially. “I went in thinking it’s going to be exactly like the old SharePoint….It is a little bit different,” explained Danielle. “It wasn’t as intuitive.”
The new service provider didn’t offer a support document or user guide. Plus, Danielle discovered that when she added folders, there was no way to return her to the main SharePoint page, so she got creative and built a workaround by adding a “home” link.
The end users (EnVision team members) encountered obstacles, too. Some of the functionality worked best with a particular browser; one team member had difficulty logging in; another had trouble opening and saving documents. Thanks to Danielle and her support, these issues resolved pretty quickly.
Lessons learned from this SharePoint experience can be applied to any change. While initially I felt a bit overwhelmed, I dug into the new software and eventually figured it out.
Three things that helped me were:
1. Planning extra time into my schedule to adjust to the new software and minimize impact to our own clients.
2. Asking for help when needed. I was fortunate to have Danielle and Tom to turn to, and relied on them. Having Danielle as my capable “point person” to train my team was a huge benefit. I will remember to choose my “point person” carefully for all future changes; it can make the difference between implementation success and failure.
3. Keeping a positive attitude. Knowing that my team and I would eventually figure out the new SharePoint helped me through the change.
I’m happy to say we successfully moved our files and team over to the new SharePoint. Weeks later, it seems like we’ve always used this service provider…and we like the new site even better than the old. Time to embrace the next change!
When you eat at a restaurant, the food arrives in a specific order. First, there’s the bread (for those who still eat carbs!). Next, you may get a first course, a salad or soup.
Then the waiter brings the main course, the heart of the meal. This takes longer to eat than the bread or first course, but is usually the most satisfying part of the meal. It’s also the most vital part – if you don’t eat the main course, you could be hungry when you leave the restaurant.
Finally, you can order dessert and coffee. Of course, dessert isn’t necessary – it’s “nice to have.”
But what if all of your courses were served at the same time? You wouldn’t know which food to start eating, or where to focus your attention. You’d have dessert at the start of the meal, whether or not you wanted it. And, you’d probably feel overwhelmed and irritated, and might lose interest in the main course.
The same principle holds true for elearning. If you are given too much material simultaneously, your mind will be overloaded, just like your stomach would be with too much food at once.
Here are some pointers to reducing cognitive overload in your elearning:
- Create a course that reads logically. People read top to bottom, left to right. So avoid having information begin on the right-hand side. This prevents your learners from having to work too hard and get frustrated.
- Use white space liberally. Placing paragraphs and graphics too close together creates a disorganized appearance and will only confuse the learner. Instead, break up one screen into two or more screens.
- Add color wisely. In an elearningindustry.com post, Christopher Pappas advises using just two or three colors per course or module; otherwise, the course may appear disorganized. When adding color, don’t forget that some learners may be color blind, and find another way to provide visual contrast. You can even find online applications that analyze color contrast.
- Be smart about animation. There are a lot of great tech tricks out there, but just because you can do something doesn’t mean that you should. Any formatting tool should support the learning objectives and/or learning styles; when in doubt, take it out!
- Focus on the need-to-know; trim the nice-to-know material. Research backs this up: Studies have shown that extra details can actually detract from learning. If you must incorporate the optional information, put it in a clickable “tips” box on the applicable screen or in a course resource list.
- Use audio rather than text to explain graphics. This allows the learner to focus visually on the graphic while taking in the description aurally, which maximizes learning. Of course, text should always be an option for sight impaired learners.
Implement these tips, and they will help your learners to be satisfied by – and not overwhelmed by – your elearning solution. And, they might even look forward to returning for another “meal” sometime!
In part 1 of Re-envisioning the Meeting, we looked at how to prepare for and open a meeting. Now let’s look at facilitation, meeting conclusion, and post-meeting activities.
The facilitator must consciously strive to keep participants on task to achieve the meeting’s goals. If you’re talking about an upcoming training launch, for example, don’t spend time generating ideas for your next team development day. Participants may have tangents that they would like to explore, due to their job role or specific interest. It is the facilitator’s job to keep them focused. Using a “parking lot” can capture important topics for discussion at a later time.
While attendees may take their own notes, the meeting facilitator should as well (or assign someone to take notes), so everyone can refer back to them later. As action items are assigned during the meeting, write down each assignment, responsible party, and due date. Similarly, for key decisions that are made, document the decision and date. For recurring team meetings, you can build on these lists and check off assignments as they are completed.
Summarize key points and assignments at the end of the meeting. For recurring meetings, remind attendees when the next meeting will be. And of course, thank everyone before you adjourn. Showing your appreciation for everyone’s time and participation helps motivate them to continue contributing in future meetings.
Follow Up Promptly
After the meeting, be sure to e-mail the attendees and other interested parties the meeting notes, or at least any major decisions that came from the meeting. Include the action items that need to be completed and their due dates.
Although not done often enough, it is definitely wise to ask for participant feedback to help the next meeting run even smoother. One idea: Pose a statement or two, such as “This meeting met my needs” or “I know what is expected of me prior to our next team meeting.” Ask attendees to write their level of agreement with the statement, on a scale of 1-5 where 1 is strongly disagree and 5 is highly agree, and hand in their responses. If you see a trend, you can follow up with the group to learn how to improve the meeting next time. Another idea: Ask an open-ended question, such as: “What worked really well in today’s meeting you’d like to continue?” or “What is one thing we should do differently in future meetings?”
Follow these tips, and your meeting will not only be more meaningful, but can result in greater productivity, too!
“Wow…what a GREAT, productive meeting!”
When was the last time you heard someone say that? Hopefully the other day …. or has it been a while?
Meetings have, sometimes fairly, earned the reputation of being timewasters at best, scheduled
snooze-fests at worst. Too often, employees suffer through meetings until they are released from captivity, free to resume “real work” again.
Yet, meetings are actually “the laboratories of real, measureable teamwork,” according to an article by Kristine Kern of The Table Group (Inc.com). They can in fact be very productive, if planned and run well.
At EnVision, we offer a proprietary and customizable course called “Making Meetings Meaningful.” The cornerstone of the course is that a meeting’s success is not only determined by what occurs during the meeting, but also what happens both before and after. To conduct an effective meeting that really furthers participants’ work, you should:
Before you gather people together, know the objective for your meeting. What do you want to achieve? Be sure to only invite those people who will be interested in and/or impacted by the meeting’s content. If certain people don’t need to actually participate in the meeting, feel free to give them a pass and instead update them afterward.
Plan an agenda to keep the meeting focused and avoid tangents. And, estimate the time it will take to discuss each topic and share your timetable, so attendees know that the meeting, no matter how beneficial, won’t go on forever.
From the beginning, set the tone that the meeting will be efficiently run. First, begin on time. If someone arrives late, acknowledge him and continue with the planned agenda. Backtracking will only frustrate those who came on time.
Share the meeting’s objective and agenda with the group so they know what to focus on. And, if needed, set ground rules for the meeting. If your company is one that claims meetings as “smartphone free” zones, remind everyone to turn the sound off their phones and put them away.
Designate a “time keeper” as a back-up for ensuring that the meeting ends on time.
An organized start will set your meeting off on the right foot.
What do stained glass windows, intricate marble carvings, and richly symbolic murals have in common?
They are all easily seen from just one spot in the Library of Congress.
Why do I mention this?
I recently visited Washington, DC and toured the Library of Congress. If you haven’t been to the Library, I highly recommend it. It is jam-packed with, well, books. But also with all manner of media. And the art and architecture! The building was built to safely house books, but it also shows off art and architecture created by over 40 American artists.
Our tour guide spent much of our hour together describing just a portion of the art and architecture we were seeing in a relatively small part of this edifice. There were masterpieces filling every square inch of wall, ceiling, floor, pillar, and even staircase. Each item’s artist, creative origin, and beauty could have merited an hour on its own. After five minutes I was on mental overload!
I can tell you the feelings I am left with after visiting the Library: awe, pride, and patriotism. But I don’t think I can tell you who the artists were or the significance of each piece of art. Maybe that was the idea — that the most important take-away was my overall impression, not the myriad of artistic details.
Well, I do remember the Library is supposedly going to acquire all Tweets. And that Jefferson’s library burnt down and they are trying to re-acquire all missing volumes so they can recreate it. But the experience reminded me that less, so much of the time, is more. So, I’m making this post really short. Just trying to make one point.
Here it is: Keep your training focused. Lots of content doesn’t necessarily translate to effective learning.
I might need a reminder myself the time next I write a post, but here’s to giving it a try!