As learning profressionals, we spend almost ALL of our time and resources cranking out new ways to motivate our audiences and design to make the content enriching, challenging, thoughtful and most importantly to enable the instructional goals intended by the production.
But what about the audience? Don’t THEY have a responsibility to themselves as active participants, even in asynchronous learning?
I point this out because a friend of mine (IRL - “In Real Life”) directed me to a website another buddy of his put together to share his lessons learned from smoking meat — barbecuing for long periods of time using smoke (hence “smoking”).
Here’s the link, but what’s striking to me is what happens after the jump:
What’s that? Terms seem a bit harsh? Well, here’s why I insist that you follow the 5-Step program as written before going off on your own.
The 5-Step program is not so much about cooking any particular meat as it is about learning fire and smoke control. Once you understand that, you have the skills to cook any kind of meat. But it’s been my experience, from both cooking and being a member of BBQ oriented mailing lists (listservs) for the past 8 years, that when newbies try to mix and match advice from different sources, disaster is just around the corner.
I am by no means saying that my way with the WSM is the only way. But I’ve found, without a doubt, that if one follows all 5 steps straight through they gain a damn good understanding of how to use the WSM—and will have learned how to prepare four different meats and enjoyed five damn good meals in the process, which is hardly a high price to pay for keeping your ego in check and following orders at the very beginning. Once you have the basics down, it’s fairly easy to cook BBQ better than 99.9% of all BBQ restaurants, and you are invited, heck encouraged, to do whatever you want after that point. But for your first five cooks, follow the program exactly—or don’t start it at all, and spend years screwing around and trying to figure out what went wrong, like I did.
I think the statement speaks for itself, but I’ll share my analysis anyway: You have a subject matter expert here, sharing lessons learned through lots of trial and error. There’s context, there’s relevance, there’s humor and personality — it’s good learning.
On Saturday night, I saw Crowded House perform. This event provided an epiphany that I will attempt to relate in my normal, long-winded and winding manner.
As a former music major and the writer of many many cheesy love songs back when I lazily pursued musical ambitions, I appreciated many torch singers. I knew a handful of Crowded House songs and I like those songs. I’d tape them off the radio or off of other friends who had them on CD when I’d make mixtapes. I just never liked them so much that I had to own their CDs until I worked at a used-CD store. Then I had so many CD’s that, like my penchant for buying books, the CDs just sat in my shelf until the advent of iTunes allowed me to rip them, store the albums electronically and sell of the CDs. As a very early owner of the iPod, I’ve now gone through several iPods, each with far less capacity than my music library. In fact, it is to the point that even though I create new playlists on a weekly basis (the modern day mix tape), I can’t dig into my library fast enough or far enough, and now I just fill my iPod up completely at random just to discover what I have in my library.
Now that’s already a paradigm shift over how most people approach their music library, but it continues further down the rabbit hole. I’ve been taking this approach — just randomly filling up my iPod to listen through my library, metal or classical or outlaw country or throat chants from Nepal — for years now. And it’s through this shuffling that I’ve now heard pretty much every Crowded House song recorded and discovered that I’m a HUGE Crowded House fan. Despite never really being aware of my exposure to them when they were regulars on the airwaves, I look through my catalog of torch songs and they sound like imitations (at worst) or allusions (at best) to Neil Finn’s work, both solo and collectively in his bands Split Enz, Crowded House and the Finn Brothers. This epiphany combined with a deep and abiding love of music drove me to network through friends of friends to find someone willing to go with me (my wife is six weeks from delivering our second child, so standing for three hours at a show is just not high on her preferences — also, she’s much more into metal as far as shows go).
I found a friend of a friend who was game, even though he was just a passive fan of the one or two songs he knew. It mattered not, I just needed to share the experience with someone, as well as enjoy a couple pints of Guinness before the show. It was a fantastic show, and Crowded House introduced several new songs — my favorite new song, “Turn it Around” is below:
After my 2am burrito following the show, I was still pretty awake ruminating on the performance I just saw. I thought a lot about this song and how much I really liked it. I spent a little time on Sunday night (after catching up on Battlestar Galactica) seeing if the set list from Saturday’s show was posted by a fan so I could, at least, figure out the name of the song. By Sunday night, I had not seen a set list from my show… but there were a number of setlists already posted up in various fan forums. This is a practice that goes back to fans of the Grateful Dead, Phish, Black Crowes and Pearl Jam: dedicated fans who chronicle every show and bootleg available. From this I was able to get the name of my song, and then I proceeded to search again for “Crowded House Turn It Around” and TWO YouTube videos came up immediately from different fans at different shows.
It immediately hit me that I should’ve brought my new Flip camera instead of chickening out, but I’m still in a mindset that cameras are going to be taken away at shows, like an unfortunate experience I witnessed years ago at a Phil Collins show. Since then, unless it’s expressly stated that cameras are allowed, I leave them at home.
This thought immediately turned me on my head with regards to how we distribute official knowledge. Let me try and work backwards. See, even three years ago, there wasn’t a YouTube (at least, not a popular one). There weren’t compact video capturing devices that people could afford easily (like around $100). There wasn’t a culture that made it de-facto “permissable” to record concerts even with amateur equipment. If you were brave and cunning enough to sneak in your equipment, there wasn’t a way to easily share it with anyone beyond your immediate friends, unless you were also resourceful enough technically to run your own FTP server, and even if you were using BitTorrent or some kind of Gnutella-based file sharing protocol, people would have to be stumbling onto it — it’s not like you would be able to easily Google it. But now you can. Now, it’s expected that after I go to a show, there are perhaps multiple ways of getting a recording of that show, even if it’s not me who’s doing the recording and posting.
There’s a lesson to be learned with how we approach Subject Matter Expertise. It’s been said on a few forums that the biggest time delay in getting Rapid Learning out to learners is in the Subject Matter Expert reviews. In the current (and arguably antiquated) model, this assumes that SMEs can’t create the content themselves — that producing instructional material is a job for ISDs or content developers — some other “title” or person than the SME themselves.
Well, what if the learners capture the content live and share it, doing the work of producing and tagging material they find interesting? That’s an idea. The rub is that much of the content we’d expose them to probably doesn’t reap the same kind of fandom as a band does. Well, okay then: what if our Subject Matter Experts just put it out there? The tools are getting better (Articulate Studio 08 looks promising and easier than Articulate Studio 5). The tools are getting cheaper. The tools are getting easier to use (try editing some video with iMovie 08 and let it just export directly into YouTube). SMEs aren’t taking up the valuable time to get information out to learners — WE ARE. We need to think differently (again) about what our job is — our job is to help craft the message. We can help contextualize it. But getting the message OUT to learners? That doesn’t have to be the job of content developers or instructional designers any more. And it shouldn’t, because we’re wasting time.
What we should be doing is helping identify what the semantic questions are addressed by the SMEs when they capture their knowledge and publish. We should help make it easier for knowledge centers, be they groups of people or individuals themselves, to know that there’s new informational content out in their periphery and provide border resources to pull information together. Field Trainers and Training Managers can be creating “playlists” of the information that is put out by SMEs, customizing that information for learners who have finite and contextual instructional needs.
We’ve tried to automate this with intelligent agents (both in the instructional technologies and even tools like “The Filter” for iTunes) — and they kinda work. But a good instructor knows how to build curriculum. If we make it easier for people to answer their own questions or for facilitators to pull on knowledge resources to quickly create informational materials, I think you’ll have a state where knowledge transfer happens faster and the information is more appropriate. If you couple this with an ability for the learners to SHARE the information that’s useful to them with others, you start to build communities, however dynamic, of learners engaged in the discourse.
This is how users of many generations behave online now. We need to think about how to capitalize on the active engagement of the learner as a possible facilitator of flexible organizational learning.
Imagine if you had the means, the open-mindedness of the client and the management sponsorship to pull out all the stops and really produce a piece of learning that was fun, relevant and “just right” for the goals you were trying to meet with your learners? I’m at the end of such a project, on the eve of its launch, and I could not be more excited to predict a huge win for the first “serious” learning game in our organization.
I’ll be the first to admit, this project could’ve gone wrong from jump in so many ways. We had an internal client who, like many clients, was very risk-averse, so the thought of doing a “game” was a risky move and required a lot of handholding. They could have bailed at any time. We had an incredibly tight deadline for a project like this. Normally, a multimedia-heavy project like this one, you’d like a solid six months to develop it out. We gave our vendor three, and by a lucky break for all of us, the delivery was delayed by a month which was needed.
We used a brand new vendor who was brought to us initially by our CLO. I’ll be the first to admit based only on the sample products they provided that I would not have chosen them — mostly because they seemed more like an Agency than a game development house and the look and feel of the products I saw were similar in nature, and I felt their production would be a disconnect with our audience (and our internal client). When the project was emerging from the Instructional Designer on the project, I ballparked the project at a certain cost. I expected the vendor to come in high and then we’d have to haggle and negotiate. I expected that working with the vendor, like many vendors I’ve both worked for and worked with, would be a painful tug-of-war, followed by some finger pointing, followed by relief that the project just “got done.”
I could not have been more wrong about this vendor, and I’m very, very happy to say so. They came in so close to the number I ballparked, I began to think they were taking us seriously. I waited with baited breath to see their first draft of the storyboards indicating the look and feel for the game, and having taken the time to visit and talk with one of our branch stores close to their office, they produced storyboards that I felt so perfectly blended my expectations for how to be at once “cool” and at the same time “mindful” of the people we were looking to instruct, as well as the complexity of the subject matter we were looking to demystify. They made learning the material (and the subject itself) “fun” and still “tasteful.” I was very impressed, and I’m the kind of person (as you know on this blog) that doesn’t run out of opinions. I became hopeful that this really could work.
They created the project in a very complete Alpha state. We tested it in our network and found that the bandwidth required would be a major obstacle for the target audience. We talked with our vendor about reducing the audio and video quality a bit and retesting it in our network before doing any more work on integrating it with the LMS — because if it wouldn’t perform as “content” out of the LMS, there was no point on troubleshooting the LMS communication. They had new files to us in a matter of days. We retested and got a green light on performance.
Then we moved onto LMS integration. I put together the API Wrapper and the rest of the SCORM packaging for our vendor, because they had not built for an LMS deployment before, and it would be just silly to make them go through the learning curve when I could just do that heavy lifting with little effort. They were able to write and read from the LMS at the prototype level (we did a technical test before they even tried to get the real content working to debug the communication issues). Not looking at their ActionScript at all, when we noticed some issues with suspend_data not being sent to the LMS, even though the code was the same as the prototype. It turned out that the content was sending consecutive JavaScript calls, which goes back to the whole synchronous/asynchronous deal about ActionScript and JavaScript (we had to use Flash 7 Player because my organization had not upgraded to Flash 9 Player at the time). Moving the calls so they were separated and event-driven made a huge difference. I was on the phone for a day and a half with the vendor. In ten years, I never worked as, for or with a more willing partner.
For reasons I’m sure you can understand, I can’t show you the game. I probably can’t talk much about what the game is about or what we’re trying to teach. I probably can’t broadcast the vendor we’re working with (though if you ask me offline, depending on whom you work for, I’ll be happy to tell you).
The point of this posting is to get off my chest in as public a means as I can how happy I am to be able to help make the vision of one of our Instructional Designers a reality — even if all I am is the babelfish (Hitchhiker’s Guide reference). We have our foot in the door for serious gaming. I’m betting it’s going to be a smashing success and will usher in a shift in instructional approach, both as far as what we propose and what our internal clients will consider.
And… I’m just happy as hell that after years of producing cool and not-so-cool page turning stuff, I get to finally be part of something different. I worked with a fantastic Instructional Designer and a really incredible Project Manager (I actually am gushing over Project Management and Instructional Design) — both of whom really “get it.” I had a boss who was willing to take a chance and a CLO who was ready to be a sponsor on something different. Best yet: I had an internal client who, despite their concerns, was willing to trust us and get the job done right.
And one more thing: I’m very humbled that there are vendors out there that can really be a partner in making great learning experiences, on-time, on-budget and far-exceeding expectations.
For the first time in as long as I can remember, I have a happy project story and it didn’t kill us.
I had the outstanding pleasure to work and travel with Jono Poltrack for the bulk of my 2+ years working with ADL. Jono was the original project lead for the Sample Run-Time Environment, and for a while before my man Doug took over for him, Jono knew more about the actual workings of a Run-Time engine for SCORM content in 1.2 or 2004 than just about anyone. Seeking other windmills, Jono joined his roommates and best friends in creating an IT Consulting business a few years back, and lo and behold he’s coming around full circle, now via the Knowledge Management space — back into working with SCORM again.
Together with his regular and some new partners, they’ve put together a site called ADL eXchange. They are trying to shepherd some new voices in the ever-ongoing dialogue with ADL from the user community.
At the last SCORM TWG, Jono and Mike Hruska were there, bought me a beer and asked me to write for them. I can’t refuse good friends, so I took a page from my presentation last year at ILCE’s conference on SCORM, and put down a few thoughts about Instructional Design since it was on my mind now (as it was at the conference) that I cobble together a standards guide for my own organization relating to E-Learning content.
If you’ve never put together a standards document for your production team, you probably don’t realize how important instructional strategy is to the final product for E-Learning — if you care at all about learner engagement or consistency or the end-user… stuff like that. At least, you probably don’t realize it until it’s smacking you in the face.
Anyway, feel free to give ADL eXchange a look, if nothing else, for the pictures of Jono and Mike, both members of now-defunct 90’s band Badwrench (this is no lie).
We tend to adopt one or more of three common sets of conceptions
There are basic tensions and questions raised by these sets
It’s worth struggling with these questions
Conceptions of Learning:
Behaviorist
Selection of behavior by educator
Pursuit of individual satisfaction
Cognitivist
Mental representations mediate
Active learning toward genuine understanding, like a scientist
Sociocultural
Tools & systems beyond the lone thinker
Humans embedded in social applications
I’ll forego much of the discussion of Behaviorism and Cognitivism because they’re pretty well-known. Learning isn’t something that an individual does — it’s something that a whole system does. Reading uses a number of different tools. There are heuristics attached — teaching kids to read isn’t about putting something in the kid’s head that wasn’t there before — it’s about filling in the gap in the system that enables them to do something they weren’t able to do before.
People are essentially social, where others fill-in the gaps. People only function in environments where they have the tools to function — maps, communication tools, etc.
Does it matter if people “genuinely understand” as long as they get the behavior right? One answer belies a cognitive slant — that given the right information and the right environmental variables, it’s still up to the individual to be able to make the right decisions. The other answer is hard-nosed, “the result is all that matters” model of performance.
At a meta-level, we need to treat people as cognitivists… but as an organization, we need to be behaviorists. Over a long enough timespan, behaviorists believe that a dumb, rigid process selects and reinforces things naturally. In the case of intrinsic motivations, though, behaviorism can subvert the things that cognitivists nurture and take away the performance unless there is the motivation to follow it.
Can others choose the direction of learning, or should people explore? There’s no solid answer one way or the other on this. Which is why sociocultural approaches may make more sense in a corporate environment.
Do individuals learn, or does learning involve others, objects and tools? Instead of formal assessments, use authentic assessments where the assessment is built into the activity, like a simulation. We think of individuals as units that can be moved around an organization (or outside of it).
Stanton Wortham from The University of Pennsylvania, Wharton School of Business is talking about learning theories, their characteristics and contradictions.
Behaviorists believe that organisms learns by feedback from their environment. B.F. Skinner is among the most prominent behaviorists. You create a situation that rewards students for doing what you want them to do. Positive behaviors lead to positive responses — negative behaviors result in negative responses. Behaviorists want control and access to good rewards, has to have clear goals for what they want students to achieve and they need to be consistent.
Cognitivists believe that students need to have control of a situation in order to understand it. Students are trying to get students to build their own models of understanding of the world. They create an environment with lots of tools and manipulables to come up with their own answers and monitor/question their actions to support the students in building their own path to the right answers. When students fail, it presents an opportunity to revisit issues and choose another path.
Behaviorists and Cognivists believe that the individual is the thing that learns, and learning transfer happens when an individual learns something from one context and applies it in another context. Socio-culturalists believe that it’s not individuals that learn, but a system that learns. It’s not crucial that an individual understands everything about a subject, but that they contribute their part to the system.
Note… this is the first time I’ve heard about a theory of social learning, so I need to find out more about this because I’ve previously thought of myself as a Cognitivist, but this sounds like a way to highlight where learning is going — possibly coupled with Cognitivism.
I was originally going to go to a discussion on Performance Support with Frank Nguyen next, but I may follow up more with Dr. Wortham.
Doug Lynch (The University of Pennsylvania - The Wharton School of Business) was one of the keynote speakers at Sunday night’s kickoff to the Learning 2007 conference, and he was quite obviously the smartest man in the room — so naturally, I flocked to his session this morning.
“Thomas Jefferson once observed the increasing strength of evidence for Isaac Newton’s theory of gravitation that Newton ‘indulged in reason and experimentation, and error fled before them.’”
Newton developed theories, and some of them have held up. LET professionals have also held up theories, and some of them work, if you don’t sweat that some parts hold up better than others. But people talking about principles as “facts” aren’t facts — they’re just assertions.
There are no scholarly writings on ROI on Google Research. None. People have been able to demonstrate the economic returns on learning, but there is more than what can be demonstrated in these studies. Zagat ratings are coming out now for health care, so there’s evidence to support the “effectiveness” of the quality of health care, but it may not be all the evidence you want.
So here are some questions to help deconstruct ROI:
What level of sophistication do you need in terms of understanding evidence of impact?
What tools do you use to gather evidence of impact?
What tools do you use to analyze performance / learning?
Do you use any methods to evaluate implementation in addition to the intervention? If so, how?
What forms of evidence do you need to determine if what we do impacts business goals?
ROI has a very narrow meaning to CLOs, Finance people and Economists. As a learning profession, we throw around the term ROI, when we should really be talking about “Impact.” The person hearing you when you use the term “ROI” is thinking something completely different.
A problem with our world:
Man is naturally metaphysical and arrogant and is thus capable of believing that the ideal creations of his mind, which express his feelings, are identical with reality. From this, it follows that the experimental method is not really natural to him (Claud Bernard, 1865).
If you want to find out what is going on, you need to look at what is going on (Yogi Berra)
Learning is loosely coupled. What LET professionals do is a messy business. We have norms in our world
Kirkpatrick’s Four Levels
Reaction of student
Learning
Behavior
Results
So… what are some underlying assumptions about ROI?
Ratio of Net Benefits: ( Benefits - Costs ) / Costs
Let’s deconstruct benefits (or profit). What goes into calculating the benefits of training?
Sales?
Income?
Stock Price?
The idea behind ROI is that it is a number that is monetized. You define it, you measure the costs and the benefits and then you monetize it. Are these things easy to do? No, they’re not. Stock price is a net-present value of what the company is doing today and it’s speculation on how it will do in the future. Stock price also gets at some other things that are going on — like a company’s performance in relation to the rest of the market.
For you to say that you have impact, you need to have control of everything that’s going on. To use ROI as a measure of learning’s impact, you’d have to control the entire market and factor non-related issues out — everything from competitive advantages, the cost of raw materials, fuel costs… you get the idea. And even if you could factor those things out… you still have to factor out all the human-factors to filter down to just Learning’s impact on an organization.
The relationship of Learning departments and HR departments probably is a better indicator of Learning’s Impact on an organization. Generally, the quality of your students is an indicator of their future performance.
A few definitions:
Evidence:
Ground for belief; testimony or facts lending to prove or disprove any conclusion
Information, whether in the form of personal testimony, the language of documents, or the production of material objects, that is given in a legal investigation, to establish the fact or point in question
Empiricism:
Practice founded upon experiment and observation
It might be interesting to look at both when looking for the impact of learning on an organization. Gather the evidence. Be skeptical. If the evidence negates your hypothesis (or the goal of the learning project), go down the trail of questions as to why? Research it. Embrace the situation and question if the training was actually bad, or if maybe it’s an implementation issue. We are all academics as LET professionals. We research. We question. We need to apply it when it comes to organizational impact.
General Principles for Research
Pose significant questions that can be investigated empirically
Link research to relevant theory
use methods that permit direct investigation of the question
Provide a coherent and explicit chain of reasoning
Replicate and generalize across studies
Disclose research to encourage professional scrutiny and debate
Stop thinking of Learning as a solo-artist. Learning in an organization is part of the symphony that lends to performance.
There’s no magic number, calculation, bullet to prove your impact on performance. Build the case with a preponderance of evidence to get to Impact.
Impact: there’s no “definition” of impact, because it’s contextual to the organization. But one way to think of it is a preponderance of evidence and empirical study that make the case for what you’re doing in relation to the organization’s business goals. It’s not something that can be monetized, but it is something that an organization can “define” for itself. Reach for the available metrics — sales numbers are there, and you can measure them pre and post. How can you measure Leadership? You can’t.
I’m at the SCORM Technical Working Group meeting and the first exciting thing I’ve heard so far has come from Valerie Smothers with MedBiquitous, talking about Virtual Patients — a model for reusable case studies to be exchanged for medical simulations around the world. Each Virtual Patient has metadata describing their patient data, media resoures, model for data availability, activities and the player that’s needed to play the simulation — that connects to the user interface, the learner profile and tracking.
And they are working with universities in Sweden (I think — maybe Finland) and the US on having working prototypes — and they work in SCORM-based Learning Management Systems.
After about a full day’s worth of talking about cleaning SCORM’s past, we finally get a taste for the future.
I’ve been struggling for weeks to present a comprehensive strategy that makes sense to both a wide audience at work (and me)… and it’s not easy. It’s probably the first time in my career as a knowledge worker that I can’t just offer up a comprehensive solution off the top of my head, mostly because there are a lot of moving parts — too many for me to articulate without capturing them all in some way.
So, in a fit of not-quite-desparation, I downloaded http://www.mindjet.com/usMindJet, which is the same mind mapping software demonstrated at Learning 2006. And within an hour of using it, I have produced the beginnings of a pretty comprehensive map of what’s been on my mind about strategy.
The software couldn’t be easier or more intuitive to use. I would encourage everyone to give it a try, especially if you’re at the planning or mapping stages of capturing content for a project. I think this tool (or something like it) could be used very effectively for couching all the random bits of information from Subject Matter Experts in context so that as you design learning content, you’d have a guide to how any one piece of it fits together.
Flash For Learning is written by Aaron Silvers, Learning Strategist for a multinational corporation. He writes about Flash and Flash-related topics for E-Learning, as well as other musings on organizational learning, within the context of Knowledge Management. Aaron is also a husband, dad, cyclist, musician and nerd.
