Flash For Learning

Inspired a bit by Tom King’s article on authoring tools, I started playing with Google Trends and was a little interested in how E-Learning is faring against the notion of Performance Support — my idea being that E-Learning is stuff we have to evaluate, manage and track the learner’s interaction with — and performance support being, perhaps, not so rigid.

Here’s my not-so-scientific report:

E-Learning is by far more popular in searches, though the volume of searches definitely has dropped from 2004 (which we can discuss by itself ad nauseum as far as reasons why people are searching less for E-Learning). But in 2007, in particular, the notion of “Performance Support” has gained much more buzz in news references. Now, this can mean a lot of things, but the fact that E-Learning never makes a blip in the news probably says something, too.

As we make E-Learning smaller and more granular… are we naturally evolving a model of instruction to something more like Performance Support?

By the way — as an interesting post-script to this, the top 10 regions, in order, who are literally looking for Performance Support, are…

1. South Korea
2. India
3. Singapore
4. Australia
5. Taiwan
6. United Kingdom
7. Canada
8. United States
9. Netherlands
10. China

Anyone want to take a stab at how employee productivity by nation matches up with this ranking for a search?

“We’re still trying to figure out what the questions are.”

Accenture had an opportunity for Mobile Learning. The Business need they had was to increase the speed and ease of uptake of corporate required training by senior executives. They also needed to provide important information at the moment of need.

Fortunately, they have a receptive audience that would benefit from and use a mobile approach to training. Mobile devices for their senior executives are enabling technologies. Senior executives make up a large population of our mobile device users.

• January 2007 = 6,000; October 2007 = 14,000

The Future Scenario:

1. The SE receives email aobut required training on their mobile device just before heading to the airport.
2. SE selects the “myLearning mobile solution” option for this training from her mobile device.
3. myLearning automatically enrolls the SE in the course and the SE has access to the mobile training course
4. SE takes a 15-20 minute segment of training and successfully completes an assessment en route to the airport.

Obstacles:

• Flash not supported as it needs to on mobile
• LMS Communication issues
• Configuration issues with the broad span of devices that need to be supported (dozens with different OSs)

Accenture decided to pilot it with a 12-screen prototype, intentionally selecting 12 screens that would present a challenge porting from their E-Learning to mobile. The decision to do this rather than designing from the ground up was intentional, as SEs wanted an experience that was as close to traditional E-Learning as possible. Accenture started in-house with a live demo with a very small population, and then they went remote for a pilot. Both groups came back and said they’d use it (about 90% in each group) when asked — stating they had a “better than expected” experience with the prototype.

Phase 1: Prototype

• Goals
  • Develop small prototype
  • Test with SE
  • Refine future scenario
• Key Research
  • Audience reaction
  • Insight on mobile technology
  • User interaction design considerations and trade-offs
• Content Interaction Definition
  • Ten minutes of content from existing ethics course
  • Simple text and graphics
  • Two types of interactions: multiple choice and fill-in-the-blank

Phase 2: Field Testing

• Goals
  • Develop a full Ethics course on a mobile device (no LMS interaction)
  • Test with a broader audience
• Key Research
  • Audience reaction and insight use of mobile devices
  • user interaction and design considerations
  • Preliminary infrastructure research

Phase 3: Infrastructure (including LMS integration for the first time)

• Goals
  • Develop LMS integration for mobile training delivery
  • Create a seamless experience from notification and enrollment on through to completion
• Key Research
  • Enrollment and completion communication to the LMS

Phase 4: Rollout

• Goals
  • Expand Accenture’s mobile learning asset catalog
  • Increase the richness of the user experience
• Key Research
  • Determine how to make the user experience better

Effective Design

Elements of effective designa nd usability of a mobile learning solution:

• Refreshable content
• Appropriate chunking of focused content
• Concise writing
• Effective and logical navigation
• Simple graphics

Doug Lynch (The University of Pennsylvania - The Wharton School of Business) was one of the keynote speakers at Sunday night’s kickoff to the Learning 2007 conference, and he was quite obviously the smartest man in the room — so naturally, I flocked to his session this morning.

“Thomas Jefferson once observed the increasing strength of evidence for Isaac Newton’s theory of gravitation that Newton ‘indulged in reason and experimentation, and error fled before them.’”

Newton developed theories, and some of them have held up. LET professionals have also held up theories, and some of them work, if you don’t sweat that some parts hold up better than others. But people talking about principles as “facts” aren’t facts — they’re just assertions.

There are no scholarly writings on ROI on Google Research. None. People have been able to demonstrate the economic returns on learning, but there is more than what can be demonstrated in these studies. Zagat ratings are coming out now for health care, so there’s evidence to support the “effectiveness” of the quality of health care, but it may not be all the evidence you want.

So here are some questions to help deconstruct ROI:

• What level of sophistication do you need in terms of understanding evidence of impact?
• What tools do you use to gather evidence of impact?
• What tools do you use to analyze performance / learning?
• Do you use any methods to evaluate implementation in addition to the intervention? If so, how?

What forms of evidence do you need to determine if what we do impacts business goals?

ROI has a very narrow meaning to CLOs, Finance people and Economists. As a learning profession, we throw around the term ROI, when we should really be talking about “Impact.” The person hearing you when you use the term “ROI” is thinking something completely different.

A problem with our world:

Man is naturally metaphysical and arrogant and is thus capable of believing that the ideal creations of his mind, which express his feelings, are identical with reality. From this, it follows that the experimental method is not really natural to him (Claud Bernard, 1865).

If you want to find out what is going on, you need to look at what is going on (Yogi Berra)

Learning is loosely coupled. What LET professionals do is a messy business. We have norms in our world

Kirkpatrick’s Four Levels

• Reaction of student
• Learning
• Behavior
• Results

So… what are some underlying assumptions about ROI?

Ratio of Net Benefits: ( Benefits - Costs ) / Costs

Let’s deconstruct benefits (or profit). What goes into calculating the benefits of training?

• Sales?
• Income?
• Stock Price?

The idea behind ROI is that it is a number that is monetized. You define it, you measure the costs and the benefits and then you monetize it. Are these things easy to do? No, they’re not. Stock price is a net-present value of what the company is doing today and it’s speculation on how it will do in the future. Stock price also gets at some other things that are going on — like a company’s performance in relation to the rest of the market.

For you to say that you have impact, you need to have control of everything that’s going on. To use ROI as a measure of learning’s impact, you’d have to control the entire market and factor non-related issues out — everything from competitive advantages, the cost of raw materials, fuel costs… you get the idea. And even if you could factor those things out… you still have to factor out all the human-factors to filter down to just Learning’s impact on an organization.

The relationship of Learning departments and HR departments probably is a better indicator of Learning’s Impact on an organization. Generally, the quality of your students is an indicator of their future performance.

A few definitions:

Evidence:

1. Ground for belief; testimony or facts lending to prove or disprove any conclusion
2. Information, whether in the form of personal testimony, the language of documents, or the production of material objects, that is given in a legal investigation, to establish the fact or point in question

Empiricism:

1. Practice founded upon experiment and observation

It might be interesting to look at both when looking for the impact of learning on an organization. Gather the evidence. Be skeptical. If the evidence negates your hypothesis (or the goal of the learning project), go down the trail of questions as to why? Research it. Embrace the situation and question if the training was actually bad, or if maybe it’s an implementation issue. We are all academics as LET professionals. We research. We question. We need to apply it when it comes to organizational impact.

General Principles for Research

1. Pose significant questions that can be investigated empirically
2. Link research to relevant theory
3. use methods that permit direct investigation of the question
4. Provide a coherent and explicit chain of reasoning
5. Replicate and generalize across studies
6. Disclose research to encourage professional scrutiny and debate

Stop thinking of Learning as a solo-artist. Learning in an organization is part of the symphony that lends to performance.

There’s no magic number, calculation, bullet to prove your impact on performance. Build the case with a preponderance of evidence to get to Impact.

Impact: there’s no “definition” of impact, because it’s contextual to the organization. But one way to think of it is a preponderance of evidence and empirical study that make the case for what you’re doing in relation to the organization’s business goals. It’s not something that can be monetized, but it is something that an organization can “define” for itself. Reach for the available metrics — sales numbers are there, and you can measure them pre and post. How can you measure Leadership? You can’t.

At work, I’ve successfully gotten my group on-board with implementing BaseCamp as a collaborative project management tool. Initially, it was to meet a request from my boss to replace a Word doc and Excel spreadsheet that had to be updated by everyone on what projects were being worked on (our weekly status reports) and how much time we were spending on these projects (resource allocation reports).

Basecamp is so easy for the end-user, it’s going to be rolled out to Project Managers in the organization, too. Finally, we’ll have everyone working on the same project on the same page — literally.

One thing that BaseCamp is not so good at is reporting. I have a pretty non-techie group and there are a lot of clicks involved (although pretty easy) to read into what everyone’s doing on their projects. Fortunately, BaseCamp has a documented API (one of the reasons I wanted BaseCamp was its extensibility) and there are a number of scripts emerging to make use of it.

Aaron Quint at QuirkyBlog posted a PHP wrapper for pulling information from a BaseCamp project. After having to fumble around with DreamHost’s implementation of PEAR (adjusting some relative paths to hard paths in some of the PEAR scripts as well as Quint’s Basecamp.php class), I was able to pull a list of projects lickety split.