As you know, there are many new technologies available now or coming soon that may influence learning and development in different ways. So many that it’s a little hard to keep track of them all, not to mention have an understanding of what they are and just exactly what influences they may have on workforce learning and development.
And of course, as often happens when a new technology comes about, it’s also easy to get a little over-excited and think this is the long-awaited, massive game-changer that will completely change learning and development forever, bringing with it all the solutions to all the problems and challenges we face.
To help us get a better understanding of all these new technologies, to see how we might use them in learning and development, and to see which may have some type of “disruptive” effect in workforce training, we’ve asked Dr. Stella Lee, a learning professional with a ton of experience studying technologies for workforce L&D, to share some of what she knows with us.
Before we begin, two quick points: First, we’d like to thank Dr. Lee for sharing her time, knowledge, and experiences with us. And second, know you can watch the recorded video of this discussion immediately below or click the MORE button to read the transcript I typed up for ya 🙂
And with that said, let’s start learning about disruptive technologies and their use in learning and development from Dr. Lee.
Convergence Training: Hi there, everybody, and welcome. This is Jeff Dalto of Convergence Training and we’ve got another one of our semi-regular webcast/podcast/interviews today.
Today we’re really excited–we’re over in the world of learning & development (L&D) and we’re talking with Dr. Stella Lee. Stella is the owner of a learning consulting firm called Paradox Learning. Stella is modest and told me in advance that she didn’t want to begin by explaining her own experiences, but briefly I’ll tell you that I’ve been aware of Stella for several years on LinkedIn, I read all of her stuff and I’ve learned a ton in doing so about today’s topic, which is disruptive technologies in learning, including what they are, what are some of the promises, what are some of the drawbacks, and some cautions and opportunities.
Stella has been in L&D for 20 years. She started in academia, from there she to the government and then to the private sector, and for the last five years she’s been working as a learning consultant at Paradox Learning, her own company. And I’ve seen Stella talking at any number of impressive places on any number of impressive topics–especially in the last year–Google, Amazon and elsewhere. She’s a really great source of information, so we’re happy to have Stella with us and with that, Stella, hi and how are you today?
Dr. Lee: Hi Jeff, it’s great to be here, thanks for having me.
Convergence Training: Oh, our pleasure. Well, I guess to get right to it, if we asked you here to talk about disruptive technologies in learning, maybe you could start by telling us what this means–disruptive technology, what the heck is that?
Dr. Lee: (Laughs). I know, there’s a little bit of confusion there. The way I look at it, the way the definition goes, is anything that forces you to rethink or transfer the way you do business.
Think about Blockbuster, the video store. So what disrupted Blockbuster? Netflix. Or more so the technology behind it, which is on-demand streaming (video). And so that forces the industry to change, either to rethink how they would do their business or it would force you to go out of business in the case of Blockbuster.
So that’s what disruptive technology does, fundamentally changing how you go about conducting your business as usual.
Convergence Training: Great. So, since you talked about Blockbuster, the one brick-and-mortar Blockbuster still left is near my current hometown. I’m outside of Portland, Oregon and Bend, Oregon has the last Blockbuster.
Dr. Lee: Oh, there’s still one left?
Convergence Training: Yep, still one left, and I read a little newspaper article about it, and I’m not sure to what they attribute their continued success. I’m sure it’s something–it’s not something like serving coffee or anything, but it’s something.
Side note: Here’s an article about the last brick-and-mortar Blockbuster, which is in Bend, Oregon (please note in the video I incorrectly said this was in Eugene instead of Bend–both are great places.)
I guess some other examples might be the hotel industry with Air B’n’B, taxis with Uber, Lyft, and such, and to your point, maybe a lot of industries might be facing that kind of thing, including learning and development.
So to that point, and I guess my next question, how do you see some of these disruptive technologies affecting learning and development professionals, either now or in the future?
Dr. Lee: I think technology is just one driving factor that’s affecting L&D. It’s coming to maturity now but we’re also experiencing globalization, we have a much more mobile workforce, we’re working with more people and there are many, many people coming into–I’m based in Canada, and I’m pretty sure it’s similar in the United States–there’s an influx of immigrants, and that also forces us to change the way we work. The aging work force is another driving force at play, and there’s a lot of talk about the Millennials, but at the same time the largest part of the workforce is 55+ and it’s getting bigger–that number if getting bigger in Canada but I believe in the United States it’s also substantial.
So all these implications are things we need to think about along with how technology is going to impact how we do work, particularly what role L&D needs to have. I think the fundamental question is not just how technology changes learning & development, but the more fundamental question is “What kind of problems are we trying to solve and how can technology support that?”
There’s a lot of talk in L&D about how we can get a seat at the table, and there’s a lot of talk about how can we remain relevant, so I think technology is part of that solution, it’s here to support that solution, but it’s not necessarily driving what learning and development needs to do.
I think we need to be smart about that. We need to stay on top of what technology can and cannot do, what are the pros and cons, and at the end of the day, technology may not be the solution that you need, but you need to be able to articulate that.
Convergence Training: Great. I like that your pointing out that technology is not necessarily a silver bullet that solves everything, AND that you’re saying that learning technologies don’t exist in a silo, there are all of these other things going on here too–a global workforce, an aging workforce, an increasingly mobile workforce, people working in multiple locations. And if it’s not exactly the same in the US as it is in Canada, with the largest percentage of the workforce being 55 and over, it’s certainly similar and a lot of our customers struggle with that as well (having a workforce that’s aging and nearing retirement).
So I like that you’re saying that we shouldn’t think of technology as being in a silo, and I also like that you’re saying that if learning and development wants a seat at the table and wants to continue providing value, we need to continue focusing on solving problems.
I didn’t plan on this question, but what kind of problems should we be focused on instead of simply being entranced by the shiny, glittery ball of technology?
Dr. Lee: Well, every organization, every business has their pain points, right?
If you listen, there are “fake” problems and there are real problems.
An example of a fake problem is when someone comes up and says “We need an email training course,” or “We need a training course on how people can run more effective meetings.” You need to dig a little deeper than that in order to understand if this is translating into a real pain point–is this something that matters in terms of your business goals and the strategic goals of your organization?
I think we need to get better at drawing out those questions and analyze it.
I believe, for example, when you’re talking about data-driven organizations or evidence-based L&D, that’s where that came from–is not just to take people’s “ask” when they say they need a course, not just to take that at face value. We’ve done that in L&D for so long. People come and say “We need a course on email etiquette” and we answer “OK, we’ll build a course for you.” I think that’s been going on for a very long time, and one of the best analogies I read elsewhere is, if you’re a patient coming to your doctor, and you say “I have this list of symptoms…my stomach is hurting, my back muscle is pulling, and so on,” the patient is not going to tell the doctor “And as a result, I need an operation.” No, right? You don’t provide a solution to your doctor. You tell your doctor your symptoms, and your doctor says”OK, based on these issues, I recommend these remedies or these solutions.”
The same should happen with L&D. We should not just take solutions people suggest and give the requested solution to them. Not every problem needs an operation, just as not every problem needs a course, and some of these “problems” are perhaps not even a problem, but perhaps it’s just something like people misunderstanding issues.
So I think, start with that mindset, it’s critical. Did that answer your question?
Convergence Training: Yep. I like that fact that you’re talking about digging down to real performance problems and not being someone who gets told to build a course.
The L&D professional Arun Pradhan makes the same point, he speaks at conferences on “How to Be Something Other Than an Order Taker,” which I think is what you meant.
Dr. Lee: Yeah, and I think in addition to that, we need to not just push back, but to push back with facts, with data, with evidence, with something to convince people. I think we need to be better at that, to be able to present both sides of the argument.
Convergence Training: So, with the caveat that we need to be focusing on results and identifying and then solving real problems, and that technology isn’t always the best solution but it can be part of our problem-solving quiver, I was going to walk you some different technologies, both well-known and those that are just coming in. And the first I want to talk to you about is the learning management system, or LMS, kind of a classic. I wonder if you can tell us the state of the LMS and the LMS industry today and where it’s going.
Dr. Lee: Sure. OK, I think the LMS is still by-and-large the main technology that organizations use for delivering learning.
I know there’s a lot of talk about whether the LMS is on the way out and is it relevant, there’s a lot of debate about that, but at almost all organizations, that’s still the main platform for delivering training.
I think what I’m seeing more now is there are a lot more supplementary tools and technologies along with that. The LMS market itself, that’s sort of constantly in flux. I see a little bit more converging of the market space, this year particularly, the number of LMSs available dropped down a bit. It used to be 700 or so, and now it’s gone down to 600-something. Either they went out of business, or they pivoted and did something else, or they merged/got bought by different companies. My mentor and I keep a master listing of the LMSs that are out there, so we’re constantly updating and changing that, so we’ve noticed the numbers have gone down this year.
So, you know, and there is a lot more fragmentation of the market, so it becomes a lot more specialized, too. There are gamification platforms, LMSs that specialize in microlearning, LMSs that specialize in collaboration and social learning, LMSs that specialize in safety training, there are also industry-specific LMSs out there, right?
And at the same time, I also see there’s a bit of an identify crisis out there. There are LMSs that are really a CMS, a content management system, but it provides a lot of learning management function.
There’s also, this year, or maybe starting the beginning of last year, a new term–learning experience platform (LEP)–so I see a lot of these companies coming up from that perspective, and also content curation is getting quite big in that space. As far as AI and machine learning, of course they’re the buzzwords of the year in learning and development and particularly with LMSs, I have seen a lot more LMSs this year saying things like “We’re AI-enabled” or “We’re powered by machine learning recommendations for your learning path.” So these are some of the things I’ve seen this year.
Convergence Training: Great. I’m glad you mentioned artificial intelligence (AI) and machine learning, and that’s going to be something I follow up on with questions later.
Dr. Lee: Sure. I know, I don’t think we can have a conversation about disruptive technologies without mentioning that.
Convergence Training: Right. But I have of course heard of learning experience platforms (LEPs), but for those who haven’t, or even for those who are new to the idea of an LMS, I was going to ask if you could tell us what a learning experience platform, or LEP is, and how that’s different from an LMS, and maybe in the same follow-up tell us a bit about a learning record store (LRS) as well.
Dr. Lee: Yeah. There’s a lot out there.
The learning experience platform, or LEP, that’s still a little bit muddy. I don’t blame outside this field not understanding. I think those of us inside the field don’t understand it too well.
I think the way I see the primary difference is that the learning experience platform really focuses on the content curation part of it, in terms of how do you personalize and make recommendations for various types of content instead of just a system for administering and and hosting learning, there’s that focus. So that’s the main difference. And then of course there are additional little nuggets of features.
And a learning record store (LRS), a very new and elusive term, it came from–this is going to get kind of technical–it came from the back in the day when elearning didn’t have a standard. So we were trying to develop a standard and one of the main standards we began to use is SCORM, 1.2 and 2004, there are two SCORMS that are really similar, and it’s a still a standard that we hold near and dear to our hearts. The whole idea of having SCORM is just making sure that a learning activity or learning package would behave the same no matter what learning management system (LMS) you put it into. But the limitation of SCORM is that it only measures what is going on within the learning activity and the learning platform (or LMS) that you put it into. It’s a little bit limited in the way that you measure things. As opposed to the newer standard, which is known as Experience API or xAPI or Tin Can (though I think Tin Can as a term is going away a bit, I don’t see it as much as xAPI), so I think with xAPI the promise behind it is it’s much more integrated in the way you can measure things. So you can measure learning activities that happen outside an LMS, such as if you watched a video, you watched it for how long, you clicked on it, you paused where, you completed it, and what did you do next after that video? So, a lot of the trajectory learning, if you will, and also the various activities happen in and out of an LMS, so the learning record store is really a place to communicate and store this information.
Convergence Training: Is it your experience that most of these are built-into an LMS, bolted onto an LMS, or kind of separate?
Dr. Lee: Yes.
Convergence Training: All of that–alright.
OK, so great. Nice intro the LMS, and I guess another standard, time-tested, or older form of learning technology is the elearning course. I wonder if you could talk to us a little bit about that–its use and maybe its future as well. Can you touch on that a little for us?
Dr. Lee: Yeah, I think it’s still the thing to do, you know? The course design and the tools have gotten a lot better over the years. There’s a lot more choices in terms of tools you can use.
But I think by-and-large a lot of courses are still page-turners. It hasn’t gotten to the point where there’s a lot of innovation in courses.
What I see more is, in terms of courses, people are focused more on learning experiences, or more immersive learning, or more about learning moments, or chunks, or nuggets, right? So I think there is a move, shifting from talking courses, per se. And that’s true in the academic world, too, companies like Coursera and edX also are trying to shift their mindset too, in saying we don’t have to think of a course as being equal to “X” minutes of time, or it has to have “X” types of information in it. I think now we start thinking more a little about looking at the needs of learners, how do we support performance, looking at providing support at the moment of need. It’s not just about a course format.
You know, when you want to learn how to change tires, do you take a course? Or do you just need to know steps 1-5? Or do you need a mentor to talk you through it? So think about it from a performance perspective, what is something you can push? I think that’s why mobile learning, it’s been around, but I think now we also have the maturity of platform and availability of data and wifi access and all that to support it, and that’s why it’s popular, and that’s why microlearning is popular. That’s why sometimes game-based learning is popular, because it’s the interactive nature of supporting what people need to do.
So I see that it’s moving a little bit more in that direction, in terms of elearning courses. I don’t even like to talk about “courses” per se, to get people to think outside of that, to think about “Really, what do you want your target learners to know, and how do you know they know that, what kind of support can you give them and what kind of assessment can you test that they actually get it?” So that’s what I try to focus on.
Convergence Training: So that goes back to your original points about performance support, being aligned with business goals, identifying problems and solving problems, etc.
I like that you pointed out that there are a lot of tools that make this better, and that’s true, and we can do a lot of things, but also (and we see this at our work too) the increasing emphasis on performance support, shorter courses, microlearning like you talked about, delivering support at the time and place of need using mobile devices.
And I love your analogy about fixing the car tire. I have spent much of this year working with plumbing problems, just sinks that were dripping, and of course I’ve spent a lot of time on YouTube looking for the perfect three-minute video on how to replace that thing on my sink.
Dr. Lee: I’ve done that too, actually, and YouTube is my go-to.
Convergence Training: Yeah. For everyone at home.
OK, great. So…
Dr. Lee: Oh, can I add another point, though? Not so much courses, but the creation of courses and content, I also see more of a trend of L&D outsourcing instead of using in-house capacity. I think it’s just getting to the point where you can’t keep up, right? You can develop in-house capacity to build more traditional course-based learning, but I see a lot more specialization being outsourced to third-party vendors, and even just the volume of courses being needed, sometimes an in-house L&D department can’t keep up, so I think the trend to outsourcing continues.
Convergence Training: That’s interesting that you added your experience there. We see the same outsourcing to third parties at our workplace as well, and it’s easy enough to understand when you think of all the things an L&D professional has to do, or even we do a lot of work with safety trainers, and it’s the same thing for them.
OK, great, thanks for adding that.
So we talked about the LMS and about elearning courses, and in doing so we already mentioned briefly artificial intelligence (AI) and machine learning. I wonder if you could open that bag now and introduce us to some of the issues–talking about artificial intelligence, also known as AI, machine learning, perhaps explaining how those are the same and different, and then in particular if you could talk about:
Dr. Lee: Yep. So, artificial intelligence has been around for about 60 years, so it’s not really new. I think what’s interesting is now we have gotten to the point where technology can actually support that. We have cloud computing, we have the ability to host these large amounts of data, we have tools sophisticated enough to do analysis, so I think that’s why there are a lot of innovations in this space right now. And also applications that have started really tackling some problems.
Machine learning is a subset of AI, because AI essentially is how can computers fabricate intelligence, right? Intelligence such as the ability to make decisions or judgments based on observations or based on the availability of information to either select a path or analyze the content to make a recommendation or…there are all kinds of abilities for intelligence, as you know, but machine learning is a very specific way of explaining intelligence, of making decisions. It’s based on pattern matching, it’s based on algorithmic prediction to then come to a conclusion about things. So machine learning is more narrow, if you will, than what artificial intelligence can do.
And of course, even with artificial intelligence, you get more the general AI and the more specific AI. So if you think about intelligence as a general concept, it could be about intuition, about wisdom, it could be emotional intelligence, it could be all kinds of things. It could be the ability to draw inferences based on certain data points. And then specific AI is about understanding very specific sub-sets of that. And machine learning is more narrow, drilling back to basically using algorithms to identify issues.
So I think now we are more focused on machine learning, as opposed to AI, because AI is such a big field. So a lot of times, when people in L&D talk about artificial intelligence, they’re really talking about machine learning.
Definitely, you and I both know that in the past few years, discussion of that is all over the L&D space, in terms of how we can use it, what are the implications, is this going to replace jobs, how do we regulate things?
One of the big topics right now is about ethics or accountability. Not just at the company level but at the government level, international regulatory body level. We know that companies like Google, SAP, Microsoft have now what they call AI principles for ethical use, but they’re companies, so they still are primarily putting these out based on the best interests of the company. What I’m interested in seeing is at the government level, what kind of thoughts do we put into that.
And why does it matter? Because you when talk about AI and machine learning in L&D, what do we use it for? A lot of the times, we use it for personalized learning, right? Understanding how a learner or an employee is performing, perhaps, in the LMS, and based on this person’s ability to complete or not complete a certain learning, for example, that might provide an intervention or provide support or do some coaching for them. So the danger for that is (a) when you’re collecting this type of data, who owns it? Are your employees aware that they’re being identified and analyzed at that level of detail? And (b), the implications for that person–is that information linked to their performance review, for example? And of course, the other danger in that is are you stereotyping this person? Perhaps this person is having an off-day or an off-week; sometimes past behavior is not the best prediction. You know, we’re human, we go through ups and downs. I know we’re wary of stereotyping people, but there’s a danger in that, too.
And we haven’t really talked about that, as organizations we haven’t really looked at it, as countries or as international governing bodies…I understand that the UN is developing some sort of guidelines on that, but it’s not out there.
Another danger is when you stereotype, and you say “this person is at risk” (there is an LMS out there that puts people into risk quadrants, for example), whether you’re failing or succeeding, right, and it’s a prediction to say something like “based on the number of logins into the LMS you’re at risk.” But just because this person isn’t logging into the LMS frequently doesn’t mean they’re not learning–perhaps they don’t need as much time, or maybe they downloaded the material offline, and maybe you don’t know they spent all that time. So those are really nuanced things that are hard to know, and I don’t think that machine learning and AI are “there” in being able to measure those things, and we haven’t come up with an overarching plan to say “This is how we measure that” and “If there are gaps, how are we going to mitigate those gaps?”
So those are my concerns, and those are the challenges I see.
Convergence Training: So, I do want to get back to some opportunities, but I love that you talked about the challenges as well. And one challenge I’m aware of, only because I’ve heard you discuss it in the past and I’m hoping you’ll discuss it again today, involves a “woman” named Jill at George Tech…
Dr. Lee: Jill Watson, yes!
Convergence Training: …and this has to do with the use of machine learning and what the machine learning can do based on the information coming in.
Dr. Lee: That’s right, but before I talk about that, I want to make sure I also present the promises with AI, because it does have tremendous opportunity for us to personalize or support learning in a way that maybe we weren’t doing in the past.
Not that we as instructors weren’t providing learning in a personal way. We do this all the time, right? If you look at a classroom of learners, and if it looks like they’re falling asleep, you might change your tone a little bit, right? That’s personalized learning.
But I think it can help in a large scale, because you can’t personalize for a 500-person audience, but a machine can. It can push out learning based on certain characteristics of your learning. So to scale, it’s helpful to look at analysis of the learning data to help interventions, supporting course designers is always great. As long as the power remains with humans to change that, and not having a machine overriding and making assumptions about things.
So, about Georgia Tech, this is an interesting story and I love it. For those who don’t know, a few years ago, for one of their online Masters level computer science courses (the course is huge, it always has more than 300 students enroll), so they always have teaching assistants to help answer questions on discussion forums–the students post questions, and most of these questions are pretty mundane, like “When is this project due” or “How many pages do I need to write” or “What citation format do I have to use?”
So Georgia Tech traditionally hires a lot of TAs to help answer these questions from students. And in 2013 or 2014, they decided to introduce chatbot to their course to help. They were still going to have human TAs, but they added a chatbot TA without telling the students. And they decided to call this chatbot TA “Jill Watson” after Watson, the IBM computer.
So Jill Watson was there along with all the other TAs for the duration of the semester, and she did such a good job that students (a) didn’t know she wasn’t human and (b) students were like “She’s great! We should nominate her for best TA of the year.” So it was quite interesting.
But the challenge of using that is you can only train and feed information into a robot based on the information you have. And so you can only feed information to this robot based on the questions students have asked in the past. So what do most students ask? Things like “When is this assignment due” or “How many pages do I write,” those kinds of things, and along with questions like that, you want to train your robot to handle conversations that are more personal, that involve more conversational data, and this is where it falls short.
Most students in the past were male students, so there’s a really interesting paper out there called “Jill Watson Doesn’t Care That You’re Pregnant,” and it talks about when a male student talks to Jill Watson, and says something like “I just found out I’m going to be a father soon,” Jill Watson will reply and say something like “Congratulations for welcoming your bundle of joy!” She had a very appropriate response to a male student talking about the experience of being a father for the first time. So, when a female student said a very similar thing to Jill Watson, saying “I just found out I’m pregnant, I’ll be due before the end of the semester,” Jill Watson had no data set from the past to inform her how to respond appropriately so she just said “Welcome to the course!”