S11: E2 Beyond the Screen: Social Learning with VOISS in VR environments with Dr. Sean Smith

Summary:
In this week's episode, we delve into the cutting-edge realm of virtual reality (VR) technology, specifically tailored to assist students in mastering problem-solving within social contexts through simulated scenarios mirroring real-world challenges. Our guest, Sean Smith, serves as the Principal Investigator for Project VOISS, alongside his role as a professor in the Department of Special Education at the University of Kansas. Sean provides valuable insights into the project's overarching goal: fostering knowledge and skills acquisition through evidence-based practices geared towards addressing social issues among middle school-aged students. VOISS stands as a groundbreaking computer-based VR platform, immersing students in diverse school environments such as classrooms, buses, and hallways. Here, they navigate through interactive social narratives and confront situations demanding problem-solving skills. Join us as we uncover the innovative strategies behind VOISS, aimed at empowering students with essential social competencies for navigating the complexities of the real world.

Resources:
VOISS Advisor: https://voissadvisor.org/  

Sean Smith Email: seanj@ku.edu 

Highlights:
-Targeting Social Competence: Providing immersive experiences for individuals lacking social skills. 

-Evidence-Based Practice: Focus on developing social knowledge and skills through proven methods. 

-Simulated Educational Environment: Students engage in less immersive VR scenarios mimicking school challenges. 

-Guided Interaction: Narrator and coach guide students through scenarios, interacting with avatars. 

-Targeted Questions & Skill Transfer: Children are prompted to respond to questions tailored to individual growth needs. These questions help guide them to make decisions/ mistakes and gain consequences in a confined environment. They can learn from them and apply them in real-life situations.  

-Domain Application: Receptive and expressive communication domains crucial, with applications in school, home, and community settings. 

-Practical Classroom Implementation: Steps outlined for middle schoolers, aligning curriculum with targeted skills and domains. 

Transcript

Lisa Dieker 0:07

Welcome to practical access. I'm Lisa Dieker.

Rebecca Hines 0:09

And I'm Rebecca, Hines. And Lisa, today's topic, virtual reality. We are so about the virtual experiences. Tell us about our guests.

Lisa Dieker 0:20

Yeah, so we're very privileged to have my friend and colleague at the University of Kansas with us today, Sean Smith, and who he can tell you more about himself, but lots and lots of work in self determination and advocacy. How's that for a new word I just made. So welcome, Sean. Thanks for joining us.

Sean Smith 0:36

Thanks for having me.

Lisa Dieker 0:38

Yeah, well, we're really excited. So can you tell our listeners a little bit, you know, this is a podcast so they don't get to see anything. But we'll link your site there. Can you tell them a little bit about VOISS and about that project and what they might experience if they choose to go out and visit your site?

Sean Smith 0:54

Sure. So first of all, thanks for having me on. And just give a little context. Well, here VOISS itself is a virtual reality experience that goes across levels of immersion. So iPad, Chromebook windows were a little less immersive environment, to the fully immersive environment and the Oculus, and came about this and the focus is on social competence. And I can give the whole review if you want, or I can stop for a moment you let me know. Okay. So here's the whole review. So we're trying to give experiences for individuals, particularly individuals on the autism spectrum disorder individuals learning disabilities, individuals that have social skill deficits, social emotional learning, social competence, challenges. But sadly, depending on what state you are, depending on what language you can use in reference to that, but we know very seriously, this has been a major challenge for a number of individuals and post pandemic. It is a challenge for a lot of our individuals. And although we've developed it, particularly for individuals with disabilities, our primary users have been individuals who would be seen as typically developing and not identify with a disability. So our overall focus, though, is to develop knowledge and skills. And the way we've created it is we've wrapped it around some good avenues in social narratives or social stories, which in and of themselves are evidence based practices to teach social emotional competencies, particularly for individuals with autism. But we tried to take it a step further, with the idea that well, we need a safe place to practice, we need more realism, we need more contextualization, and we want to go beyond knowledge and skills, we want generalization. And so taking virtual reality, which again, people think of a headset, so really, it's a simulated learning experience in a less immersive environment, on the primarily the iPad, Chromebook, etc. That's the primary users right now. We create a series of scenarios around this idea of social stories, around 180 plus skills, 140 scenarios and the scenarios lasts between 3, 5, 7 minutes depending upon the user, and basically walks them through an experience at school experience, gym environment, hallway environment, classroom environment, you name it, bus environment, where there are basically presented with a social challenge. And we've organized the 140 scenarios, under 10 specific domains like receptive language, or receptive communication, self advocacy, things of that nature. And in that the student is stopped along the way, for from a variety of reasons. First of all, they're stopped along the way with a social coach, which is a narrator, it's all prepackaged. It's a story. It's all organized,depending upon the media, the user. And so as the user enters, they'll have a social coach and individual kind of walks them through, they're about to experience this in the hallway, they're about to experience this in the cafeteria. And then they'll interact with avatars, which are pre programmed avatars, the script is already written very purposefully, who will make sure it's reliable and valid to actually teach the skill that we're seeking to teach. And the language is very important. And the user then interacts with that avatar, and right now it's very text based, so the avatar that they interact with a series of questions that are presented to them. So someone's in front of them in the locker, someone's sitting in a seat, they like to sit in, a teacher's respond in a certain way, and they need to respond appropriately, or they're trying to check out in the cafeteria line, and the lunch person is asking them a series of questions. And then they're prompted with questions that they need to respond to. They can hear those questions that can read those questions. And then they select to him. They point to him, if they're in the virtual reality, fully immersive. They click on it, if they're on the the Chromebook, they touch it on the iPad, and then it takes them down to pay upon the selection. It takes them down a reinforcement of what they selected, it takes them down an avatar responding potentially in a negative way. It's like you just walk in the avatar like a student would would respond back with what are you doing? Or maybe the social coach jumps in and talks them through about what their selection was and why it was positive or it was a challenge. And so overall, the expectation is that all 10 domains, not all 140 scenarios, not all 180 plus skills, but rather very targeted depending upon the student, which the educator would select. And then the growth would be knowledge and skill. There's an expectation to generalize it into the real world. So I'd say that's in a nutshell. But I think that's in a, like a bag of nuts. How about that.

Rebecca Hines 5:09

A big bag. That's, that's amazing. So, Shawn, let me know if this is a correct summary. So the tools can be used, either with or without glasses.

Sean Smith 5:26

Right. The majority of users have used it with the iPad and the Chromebook. And when we created it initially and go into some language, it's not relevant, but it's important, and that is unity. So Unity is a computer programming language. And do I need to know about it? No. But the reality is, it allows us to make it available in a variety of platforms. And so we knew going in that the headset was our goal, the gold standard, but we knew the majority of our students would probably never be able to access it. And so the Chromebook is primary, but 60% Plus, in our schools is a Chromebook, iPads, windows, etc. So that's how they and then the less immersive experience of it, sorry, interrupt, but we did some research on it. And we found out that the growth across all three platforms, iPad, Chromebook in the Oculus is significant in terms of some reference examinations on their competency development. But it's not significant across the platforms. So they grow. But it doesn't, it doesn't seem to be significant in terms of what system they use, and some of our preliminary work.

Rebecca Hines 6:28

Interesting. And I know, I know, as you have written, and as those of us in the field know, sometimes the glasses, they do help certain kids with distractibility. So certainly a teacher by teacher in case by case decision if you do have access, but important to know that you don't need a lot of peripheral tools to make this work. So can you give us just a just a description if I'm a user, I'm a kid, I log in to the app. What do I see first? You know, what is this look and feel like to me?

Sean Smith 7:14

Sure. So let me kind of walk you through a couple different ways. First of all, the user, they download the app, they select, and they select it on the Chromebook they Play or App Store the iPad, they select thedomain, there's one of 10 domains, they select one of them. Now what let's clarify why they select which one, a teacher would tell them which one to go into. And/ or if the teachers are assigned it, that domain would appear, they select that domain. And then they are the scenarios are organized into tasks. And either those tasks will appear that are assigned to them, or they're told which task to select. And then they go into a scenario when the scenario begins. It literally there's a big white arrow, they click the white arrow, and it just begins to move them into a classroom, moves them into a cafeteria, then you start to walk down a hallway, all of a sudden, they'll start hearing noises if they're in a classroom, or a cafeteria, or a hallway or bus. So all that becomes fairly realistic. And then the scenario begins. The scenario may begin with a teacher starting to talk, it may begin with it's they're sitting at a table with a series of students, and they're being offered the following opportunities to be able to engage. So very quickly, the individual is part of that experience. And you didn't ask it but I'll offer this side, presence is important to us. And so we've done a fair amount of work on do they feel presence? And we're finding out over a very significantly, yes, they feel that part of the classroom, they feel that part of the cafeteria, they don't feel they're part of some sort of virtual experience, they really feel that they're interacting with their peers. So that's how it worked initially for the user. And as they respond, it continues them through until the scenarios ended. And when it's ended, they're thankful for that. And either then they're assigned to the next one. Or if that was the last one, they would simply sign out, they have a navigation tool throughout, they can always exit whenever they feel comfortable. And it will remember them when they return. They can always ask for help things of that nature.

Rebecca Hines 9:02

And this this type of virtual environment for people who aren't familiar with this type of thing. So a big difference is, if I'm correct, the students experiencing it as a first person, so it's them walking through, they're not watching something where someone else is going through when they're just a passive,

Sean Smith 9:22

No, it's very much first person first. In earlier versions, they could actually see themselves, they would see an avatar, if they looked in the mirror. So if they happen to walk by a mirror in the hallway or in the bathroom and things of that nature, we've taken that out. And we may add that in going forward that they get to select an Avatar, but everything's first person, they're seeing things as if they're in that environment. But right now they don't know what they look like. So not selcting an avatar, but the children think they're in the students think they're in a game, and they'd like that game feature. And so that's something we'll probably going to add on in our next iteration that they can select an avatar so.

Lisa Dieker 9:58

Well and I love to having been in it that you also get to make bad decisions and the consequences are real in the virtual environment, but then you do walk away, and they're not real in your life unless you choose to make them there. So I think that's the transfer of skills for teachers and parents listening. It's a great place to do that. And I think the other thing, Shawn, is, it's free, correct, no cost.

Sean Smith 10:19

It is free, supported by the Department of Office of Special Education Programs, US Department of Education. Currently, we're still collecting data. And so folks that download it, and it's V.O.I.S.S, it's an acronym, don't worry about the acronym, but VOISS, and I know it's phonetically not correct. But if you Google the Play Store or the App Store, you'll find VOISS, part one, or part two, it's fairly big right now. And then you'll download five of the 10 domains and each part, but it's free of use. It's very game based. And yeah, as you mentioned, Lisa, you know, the initial reason we went into this was, we need a safe environment, we need an environment to, for lack of better words crash and burn, that would be out of my brother in law's pilot background. He'd say, Yeah, we, we did a lot of crashing and burningbefore we got into the real thing. And that opportunity to crash and burn in real life, to make mistakes, and have someone else to practice with you again and again, and again, is often fairly limited. And sothat's why we created this simulated environment.

Lisa Dieker 11:23

Well, I love that the virtual coach gets to help you too, because then you don't have the adults, you know, being the parent and nagging. And so I am curious of all the domains, you've done a lot of research and have lots of schools using this, of all the domains, which one is most often used in the classroom, and putting on your parent hat, which one would be like, if you're going to do this with your kid, where would you start? Like, what is there a domain, you're like yeah this is the one that seems to be the the teacher's favorite fan favorite, and maybe the best domain to think about?

Sean Smith 11:53

Well now you're asking me, of course, which I should know, I should be memorized in terms of my 10 domain.

Lisa Dieker 11:58

I was just curious if there was one I can only imagine there's so many, but I didn't know if there was a certain skill is that the cafeteria is that the hallway? Like what's that people use the most or find most helpful?

Sean Smith 12:09

That's a good question. So we have not collected data on particularly in terms of that did that use right now? That's it's an interesting question. I'll tell you, though, the popular ones, the popular ones, oftentimes are the receptive and expressive communication. Those are big ones. But there is one and our focus has been primarily where our funder, U.S Department of education, is very much in education right now. I’m sorry about that. I'll say that a little bit better. They are very school based, right. But of course, increasingly, they're thinking family, which is awesome, as a family member of a parent of a child with disability. So the parent and family in the home environment has not been a primary focus. And yet, parents have been once they hear about this, “Hey, how can we get involved?” We have an area, it's called school, home and community. And that's a domain that it seems to be popular amongst the families. But at the same time, the skills in the scenarios overlap. So we organized in the 10 domains, but very possible, you could get an executive and organizational skill domain scenario, but also it's in school, home and community. So yeah.

Lisa Dieker 13:16

Well and I'm wondering, like the cafeteria would transfer over if I go to a restaurant, and someone is at a table and you know, they're by themselves. And it's nine. And I mean, like, I'm just thinking that there's so many overlaps, you know, might not be in the hallway where there's lockers, but I might be in the hallway of a hospital or a grocery store.

Sean Smith 13:35

Let me clarify that. And this is probably really the question you asked initially that I missed. And that was like, for example, we have things in physical education setting. Well, in passing a basketball, there's nothing to do with the skill of passing a basketball, it's turn taking and cooperating. And we just happen to when we identify these skills, we said, well, where across the environment, could we create something that would engage the individual that would offer those opportunities, and we wanted to get it across enough variety, not just in the classroom, realizing a lot of these things happen without the educator there. And of course, like in the hallway, cafeteria, bus, PE, and so yes, the intent is for those to be transferred. I mean, if it's a hallway, they should be able to take it to the hallway, going to a movie, they should be able to take it to a hallway at a department store. You name it. That's an expectation. So hopefully that helps them translate an initial question.

Rebecca Hines 14:28

And so Sean, my final question is for those teachers, or pre service teachers, who wanted to use something like this, or have a need to use tools like these. Can you give us a simply stated like, steps of how would they use that practically speaking in the classroom?

Sean Smith 14:49

In VOISS or virtual reality in general?

Rebecca Hines 14:53

Well, we'll go with VOISS just because it's a tool that we will link to for our audience so they could give it a try.

Sean Smith 15:00

So first of all, our target audience has been middle schoolers. Fifth, sixth, seventh, eighth and ninth, the user base has been fourth grade up. And up as well to high school, we've got a lot of questions about using this post secondary, and our responses, the domains, and the skills are appropriate, the setting may not be appropriate in terms of where they're learning those skills, it's not out in the workforce, etc. In terms of the use our intent is to view a full curriculum, that was what we heard from the field, that's why we've gone across the skills in the scenarios that we've covered. The use by teacher to get started to work with their student as well as a parent is we created an inventory, and then inventories as off voissadvisor.org. And we'll make sure you have that link. And that allows educators as well as parents to answer a series of questions thinking of a student or students they want to use this program with, and that's how they would get started. So the inventory allows him to say, hey, this domain is very much aligned with your need. And these skills are very much aligned in that domain, this is where we'd recommend you get started. And that's where then the student, as I mentioned earlier, when they pop in, they'd have that direction, they'd have those areas to focus in on. And then hopefully, as a teacher, they would be able to give that opportunity, an opportunity for knowledge and skill practice. And then we also have some guidance to say with the progress monitoring we collect. So there's data on the other side, now support them journalizing that in the general education classroom, or wherever that student is. So all those kinds of parts come together to assist that educator and or parent to be able to facilitate this.

Lisa Dieker 16:32

And I know you all have had a lot of fun building this and it's been a lot of work. So thank you for the contribution to the field. So my final question for you, like what's been the best fun, the the most interesting things somebody said, a student has said? What's something that you already are like yeah, this has been fun because of blank? Is there something that comes to mind?

Sean Smith 16:51

You know, early on very, very early on, we were doing some usability testing with a group of science teachers and middle school science teachers, and the science teacher who was very much I think she had been volunteered to do this. She had been, you know, we've done this at the beginning. It was in the pandemic, during the pandemic, we were remote, at least for our trainings. And so via zoom, she getson and she goes, I've got to tell you, you know, I've got 120 students. And over these classes, these kids kept asking me about this game, they wanted to keep playing the game, they wanted to keep playing. I guess she had introduced it when she was not there. And it was a substitute. And she could not figure out what the game was. And so finally, she figured it out. Oh, it's VOISS they want to play VOISS. And that hooked her that hooked her she was I like I said, I think she was volunteered for the project. She started realizing the kids saw it as a game. But she saw what they were learning. And we were sharing the evidence what they're learning on the other end. And that early on was like, you know, because I'm looking at this, I've been living with this and looking at this going, this is not, you know, the games my kids play down in the basement. Clearly, the middle schoolers saw this as a game, which of course yet they're connected. They're engaged, they're wanting to do it. Awesome, because we know what's going to happen on the other end. So that was pretty exciting.

Lisa Dieker 18:14

Great. Well, thank you again, for making the time and we're excited. Our listeners will have a chance to learn and download and play with VOISS. And I know you all are always open to feedback and questions, if there is any.

Sean Smith 18:24

Reach out and hopefully, we can put my email there for folks listening to this. And thank you so much for letting me be part of this. I really appreciate it. Thank you for doing this for the field. This was awesome.

Lisa Dieker 18:35

We really enjoy it because of people like you. So thanks again, Sean. And if you have questions for us, please send us a Tweet at Access practical or you can post your question for us on our Facebook page. Thanks again, Sean. We appreciate your time.