>> Good afternoon, this is fantastic. Welcome to the... Which union are we? The Texas Union. My name is Lorraine Haricombe and I am the Vice Provost and Director of the UT Library system here at Austin. And it's my pleasure today to welcome you to the kickoff of our launching event of what we call The Year of Open. Phil Long, so why don't you, everybody knows Phil but here he is, Phil Long and I arrived here a few months apart I believe. And somebody's already dubbed us partners in crime. (crowd laughs) I'm not sure what the crime is, at least we're open about it. So I just wanted to say thank you to Phil and his entire team who are co-hosting with us. We are doing this together to put on and bring speakers in such as you will hear about in a few minutes. To launch The Year of Open here. I am a very strong advocate as is Phil and I'm assuming all of you to learn more about the open agenda. If you have this sheet, you'll see some of the upcoming events for the year of 2015/2016. There are a lot of events external to the academy that's helped drive it. The urgency and the momentum around open. Today we'll learn more about open educational resources from the guru, I believe, in open educational resources. And there is some national event just earlier this month that makes this a very, very timely opportunity for us to have somebody like David Wiley here. Next year the funding agencies will begin to implement their policies and plans around open data. Another opportunity and another reason for us to get with the open agenda. I want to make a plug here for the libraries because we are going to be very key partners and stakeholders on this campus to lead these efforts. If you need, if your faculty or graduate students and you need help or support in understanding what those compliance issues mean for you, please contact the libraries. Colleen Lyon is your contact or consultant within the library but, please feel free to contact your librarian in the library to make sure that you learn more about what these might mean for you. Especially if you're receiving grants from some of these funding agencies, okay? I also want to let you know that we have an institutional repository called UT Scholar Works. We will launch the new one soon. And that's the repository for you to put scholarly articles in an open access repository that will be accessible to the world. We would love for you to do that. We would love to come up and work with you. Speak at your department meetings, or other groups as you deem fit to learn more about the services in the libraries that we will be offering to you and over the course of the next year and beyond. So with that, I just want to again say thank you for being here and thank you Phil for the idea of Year of Open with us. You will be seeing a lot more around this topic over the next year and now I'd like to call Phil to introduce our keynote speaker today. Thank you. >>Well, I'll be brief because I'm very eager as you are to hear more from David. I'll simply say if you wanna read a lot about his bio you can go to davidwiley.com and you'll see that he's been engaged in this area for several decade plus. And currently acts and serves as the CEO and Chief Academic Officer, Co-founder and as Chief Academic Officer of Lumen Learning which is a company that's dedicated towards increasing student success through the effective use of open educational resources and open textbooks. But I just wanted to point, to use one anecdote 'cause I ran into David at a conference that he sponsored back at BYU. Which I have to say I was bereft of the fact that I got up in the morning and failed to get my cup of coffee when I came to campus and if you know BYU you know that would be a tough thing to find. But I knew there was something that was worth following with respect to this guy when he published a graphic that showed the inverse relationship between learning objects, at the time was a big deal, between learning objects and their re-usability and the context of those learning objects. And there was this lovely inverse relationship that if something was built to be really valuable in a particular context, its re-usability was next to nil. And vice versa, if something was built that was particularly generic and not particularly useful for anything, then it would be used everywhere. And I thought well someone would have put that out on a graph in a paper and then make a very salient argument about what we need to think about with respect to open educational resources is someone I certainly wanna follow and I have ever since, and it's my great pleasure to introduce David Wiley to give us a talk today on the future is open and it's now. (audience applauds) >> Thank you, and thanks for the invitation. Any time Phil calls me to go anywhere I wanna go, so this is fun, thanks for, and Phil recommended this title and I think it's a great title particularly for the work you're trying to do here between the joint work between the library and between Phil's unit. I'll say of course that the talk, all the slides of course are open educational resources. You can take them and reuse them internally for whatever purposes you might want. Maybe to burn to stay warm during winter or something like that. Open UT is the hashtag, if you're tweeting my twitter handle is @opencontent. Just to clarify I'll wear a couple of hats during the talk. And I'll try to be specific about when I have which hat on but you know my primary full-time hat, let's see if we can make that stop happening here. My full-time hat is in my chief academic officer role at Lumen but I do still have two other hats that I wear. Until just two years ago I was tenured factory in the graduate program on educational technology at BYU, where I'm still an adjunct and still have a research group there that I run. So I'll make a few comments with that hat on and I'm also the education fellow at Creative Commons. So when I talk about the Creative Commons piece of that, of this work I'll put that hat on. If I move around, do you have trouble hearing me? >> No. >> In the back, are you sure? If I use my teacher voice I'll be okay? Okay. So I wanna start out by making an argument about education. I wanna try to connect education broadly to as Lorraine said the open agenda. I think that's a good phrase, I wanna try to connect these two things together by suggesting to you that education is a very particular kind of sharing. And if you think about what we do as educators, we spend most of our time trying to share the things that we know, the things that we know how to do, the things that we're capable of doing or excited about doing with students. We're trying to share what we know. We're trying to share, hopefully we're doing a good job of sharing feedback on work that they produce that when they hand in to use that we look through that and a lot of that feedback tends to be critical but sometimes hopefully, we're also sharing encouragement with students and trying to persuade them that we believe in them and that they're capable and they can do it. We're sharing the passion that we have for our discipline, trying to persuade people that they should want to be economists or they should want to be physicists or they should want to be educators like we are. And when it's at its best, education is really about sharing part of yourself. So not talking about sitting through faculty meeting. Not talking about hunting for parking spots. Not talking about the rat race of pulling your tenure binder together and stuff like that. But the actual educative acts that we engage in, all of them are acts of sharing. And I'd go so far as to say that if there's not any sharing happening there might be a lot of administrative going on but there's no education happening if there's not sharing happening. So I wanna try to connect those two ideas in your mind. Several years ago, there was this technological advancement called the internet that was invented. You've probably heard of it. But the thing I think that's interesting about the internet is if you look at, you know the internet gets compared to the press a lot of the time and I think that's a good example. Because what the internet has done, if you think about say in the library context, if you think about books, 'cause really the printing press decreased the cost of making a new copy of a book by orders of magnitude. And when they're smaller they're easier to distribute and to move around. And you know the internet did the same kind of thing. Has made it today that you know 300, 500, 800 page book as an epub costs essentially nothing to copy and essentially nothing to duplicate. And so that's pretty interesting. When it costs nothing to make a copy and it costs nothing to move that copy from here to there, then that gives you much greater capacity to share than we've had in the past. When in order to share a copy of a book I had to go cut down a tree and do something else and print it and ship it and warehouse it and get it to you, then it's harder for me to want to share copies of books with a thousand of my closest friends. But when I can do it for essentially no cost then I have this great capacity to share. And in as much as education is about sharing, then the internet gives us this really unprecedented capacity for education as well, except when it doesn't. So let me tell a brief fairy tale. Well, maybe not fairy tale, maybe morality tale. Maybe it's a parable, I don't know, you can tell me what it is later. But it's a story about this beautiful land in which people lived and worked and they were all happy and had a good time with the things that they were doing. And one day this young man invented this new piece of technology. There's still four chairs in the front so you can come in through this door. Invented this piece of technology to make it easier for people to move around the kingdom. And he called it the automobile. And it revolutionized things in a lot of ways but it had kind of an impact on the kingdom. And people looked at the impact it had on the kingdom, they were kind of unhappy about that and they said we really have to kind of get this under control because the automobile's gonna be the death of our lovely kingdom so they said, let's take some of these places that are already torn up, let's just completely tear them up and we'll build something that we'll call a road and the trade-off will be we'll tear up small parts of the kingdom to make these roads but the deal is whatever it is, whether it's a truck or a car or a this or a that, any big thing with a motor that's gonna take you from here to there whenever you're gonna call it has to stay on the road and that's the deal that we'll make. And that seemed like a great deal and they built roads and people wandered out and it was wonderful. And a few decades later, a young woman had an idea for another invention which she called the airplane. And the airplane had this really revolutionary potential. And she attracted some money and started building them. You know, built a facility and started producing airplanes. And the day before her inaugural flight, the police showed up at her house and told her, "We understand about the airplane, it's very cool, "we're very excited for what you're doing, "we just want you to remember that the law says "all the big things with motors that move you "from here to there must stay on the road, okay? "So enjoy driving your airplane on the road, "but if you take it off the road, "you'll be breaking the law "'cause that's what the law says." So what's the moral of this story? The moral of the story is that long before the internet was a glean in an engineer's eye, there was this other old antiquated law called copyright. And if we go back to our table, you know, when you think about the four groups of activity that copyright regulates, two of them are actually producing copies and distributing those copies. So that in some kind of really interesting way, the primary technical capabilities of the internet, everything that the internet allows you to do is actually regulated or prohibited by copyright. So that puts us in an interesting position. How to get the plane off the road? How to get the plane into the air without breaking the law? So I wanna talk about open educational resources in this context and start with some definitions, you know, so which open? If I ask you to define open, there would probably be as many definition as there are people in the room so I wanna tell you when I say open what I mean. I think there's a common misconception that open means free. And I say I think that's a misconception because actually if you think about the entire internet if you think about when you go to CNN or you go to National Geographic, or you go to BBC, or you go to wherever you go, the entire internet is already free. You don't have to pay to read the stories at CNN, or pay to watch videos on YouTube. So free is actually kind of uninteresting. Free is assumed, free is the starting point. But if free is the starting point, then what is open? I wanna suggest to you that open is free plus permissions. And by permissions I mean copyright permissions specifically so I would give you this two part definition of open that for materials to be open, you need to have free and unfettered access to them. You don't have to pay, you don't have to give out your zip code, or your email address or create an account. You can just click on the link and go to them and that with those free materials you are granted perpetual and irrevocable set of copyright permissions in a strong legal sense. Yes? >> [Woman] So if you say unfettered, would you call cookies fettering? >> No. >> Okay. >> But that's an interesting conversation that we could have after. Yeah, if you've used a website hosted in the EU in the last year, you're getting a lot of cookie information in your face, but... The particular sets of copyright permissions and I should have said, sorry, thank you for raising your hand, I should have said if at some point you violently disagree or violently agree or something, please feel free to interrupt me. Otherwise, being a professor, I will just speak unprovoked until the bell rings and then we'll all be done. Okay, so free and unfettered plus permissions. So what permissions? We call these the five R permissions. So retain, reuse, revise, remix and redistribute. So I need to be able to make and own a copy of whatever it is I'm talking about. Not streaming access to a movie like on Netflix. I need to be able to download, post, own and control my own copy forever. I need to be able to reuse that copy in a wide range of ways including opening it up and making some changes to it. Maybe the video was done in English originally and I need it in Spanish. I need the right to dub over top of that. Maybe there's a big example about the mountains of Utah in there that I don't like and I wanna be able to take that photo and the accompanying paragraph out and put a new photo and a new paragraph in. You know, maybe I've got three or four OER that I wanna combine in some way to create a mashup and whatever I've done, whether it's the verbatim copy that I've made in the first place, or my revised copy or my mashup copy, I need to have permission to distribute that to others. A show of hands familiar already with Creative Commons. Awesome, okay. So not much more to say there than except that Creative Commons provides you with a very, very very easy to use and of course free set of licenses that you can apply to the videos that you create or chapters that you write or lecture notes that you create or whatever it is. You can put this Creative Commons license on your material and that tells everyone yes, this is free and no, don't call me, don't email me, don't ask if it's okay if you do X, Y and Z because I've already gone out of my way to put the Creative Commons license on here to tell you you have permission to do these things so just go knock yourself out and do them. The point of the Creative Commons license is is to take the friction out of that process so you don't have to contact me, you don't have to ask my permission because ahead of time I have already granted you the permission by putting your Creative Commons license on there. Just a quick point to make that of the five Rs retain is really the fundamental R. I have to be able to have my own copy. If I don't have my own copy, it's pretty hard to revise and remix. So the five Rs. Contrast, if you will, and I wish I could take credit for this word, it's not my word but it's one of my favorite words ever. Contrast open with faux-pen or fake open in which you have free access which is probably gated in some way, but you're still working in an all rights reserved context or maybe a context where the terms I've used put even more constraints on you than just typical copyright would. There was a long time... What's the right word to say that? When you look at affordability initiatives on campus, a lot of times the first place in affordability initiative will be to the library because there's a lot of material in the library that's already been paid for that students can use for free. But what you'll see is that, you know, with commercial resources, they're expensive for students and faculty and students get no permissions with them. Library-based resources are better in that they're free, but you typically still have very restrictive all rights reserved copyright plus crazy terms of use but open educational resources kind of get you all the way there in terms of being free to access and use and giving you those broader permissions. Two parts to the definition of open educational resources. Educational resources that you have free and unfettered access to that give you these perpetual and irrevocable permissions. So at the risk of, I mean, Phil wants me to be slightly provocative. I don't know how good a job I'm going to do. Maybe one of the things I'll suggest to you is that if you're teaching a class where you're using the internet at all in your classroom, if you're having students go to Canvas to grab materials from there, use that. When you make the choice to adopt traditionally copyrighted materials and you put those on the internet, you're making the choice to drive your airplane on the road. But when you make the choice to instead of adopting traditionally copyrighted materials to adopt openly licensed materials, then you're actually able to get the plane into the air because at that point everything that the internet technically allows you to do, technically gives you the capability to do, the open licenses of OER give you permission to do so that you can actually leverage the full technical capability of the internet in the service of teaching and learning which you can do in the context of traditionally copyrighted materials. So I wanna talk about adopting OER and you know, just again to be clear on definitions. When I say adapting OER, what do I mean? In the next several slides, I mean taking the syllabus for whatever course you're teaching and where it says required textbook or required materials, you delete whatever was there before and you replace it with open educational resources, so we're not using OER in a supplemental way where we're saying hey, go buy this $130 textbook and here are these free supplements to it. We're saying actually to take this class, there is no cost, we're not even asking to you spend money on a textbook because we've assigned OER in place of the textbook we were using before. And particularly I wanna talk about kind of high impact adoption on the student side. I'll talk from the student perspective first and then from the faculty perspective and then even a little from the institutional perspective just to try to run through some of the impact that adopting OER can have. So let's talk about adopting OER and improving student success. Now, grumpy cat fans in the room? Yes. So grumpy cat says you get what you pay for. And this is the... I think this is the default assumption, right? If I buy a $100 stereo or a $1,000 stereo, the $1,000 stereo is a nicer stereo. If I buy a $10,000 car or I buy a $100,000 Tesla, Tesla is a nicer car. There's this up and to the right assumed relationship between cost and quality that says the more I pay the better it is, you get what you pay for. Faculty, in particular, will be very quick to admit yes, I believe there are materials online which claim to be educational in nature and which are openly licensed, but... If they're free, they must be garbage 'cause you get what you pay for. Nobody would share under a free license, under an open license anything that was worth it sold. So I wanna show results of a couple research studies with you that kind of switching hats now, my group and I'll share a URL to these in a minute. This is not in print yet but it's been accepted in Journal of Computing in higher ed and you can get it digitally but it's not in print yet. But this is a study looking at the impacts of open textbook adoption on learning outcomes for post secondary students, as the title says. Very descriptive title. So this is looking at about 15,000 students across 10 institutions in 50 different undergraduate courses about 130 teachers teaching those different courses. And I think I said 10 institutions already. And this is not the tightest research designed in the history of mankind but it's pretty good, right? So it's not an RCT but it is quasi-experimental with propensity score matching to make sure that we actually are comparing apple students to apple students across treatment and control. Looking at post test, using post test as our measure and looking at completion rates, C or better rates among students and also the number of credits students enrolled in this term and next term because in some early survey work when we asked students what do you do with the money that you save from the faculty assigned OER instead of a commercial textbook, take said I take more classes. I said okay, well. There's probably a way to figure that out. Let's go and see if that's true. In this case across all these different institutions it's a very small set of common data, we're able to get about students in terms of just age, gender and race but nothing more like Pell eligibility or interesting variables like that. But this is a table from the paper and what you'll see here in these three columns are completion, you know, the number of students actually making it out the door on the last day of class, this is C minus or better as a binary variable, either they received a C minus or better or they didn't and this is course grade as a continuous variable kind of from zero to four. And the blue squares are cases where students whose faculty assigned them OER outperformed students whose faculty assigned them commercial textbooks. The two red squares are where the commercial textbook students outperformed the open textbook students and the NSs are all no significant difference. And I apologize if I bounce back and forth sometimes between OER and open textbook because to me an open textbook is just a bunch of OER that you've pulled together in such a manner that you use it to replace the textbook that you were using before. So I think about an open textbook as just being a collection of OER. But anyways, you can see only in one case that the students whose faculty assigned them commercial textbooks do better. It's frequently no significant difference but in several cases, students who were using OER actually outperformed both in terms of how many made it through and completed and the grade that they received in the course. And in some ways even more interesting is looking at the number of credits that were taken by the students whose faculty assigned them OER versus students whose faculty assigned them commercial textbooks. It's about a credit and a half more that students who were assigned OER taking per term than students who were assigned commercial textbooks so that's pretty interesting. Now, in a separate paper that we supported this work but we didn't participate in the research, we're not named on the article. This is at Mercy College in New York which is a four-year private looking at their college algebra success rates. And so you can see here, there's a pretty distinct pattern between fall and spring or fall and winter, I don't know what you call it here? It's just summer and summer and summer and summer in Austin, right? (audience laughing) Yeah, okay. 'Cause, you know, you get these kids in college algebra and many of them pass and the ones who don't pass show up again in spring to try again. So this is a very different kind of group of people than this group is, but you can see, you know, from fall to spring, and the years are commonly colored, you can see that typically spring is a much lower success rate than fall. But what you'll also see here is that starting in fall of 2011 when they were still using a commercial bundle of a textbook and an online math practice system that generates problems algorithmically and you answer them and it gives you feedback about whether you got them right or not and you can get all the practice that you want. You know, so this was a very expensive bundle they were using and then in spring term of that year, we switched them over and helped them move to open textbooks and an open source online math practice system. We did that in a couple of sections and it worked really well in a couple of sections so then in the fall of the next year they flipped all 28 sections of this course to OER. And you know, you can see what happens here and it's even more dramatic, of course, on this other side because they had more headroom to game. But among that spring group you're look at 48% pass rate before and 77% pass rate after, so it's pretty significant improvement. This was published in EduCals Review. And now the paper with the world's worst name. I will buy a milkshake for anyone who can help me come up with a better name for this framework. It's really a way of looking at return on investment in terms of student success for the money that we ask them to spend on educational materials, okay? So again, if I was asking you to buy a more expensive stereo I'd expect to get better sound out of it. But just in terms of, you know, quick framework, again, here is our grumpy cat kind of up and to the right, this is our default hypothesis, but the more money I'm asking you to spend on these materials, they must be better materials, so if I'm asking you to spend more money, more of you should be completing with a C or better. You know, so if I spend a little bit, learn a little bit it's one thing, if I spend a lot and learn a lot, that's another thing. If I spend a lot but don't do well, that makes me angry and if I don't spend very much but still have great learning outcomes, then that's really exciting to me. So, you know, taking the past two examples so the Mercy College example started out at about $180 with a 48% pass rate and three years later they're doing a $5 fee on that course to support hosting and maintenance of that online open math practice system and some other tech and the pass rate moved up to 77%. As a single example. If you take the data from the first article that I showed and go back to the bookstores across all those 10 different institutions and told that numbers for what the textbooks in those courses cost, what students were asked to pay, then you get a distribution, again, of students whose faculty assigned them OER, and students whose faculty assigned them commercial textbooks and you'll see a couple of things there that are interesting. I think the first is that the lowest low and the highest high are further out for OER which I think says that there's a greater variability and effectiveness among OER which I think we'd expect. You'd expect more kind of homogenization in the commercial materials because they all run through very similar production processes. But this average is further to the right than this average. And actually you might not be surprised to know that if you take these OER out and plot the best fitting line though that cloud of dots at the top, it does not go up and to the right, it goes down and to the right. So openedgroup.org is the website of our research group at BYU. And my colleague John Hilton who's another faculty member at BYU who kind of owns this project we call the review project. And so the review project gathers up all the empirical evidence on what are the impacts of OER, what are the impacts of OER adoption by faculty on students. And just to give you a quick summary of this. This tells you how new the field is. Right now, there are 11 peer reviewed published studies in the review, just about 50,000 students in those 11 studies and the looking at outcomes like C or better and completion and things like that. 93% of those students are seeing the same or better outcomes than their peers who are assigned commercial textbooks as opposed to OER. Yes! >> Your examples so far have all been in sort of math and business every time it tangles for humanities or maybe-- >> Yes, so that study that had the 50 undergraduate courses in is all across gen ed. If you look at... There's a lot of math that, oh, okay, so there's some bio, business, English, math and psych in this study. And you'll see that most of that effort, most of the focus in OER to date has been in the general education kind of band, introductory kind of classes. Because the foundations that have historically funded OER have said if we're gonna spend $100,000 on this, how can we do that in a way that impact the most students possible? So it ends up being the highest enrolling courses, the gateway courses, things like that. But there is, you know, art appreciation, history of music, things like that that do exist. Let's see where did we leave off here, I think we left off there. In peer reviewed studies of perceptions of OER quality, it's about evenly split between professors and students that have responded to these surveys. When you ask them and it's worded differently for students than it is for faculty, but when you ask them questions about the quality of the OER, you know, it breaks down in essentially this way. If you ask a student who's just finished a course using OER and you say and now imagine you're doing a survey at the end of term, imagine next semester there's a course you're required to take in order to graduate, there are multiple sections of it offered, they're all at equally desirable times. One of them uses normal textbooks like your other classes do and one of the uses all online material like we used in this class, OER, which course would you register for? So those kinds of indicators of their perceptions of quality and what they would like to use. So maybe this is the most interesting slide of the whole presentation. This is super fun to play around with. I'm sorry, the URL's at the bottom, I should put it at the top, but it's impact.lumenlearning.com. And this is an interactive calculator where you can go and play with several variables to estimate what the impact of adopting OER might be. So it starts out with a set of institutional settings, we can move these sliders back and forth to make the kind of context match your institution or particularly the course that you're thinking about replacing commercial textbooks with OER. So what's the textbook currently cost? How many students are gonna use it? You know, are you gonna levee some kind of course fee in order to support whatever effort is being made to make that happen? What percentage of students have passed it historically? What's the cost for in-state credits and out-of-state credits, things like that. So you kind of set up how things work at your institution in that course and then there are these research-based settings where we've taken kind of the results of our work and use those as defaults here. So what will be the increase in the C or better rate among OER users? What will be the decrease and the drop rate among OER users? Things like that and again, they're all sliders so is you think our estimates are too conservative or too aggressive, you can play around with them. And then it aromatically kind of generates five or six graphs looking at kind of total text it would cost the students. This metric we call throughput which is really interesting. You know, the percentage with students who were there on day one who graduate with a C or better. Not the percent that graduate with a C or better after all the ones that are scared off, drop at the beginning of the term, like how many of them that started on day one. Here's that mad, glad, sad, rad whatever the order of that terrible name is graph here and there's more that. So you can play around with this and look to see including further down there's some financial stuff on the institutional side. What's the impact on revenue from the bookstore? And where do we make up that revenue somewhere else? Go ahead. >> I really like, those are some very clever tools, thank you for sharing that. I do have a couple of questions 'cause it seems like it could have a pretty big impact with a tool like this. Do you control it all for sort of how course grades are determined or the level of difficulty at exams in making inferences? Or things like, you know, people who maybe are more knowledgeable about educational practice using OER resources? And so there's been lots of variables at play-- >> There are a lot of variables at play, right? So frequently, teacher and OER adoption are perfectly confounded. >> Right. >> Right? So really that you could theoretically have somebody teach with OER and teach two different sections differently but typically once somebody starts using OER, they're all in so those tend to be interrupted time series studies where you compare the teacher to herself, you know, previously so you try to control it for teacher effect that way. Depending on... I don't think I have it, no, okay, I don't have it in this deck. Probably the most rigorous of these studies we've done was actually in high school with open textbook adoption in place of commercial textbooks adoption. The state tracks a lot more data and the state also administers a state standardized test at the end of the year so, you know, in that study, it's got the same kind of setup as the first one I described here, but instead of three variables we're controlling for, it's 17 variable we're controlling for including whether they're English language learners or not, if they're special ed designation or not, reduced lunch or not. What did they score? What did the students score on the previous years? Science exams. >> It's just random assignment >> It's not random assignment because this was a district-wide move to OER so really it's interrupted time series. No, it's the district that did compared to the same district the year before when they hadn't. So that's how you're controlling for teacher that way and then also controlling for teacher for that individuals teacher's contribution by using the average score on the state standardized science exam that students in her room received previously which is a ton of control in that because there's more data better data and what you'll see again across earth science bio and chemistry is either no significant difference or actually on the IRT scale scores under state standardized tests, the open textbooks students outperforming the commercial textbooks. >> Is that cited on your posted review? >> It's on the... It was in ed researcher, so if you go to ed researcher and it's in open access, of course, so if you Google ed researcher and Wiley and Utah open textbooks or something like that you'll find it. But I didn't put it in this one because I didn't imagine you guys would care about K-12. But because of the quality of the data are better, it's actually the tightest of all the-- >> Yes, get a look at K-12 >> of all the studies. Say it again. >> I understand your decision but it's also get into K-12 because they do have better data. >> Yeah, they do have way better data. Okay, so-- >> What's the role of taxpayers? >> Yeah, well, don't even. Okay, I'll start talking on my pulpet if we get onto the taxpayer part of that, maybe we can talk about that at the end. So at this point, hopefully, I've persuaded you to at least consider the possibility that when students use OER they might, they at least don't do worse than they do when faculty assign them commercial textbooks and it might be possible that they actually do better. If I can just have brought you along to there, I'll be really happy and feel like I did some good work. So if that's true, how does that happen? So I wanna talk about patterns of OER adoption or types of OER adoption for just a second. And yes, it was pointed out to me recently that I have some kind of fetish for frameworks where all the words start with R-E, I understand. (audience laughs) I'm thinking about trying to rename this one. But basically, you know, when you adopt OER, there are kind of three ways of going about that. One is a kind of naive substitution strategy where I was using this textbook and I'm gonna go grab this open textbook and I'm just gonna use it in STEM. A second is realign and I'll talk a bit about that in a second, I'm gonna give examples of all of these. And then rethink is really asking questions about how open interacts with pedagogy, which I think are some of the most interesting questions for us as faculty. So, you know, I don't want to say that replace is bad. I don't wanna speak bad about replace. But when you think about the whole universe of OER that are out there, and there are a lot, Creative Commons collaborates with Yahoo being in Google on a count of all the material on the public-facing web that use a Creative Commons license. And in 2014 that count was 883 million items, so by now we're probably about to a billion items that are out there and available under an open license. And so replace strategy absolutely saves students a ton of money because you're asking them to use OER instead of asking them to use commercial textbooks. But if you go there and kind of stop, I think about it as like going to the buffet and then saying I really like beef and broccoli and just piling beef and broccoli onto your plate and then going sitting down and not doing like not getting any ice cream, not doing anything else. Like that's acceptable to do, but there's some kind of sense that you're missing out on kind of broader opportunities. Realign and this is where Lumen does most of its work with faculty and with institutions is it really starts with the process of going back to your syllabus and saying let's just all pretend for a minute that no textbooks exist, no textbook has ever existed, just look at your syllabus and see what are the six, eight, 12, 18 learning objectives or learning outcomes, I don't wanna get in the middle of a holy war, what do you call them here? >> Outcomes. >> Outcomes, okay. What are the outcomes on your syllabus. And then for each individual outlets make that outcome list into the table of contents for what we're about to build and for each outcome let's go find two or three OER that you think with your students and the way that you like to teach will support that kind of learning that you wanna see happen. And just build up a textbook replacement by aligning ROE with the actual outcomes that you claim to care about at least according to your syllabus. Coming back to K-12, the Utah work, the work we're doing in Utah. So we have a... It's been going on for seven years now. Open textbook science, open textbook initiative in Utah, we do math, too, but in the science context we get the teachers together for two days a year, we get them together in January for two days and they look, we bring about 120 together, kind of 10 to 12 per a discipline 'cause we're graders three through 12 right now. And the people in the biology group sit own and talk about what worked well for you last year? What didn't seem to work? Like what part of this didn't seem like it was supporting what we wanted to support? And what new OER have come available in the last year, like what was new in plus biology that we could out or what did NASA release about life on Mars or whatever it is. And every year they pull together a new edition of the textbook. And the thing that's really fun about it is you would think that in a state, in K-12 or a textbook review has to happen that you could publish new editions of a textbook every year but when that book goes up to the state board, and by the way, it's the state board that runs this program, but when the book goes to the unit in the state board that does review an alignment review, you open up to the table of contents and says Utah state science standards outcome one page one, outcome 1.1, page four, outcome 1.3, like literally the standards are the table of contents. And so that review that normally takes hours and hours for a traditional book takes about 15 minutes for one of these books because it's built around the standards. And you can do the same thing in higher ed with your learning outcomes. I like to think about it this realign process kind of as furnishing a room and so here's a room, it's already furnished, that's great, but maybe I hate this table and I hate that painting and I wanna pull some pieces, like I can take it as a starting place and I can pull out the things that I don't like and replace it with other things that I do like and there's a whole catalog like of tables. And there's a catalog of paintings and a catalog of couches, and I can just go through and pick the one that I like and build something that I think my students will actually really like. And that sounds like a lot of effort, but even in the higher ed context we do that in a two day workshop and you come out with time to spare typically on the other side of that process. But rethink I think is the most fun of all of these. So the question really here is if I can assume that all the materials I'm gonna be using in my course are openly licensed, what can I now do that I couldn't do before? What's open to me as an educator that was not open to me before? So I tend to go to assessments first, I mean, I'm a old instructional designer and tech person. And so I like to make this distinction between first what I call disposable assignments, these are assignments like you know, two page response papers, so I can make sure that you read the chapter before you came to class and you as a student look at writing those pages and absolutely hate it and you as a faculty member look at grading 50 of these two pages and you absolutely hate it. And I call them disposable assignments because there's an implicit agreement between you and the student that as soon as you've marked it and handed it back to them, they're gonna throw it away and that's the sum total of the value that it added to the world. So when you think about assignments based in OER I think we can do something that's more interesting, something I call renewable assignments. So two examples really quick. Are we out of time yet? How are we doing? We're okay. I wanna make sure there's plenty time to talk. So this is my kung-fu assignment and this is one of my favorite assignments to give. The kung-fun assignment is to go back and find either go find some openly licensed video or go find some old public domain video and re-dab the audio over top of it in a way that makes that video teach a different lesson than the video was originally meant to teach. And I call it the kung-fun assignment because you know, in the kung-fu movies the mouths are never actually matched up with the audio that's happening. So you just kind of own that, so I understand that your audio is not gonna sync up with the person's mouth, that's fine. But so this is in a social media class. This would historically have been, you know, write a two page paper about blogs and wikis, describe how they're similar, describe how they're different, demonstrate to me that you actually understand what blogs are and what wikis are and what the difference between them is. But so this is a kung-fu assignment goes to a small group of students to get together and go out and find this footage and this is a Kennedy and Nixon debate and it's only about four and half minutes long, if there's more time, I'll play it, but if you search for bogs versus wikis in Google, you'll find this right near the top of the YouTube results. And this video is just so awesome in so many ways. First, because of the bad kind of attempt that actually doing Nixon's voice and going Kennedy's voice that the students do. But more interestingly, like they chose this video because so Nixon is on the pro blog side and Kennedy is on the pro wiki side and for Nixon's part in the debate, he talks about how, you know, wikis are a problem 'cause anybody can edit anything, anybody can do anything, he prefers blogs because the author and owner of the blog can control the information that's presented to the public. He's just created a blog called the Watergate blog he says where he's putting out the official narrative of here's what really happened and people are allowed to comment there, but you even have control over comments because you don't want internet predators, you know, kind of hoodwinking you and whatever. And then Kennedy, of course, is all about Kennedy's saying that he's about freedom of speech and freedom for the people and he'll guarantee that every citizen has access to every wiki and anyway, so Nixon does this whole thing about control and access and then Kennedy closes the debate by saying, now think really hard what, think about wikis and think about Kennedy and what is the ultimate most perfect thing that Kennedy could ever say about wikis ever? He says, "Ask not what your wiki can do for you, "but what you can do for your wiki." Now, if you've ever been to a wiki that somebody set up and said oh, we've got this great project, we're gonna make wiki and everybody's gonna come contribute and 99 times out of 100, nobody comes, nobody contributes anything, it really is about what you can do for your wiki and a wiki that's healthy and vibrant and interesting. So, you know, this was a homework assignment that otherwise would have been three separate two page essays about how blogs and wikis are different from each other, but instead got published on YouTube and now has been viewed over 52,000 times and is used in classrooms I hear people tell me that they use this video in their class. And this is just an assignment that they would have done and would have been thrown away but they did understanding it was gonna go online that people could see it, that it could be used by other people and they're like wow, there's... We actually need to do a little better job with this because real humans are actually gonna see it like I'm not a real human, right? Real humans are actually gonna see it and it just turns out to be a terrific piece of work. Another example here. This is project management for instructional designers which is a course I used to teach when I was still full-time. There is no textbook in the world called project management for instructional designers, right? I think this course has offered at like 15 or 20 colleges total in the country, there is no book for it and I will fully confess. I was complaining to a friend of mine one day about there's no textbook for this, I found this open textbook that I was really excited about but it turns out it's very business-centric, all the examples are about getting so many tons of rebar and concrete to Singapore and they're coming from here and there and you're managing the logistics of it and something else and it just wasn't what I needed and my fried, bless his heart, got in, we're in the car, complaining about, driving somewhere, complaining about, he basically backhanded me 'til I drove off the road and he's like, "You moron, it's an open textbook, like fix it! "If it's not what you wanted to be, fix it!" And so then I started staring down the idea of editing this entire textbook to make it from a business project management book into a project management book for instructional designers, I'm way too lazy to do that, so it's kind of caught between a rock and a hard place and then I realized what is the solution here. Assign the students to do the work, right? So I've taught this course a couple of times in that format and here is a summary of some of the things I've changed and if you go to this page and scroll further down you'll see more of the changes, but in the first semester, students went in and they found all those, you know, on textbooks there's a little yellow or blue box kind of pullout for having a little case study inside them so there's one group of students said, "We're gonna fix all the case studies in the book." So in week one, in class when we talked about week one, we talked about the topic of whatever week one is. Introduction to project management. Then they went back and for the homework they got together and they rewrote drafts of all the case studies that were in chapter one and that's what they turned in and that's what I graded by editing it and sending it back to them and we kind of went through revision cycle on that. And over the course of the term, that team rewrote all of the case studies in the book. And if you think about how well you have to understand a principle to be able to pass an end of chapter quiz versus to be able to rewrite a new case study that actually exemplifies the principles that are involved, it's very good work. Another group of students did a series of video case studies where they found three project managers actually in different units on campus, project managers who manage instructional design projects. And then for each of the chapter topics asked them to talk for three to five minutes on video about a story about, you know, tell us a time when interpersonal problems blew up a project that you worked on and the importance of people and personal relations on a project. So now at the beginning of each of these chapters, there are these three video cases that are right at the beginning of each chapter. Another group went through... Everybody hates this class, nobody wants to take it but it's required, so I start this class by saying, "If you earn the PMP certification, the project manager professional certification, average salary for a person with this certification is $106,000 a year. Then suddenly all the students care about it and at least for a half an hour are interested in project management 'til it wears off. (audience laugh) So, you know, one time when I said that the question became well, how like where does that exam happen and how much does it cost to take and how do I prepare for it? And I didn't actually know the answer to any of those questions, I just needed the propaganda about how much money you made if you had this certification so we started digging around and found that there are a couple of, but it's like the GRE or one of these exams. There are prep books for them, but they're giant and they cost like $120, $130 a piece. So this one group of students got, we got some from the library and I acquired a few from Amazon but we got all these study books together for the exam and then one group of students went through and restructured the book and wrote new introductions to each chapter saying, "This chapter is covered "by these sections of the test, there are typically "this many question on this subject, blah, blah, blah." And turned this book into a free book that you can use to study for the certification now as opposed to having to go buy $120, $130 book to study for the certification. So, you know, it's not end of chapter quizzes, it's not the kind of assessment that we're typically used to but the students get way more engaged in doing this work and you, as a faculty member, as you're grading the work, you feel like you're doing something that's more valuable to the universe than maybe what you were doing before. And the most fun thing of all is after I did this the first time around, each time I taught it after that I told all the students on the first day of class congratulations, you all just received an A in this course. Aren't you happy now? Now that you have your A, I want you to understand that you're actually now the rest of the semester you're just playing for reputation among your peers because we're gonna break up into these project teams and every week you're gonna stand up and show your progress in front of god and everybody in the class and everybody's gonna see the kind of work that you're doing. And so at the end of the semester now I ask students two questions. Was your grade ever in doubt in this class? They say no, you told us we were gonna get an A, my grade was never in doubt. Then I say was there any class you took this semester that you worked harder or spent more time on? And you see the gears just kind of grind to halt and they're like they figure out that they've been hoodwinked somehow, like why did I waste all this effort on this no, I worked harder on this class than any other class but I knew I had an A, why did I... They just almost break down, you know, you can see that they can't figure out how they got tricked in that way. But they invest in this kind of work in a different way, even when they know that their grades aren't... I wouldn't do that with every group, but with this group, I could do it. So, you know, the common thread between these two pieces here, this is all public domain video and this openly licensed textbook are that you can only give these kinds of assignments because you have those five R permissions. Without those five R permissions, you can't do this kind of stuff. And so I think there is a very strong argument to be made for faculty from a pedagogical point of view that choosing OER over commercial textbook actually expands your academic freedom in terms of what you can pedagogically in your class. Choosing a commercial textbook prohibits you from doing this. Choosing OER opens it up to you. Just very quickly 'cause I feel like I've gone on too long. We don't have a good study that actually demonstrates this yet, but looking at the studies that we've done and the studies in the review project, I feel pretty confident in this hypothesis that just swapping out an old textbook, an old commercial textbook for an open one will result in very small changes in learning, if any. There will probably be no significant difference. Realign will actually get you some moderate changes and it will get students who you thought weren't paying any attention at all will say things unprompted on their end of year smiley sheets that they fill out at the end of the course. They'll write comments like the things that I've read for this class were actually related to the things the professor talked about. Who had thought? Who had thought? Okay, so that's high impact adoption in terms of student success, just two quick slides on improving affordability, we don't need to beat this point home anymore. There are a thousand versions of this graph. This one's from the Economics showing textbook price is up 1,500% over the last 40 years compared to consumer prices for other kinds of things. My favorite way to think about textbook price is to put it in this context. So to get one month of streaming access to basically every movie and TV show and every song ever recorded cost me about $18 a month or for one month of streaming access to a single biology textbook I'll pay 20. That's the way that students get 20 bucks to spend, what should I do with it this month? That's kind of the math that they would do. And I think unfortunately the... You know, our approach which has been totally good-hearted and wanting to do the right thing, trying to find any way to make it less expensive for students when you think about all of our kind of textbook affordability strategies, they all end up being what you might call disappearing ink kind od strategy. Because at the end of the semester, either we're buying the book back, so the students don't have access to it anymore, they're just renting it so they don't have access to it anymore at the end of term. They do a subscription where they get a passcode that works for a 180 days and then they don't have access to it anymore. It's a kind of ebook with DRM inside of it that deletes itself off your device at the end of the semester so you don't have access to it anymore. With all of our traditional affordability strategies, students end up not having access to the material anymore and it is not lost on them that on the one hand we say to them this bundle of courses is so important that we absolutely will not let you graduate unless you take them. And 'cause you'll never need them ever again in your life, sell your book back at the end of term. I mean, which is it? Is it so important you shouldn't be allowed to graduate without it? Or is it so meaningless that you'll never even need to refer back to it in the future? Because it can't be both. It can't be both. So doing it at scale. Doing it at scale was really interesting because now, what we've seen in the last two years is the emergence of this idea of an OER-based degree. And the OER-based degree is a degree program where a sufficient number of elective courses and all the required courses are flipped over to use OER so that a student can graduate from a degree program without ever being asked to buy a textbook. You know, the first of these was a Tidewater in Virginia. Decreased the cost to graduate by 25% in one year. So I met Daniel DeMark who's the provost there in December of 12 and we launched this in the... Oh, oh, animation. We launched this program in the fall of 13. And if you look at the in-state tuition rate for 61 credits, this is a two-year degree program in business administration and you put textbook cost on top of that, for the students who entered this program in the fall of 13 it was literally 25% cheaper to graduate than it was for students who entered in fall of 12. That work went so well that the Virginia community college system got together and said we need at least one of these OER-based degrees at all 23 community colleges in the not state, in the Commonwealth. And so other schools started doing this kind of work, as well. So northern Virginia, which is very large now has their Open General Education Project where students can either complete the gen ed requirements for a two-year degree, or a full two-year degree which is the gen ed transfer degree, so do all your gen eds at the community college and then transfer up. In the last two years, 8,500 students have save $1.2 million and their internal data, this isn't our data, their internal data says the overall student success rate is 9% higher for those students that are using OER. You know, one of the really interesting things that we did in Virginia is that all 23 schools in the state of Virginia use Blackboard. Let's all have a moment of silence for them. Sorry, are there Blackboard fans? Okay, anyway, not only do they all use Blackboard but they all use a commonly hosted Blackboard that's run by their central office. And so we work with the central office to put this OER courses tab into Blackboard. So now regardless of whether you're at Tidewater or Germana or Sergeant Reynolds, whichever college you're at in Virginia, when you log into Blackboard, you see this OER courses tab at the top there and if you click on that, it'll take you over and show you a list of courses for which OER are available to adopt instead of commercial textbooks. And you can jump out to a public facing link with just the materials and see what the materials look like or you can jump in to see the materials already built out as a textbook replacement inside Blackboard and it's about three minutes from the time you see this screen to the time that your own Blackboard course shelf for this course is populated with all the course material so that you can use this instead of your commercial textbook. I said I'd say something about institutions, so I'm gonna say two things about the institutional impact of the OER adoption really quickly. This also isn't in print yet, but it's in Ed Policy Analysis Archives coming up later this year in a special issue on OER. So when students drop a course before the ed drop deadline, it does multiple things, but at least two, both of which very bad ones. It slows down their graduation 'cause they have to sit on the bench for another semester and wait for it to come back around so they can try to take that course again. But from the institutional perspective, it actually costs us dollars because we refund tuition if they dropped for the deadline, right? So what we did was with Tidewater, we look at data from their first year of what they called the Z degree, their zero textbook cost degree, their degree that uses OER. And we looked at the difference in drop rate between the kids in these OER sections and the kids in the other commercial textbook using sections of the same courses in the same semesters, fall and winter. And if you do the math on this, there'd be 182 students who won't drop who would have dropped times the percentage that are in-state versus out-of-state and the figure whatever the tuition rates are. It ends up that Tidewater will not refund a little over $100,000 in tuition because they made this move to this OER-based degree program. Oh, in summary, I'm almost done. If you also think back to that slide from the very first study I talked about that had the number of credits that students stake where the students were assigned OER take a credit and a half more than the students who were assigned commercial textbooks. That is a credit and a half more net new tuition revenue from students that you didn't have before either. So in summary, very quickly, water open educational resources, they are resources that are free and that grant you permissions to do interesting things with them. It can be done at scale. As we're seeing now, there are... There's probably about 30 institutions in the country that either have already completed this process or in the process of transitioning to degrees where you can graduate without ever being asked to buy a textbook. Student success can be improved when you do it this way and students can save a lot of money. Faculty have a broader range of things they can choose to do pedagogically. It also happens to be good financially for the institution, as well, so I would argue that this is a win-win-win kind of thing and we ought to be doing more of it. So having said that, I hope there's still some time to have some conversation. (audience applauding) >> This is the chance for you to ask questions. Are there any that you would like to refer and ask a question? >> Not all at once! >> So I was kind of having a hard time keeping up how this model kind of naturally go into the humanities especially in classrooms where we don't typically have textbooks. So a lot of humanities classes and maybe English 101, you have your rhetoric, I don't know, maybe have a great English speakers. But beyond that, you don't typically have a textbook and the things that we end up, I'm taking about, you know, just thinking about literary study the things that we tend to look at are inherently in the copyright and can't be shared by many for many reasons, right? Same in history, and you can there are concepts and perhaps you can achieve those outcomes by focusing on the things that are readily available so that you can do a little bit of that but there are things that we study in certain disciplines that are not freely available. >> Where the object of study isn't an idea, it's an artifact. >> Exactly. And that the part of what you're studying is a conversation around that object, as well, and you know that has worked historically and all of those things are still meant to be-- >> Under copyright, yeah. So that's a case where you would fall back to your library to say if I can't get... 'Cause ideally you get open access to everything you can get or excuse me, now I'm mixing my metaphors 'cause open access is a technical term in the library world. Ideally you'd uses OER for everything, there are instances when you cannot, when you can't, dropping back to like library resources are at least free to students even if they have permissions with them, so you'll package those together. You know, so for a... Oh, what is it? It's a music... I think it's just music appreciation, it's like the music listening class. The group of faculty we just finished working with to build a version of that, all of the listening is actually done by going and getting a free Spotify account and listening to recording on Spotify, right? There are some... There's this great Open Goldberg Project, there's so music where the recordings and everything is openly licensed, but for a lot of it it's not and if you wanna listen to Shostakovich, then you're gonna listen to a copyrighted recording somewhere, right? So it's just literally impossible to do in these cases where the object of study's an artifact, not an idea. But there's kind of a cascading, if not open, at least free, if not free, at least something else until you get down to the point where you're like I've done everything I possible can, you still have to go buy a $300 textbook, sorry. >> But it could actually include the link to the copyrighted material instead of the text and then just create your activities and license those as OER or pool activities that are OER. >> You can still openly license all the things that all the pedagogic stuff that you build around the artifact, right? I think that's the point that you're making. The discussion questions or whatever it is, you can create all of that and open that but if you're studying Harry Potter, sorry, that's the example that comes to mind, like there is no open equivalent of Harry Potter. Either you read Harry Potter or you don't read Harry Potter. So yeah, there are some cases where it's literally impossible to replace what you're looking at with OER. But there are lots of ways that you can be creative with kind of these successive fallbacks to still if you can't be open, you can at least be free. If you can't be free, you can at least be really affordable, you know, kind of back that way. >> I just had an example 'cause we're interested for open educational resources and so like so my professor made a lesson about memes and taught the students like the idea of illusions through memes and so that was an open resource, like they had to enclose text and then (background noise drowns out other sounds) yeah, you can use open text to teach the ideas, but then (background noise drowns out other sounds) >> And then your library and can talk to you about fair use for things aren't copyrighted that even the library doesn't necessarily have officially in collection, right? >> Question. >> Yeah. >> Hey, first of all, thanks for talking to us a bit more. It's kind of a two part question, okay? So the first part is in regards to like the digital divide and access to sort of technologies to read these things, so like you know in your presentation you sort of described these as like digital textbooks, or digitally accessible materials. And so obviously there are sort of the capital investments on the side of the student to be able to access those. If not, there's time investments, going to the library going somewhere where you can access that. In regards to that access, the second part of my question is is sort of durability of the title, right? Because like theoretically if you had a textbook and you don't sell it back, you could keep it forever in a way that a digital file frankly, the digital files from my undergraduate degree are long gone and they're long inaccessible. So it's possible that they're out there still, but that's a big theft if you're hoping to keep something. So can you kind of speak to those issues? >> Yeah. On the digital divide part, like what I didn't say about our K-12 work in Utah is that it's all print on demand. Nobody that's using OER in grades three through 12 in Utah is doing it digitally because... Everybody likes to claim that they're the lowest funded per a pupil in the country, Utah actually is the lowest funded per a pupil in the country and has to do with Mormons and big families and something else, but there are no devices in those classrooms for those kids. And so part of what we do in that project is we after the teachers do all that pulling together that I talked about, the two-day workshop, we get everything formatted for print and uploaded to CreateSpace which is the Amazon print on demand service. And if you Google USOE, Utah State Office of Education, usoescienceoer, you can find the page with the links to CreateSpace where the individual schools can go to buy books and the books range, delivered they range between 2.99 and 5.99 delivered. Again, when it can't be open because we don't have the infrastructure, we can at least fall back to, if we can't fall back to free, at least a very, very, very inexpensive, right? The digital divide issue is a real issue and there are smart people that work on that problem. I can only work on so many problems, so I trust that they're doing good work on the digital divide side, but when you can't do that, there are print solutions that can be really significantly less expensive than printing them at Kinko's or at the campus book store or something like that even. What was the second? >> Persistence. >> Oh, persistence. (coughs) So yeah, so I mean, I've got my dissertation files are on a Zip drive, anybody remember Zip drives? All of these OER, is that true? All the ones that we work with all start their life as HTML and are delivered to the students as HTML. Because they're open, there's no tricky drming, concealing, hiding, locking. It's just Vanilla HTML. So I mean, if you had Vanilla HTML from 93 or 94, it might not be as pretty as you wish it would be but you can still read it now. And that idea of kind of open formats for open educational resources is a really important idea specifically because of this durability problem. So if you were... I didn't talk about this at all, but we develop an open source kind of content management platform that speaks LTI and plugs into Canvas and things like that to make OER easier to reuse and you can pull entire copies of the textbook replacement thing, the OER that you've pulled together. You can pull it as HTML, you can pull it as PDF, you call pull it as epub which is just specifically kind of package HTML, right? You can pull it in I think nine different flavors for offline, but you can always pull it as HTML. And I think HTML is certainly more durable than the Word perfect files we come on. You know, you used WordPerfect. The WordPerfect files we used to make or the, you know, Word version two or whatever I think HTML's about as durable as it gets and that's the native format for all of this OER. >> Thanks. >> Yeah. With the exception of the multimedia stuff which has issues, yeah. But, you know, you do the best you can do and I think you'd be hard pressed to find a format that people can actually use with better durability than HTML. Good question. >> Any other questions? If not, I'd like to thank-- >> Oh wait, one more. >> Wait, wait-- >> There's more. But wait, there is more! (background noise drowns out other sounds) So hopefully, that work's already been done 'cause they're teaching a course where some other group of people has already done this and it's actually been used several times. And every semester longer that you wait, the proportion of the course that's already been done for you gets bigger and bigger. In a case where that hasn't been done, then it really becomes a process of sitting down, as you know, as instructions designer, sitting down next to the faculty member, talking through their outcomes and then really the best the single best piece of advice I can give you is take the entire learning outcome, put it in Google advance search and at the very bottom of Google advance search there is a usage rights constraint, we can set it to only show you results that are licensed for free use modification and something else, I can't remember exactly what it says now. But basically you can use Google in such a way that it will only return OER as results. And if you put the entire, don't say photosynthesis 'cause you'll get a billion things, if you put the entire learning outcome in as the search query and constrain the search by openly licensed material, then you can very quickly kind of do that realign process and pull that back together. But you do have to be super careful about all of the Creative Commons licenses require attribution of the original source where the material came from. So if you're not, like if you're not taking one of our open courses and just downloading and using it, well, if you were, then you would see that at the bottom of each page, there is full kind of citation management of every OER that's been used on the page and where it came from and how it's licensed. You have to do that yourself kind of from scratch and that can be somewhere between tricky and painful, yeah. (audience laughing) Yeah. >> All right. >> Thank you all very much for attending. >> Okay, thank you. (audience applauding)