|Posted on September 5, 2019 at 8:15 AM|
Let me begin with a brief sidebar on assessment vocabulary. Assessment in higher education is relatively new—only a few decades old—and we don’t yet have a standard vocabulary. Specifically, we don’t have agreement on the terms “learning objectives,” “learning competencies,” “learning goals,” and “learning outcomes.” Some people draw distinctions among these terms; I don’t. Many people use the term “learning outcome,” even creating acronyms for course learning outcomes (CLOs) and program learning outcomes (PLOs). I prefer the term “learning goal” because I’ve found some people think “learning outcomes” refer to assessment results—the actual learning outcome as opposed to the intended or expected learning outcome. I don’t want to make assessment any more confusing than it already is!
Learning goals (or whatever you want to call them) describe what students will be able to do as a result of successful completion of a learning experience, be it a course, program or some other learning experience. So course learning goals describe what students will be able to do upon passing the course, and program learning goals describe what students will be able to do upon successfully completing the (degree or certificate) program.
Course and program learning goals are not comprehensive lists of every single minute thing students will learn. (An important exception: some specialized accreditors do have long lists of required competencies.) Instead, an effective course or program focuses on a few key learning goals that are so important that they are addressed throughout the curriculum. Key course learning goals should be addressed through multiple assignments. Key program learning goals should be addressed in at least two required courses or other program requirements. The reason is that we want students to learn these important things really well, and students learn best through repeated practice in a variety of contexts. It’s simply unfair to both students and faculty to place full responsibility for student achievement of a key course learning goal on just one assignment. It’s similarly unfair to students and faculty to place full responsibility for student achievement for a key program learning goal on just one faculty member or one required course.
Because programs are of course broader than courses, program learning goals are typically broader than course learning goals. Course learning goals may address the building blocks necessary to achieve the program learning goal. Or they may address aspects or contexts of the program learning goal.
Here are three examples from Chapter 4 of my book Assessing Student Learning: A Common Sense Guide.
- Several courses in a program may each help students develop a specific technological skill. Those course learning goals collectively help students achieve a program learning goal to use technologies appropriately and effectively.
- A course learning goal that students solve a specific kind of problem helps students prepare to achieve a program learning goal to design appropriate approaches to solving a variety of problems in the discipline.
- An English course on Shakespeare might have a course learning goal to analyze scholarly views on character motivations. This learning goal, along with other course learning goals in other English literature courses, prepares students to achieve the English program learning goal to conduct research on issues in the study of literature.
By the time students reach the program’s capstone requirement, the course and program learning goals may be the same. If the capstone is a research project, for example, the capstone’s learning goals may include program learning goals addressing research, written communication, and information literacy skills. If the capstone is a field experience, the capstone’s learning goals may include program learning goals addressing clinical, technology, communication, and interpersonal skills.
I’ve found that, if faculty are struggling to articulate program learning goals, the problem is often the program’s curriculum. As I frequently point out, a collection of courses is not a program. But I see a lot of academic programs that are exactly that: collections of courses, nothing more. They lack coherence and focus; there are no common threads of shared program learning goals that bind the courses together.
For more information on learning goals and curriculum design, see Chapters 4 and 5 of the 3rd edition of Assessing Student Learning: A Common Sense Guide.
|Posted on March 27, 2019 at 5:40 AM|
Burning Glass Technology recently released a report on a study of skills that employers included in online job postings in over 50,000 online job boards, newspapers, and employer websites.
Before I get to the meat of their findings, an important caveat: While 50,000 online employment sites sound impressive, they’re clearly not representative of all jobs sought and filled by college graduates. The jobs discussed in the report are heavy on information technology and business. There’s no mention of many other fields such as teachers, social workers, scientists, clergy, or musicians. The report acknowledges the heavy weight on IT by separating results for digital occupations from other occupations, but I still don’t think the results are representative of all employers everywhere. That said, let’s dive in.
A few of the skills that employers seek are ones that already show up on virtually every college’s list of institutional or general education learning goals: communication, critical thinking, and analytical skills. Two others—collaboration and creativity—show up occasionally although, in my view, far too infrequently.
The remaining skills are largely absent from institutional or general education learning goals:
- Analyzing data
- Communicating data
- Digital design
- Project management
- “Business process” (skills with cost control, business operations, planning, and strategy)
- IT skills including computer programming, software development, data management, and digital security
I’m not going to recommend anything based on this one, somewhat flawed study. But it generates some ideas for all of us to think about:
- Should our curricula be giving greater emphasis to creativity, collaboration, and visual communication?
- I’ve heard arguments that gen ed math courses should be statistics courses, and this study, showing the need for skills in analyzing and communicating data, reinforces them.
- Should we not only require program capstones but require that they be projects that students are responsible for planning and completing, thereby developing project management skills? Should we encourage group capstone projects, thereby helping students develop collaboration skills?
- Would liberal arts students benefit from a course or badge that gives them basic skills with IT and business processes?
|Posted on November 13, 2018 at 6:50 AM|
I’m mystified by how Bloom’s taxonomy has pervaded the higher education assessment landscape. I’ve met faculty who have no idea what a rubric or a test blueprint or a curriculum map is, but it’s been burned into their brains that they must follow Bloom’s taxonomy when developing learning goals. This frustrates me no end, because I don’t think Bloom’s is the best framework for considering learning outcomes in higher education.
Bloom’s taxonomy of educational objectives is probably older than you are. It was developed by Benjamin Bloom in the 1950s. It divides learning goals into three domains: cognitive, affective (attitudinal), and psychomotor. Within the cognitive domain, it has six levels. Originally these were knowledge, comprehension, application, analysis, synthesis, and evaluation. A 2000 update renamed these levels and swapped the positions of the last two: remember, understand, apply, analyze, evaluate, and create. The last four levels are called higher order thinking skills because they require students to do more than understand.
So why don’t I like Bloom’s? One reason is that I’ve seen too many faculty erroneously view the six cognitive levels as hierarchy of prerequisites. Faculty have told me, for example, that first-year courses can only address knowledge and comprehension because students must thoroughly understand a subject before they can begin to think about it. Well, any elementary school teacher can tell you that’s bunk, but the misperception persists.
Even more important is that Bloom’s doesn’t highlight many of the skills and dispositions needed today. Teamwork, ethical judgment, professionalism, and metacognition are all examples of learning goals that don’t fit neatly into Bloom’s. That’s because they’re a combination of the cognitive and affective domains: what educators such as Costa & Kallick and Marzano and his colleagues call habits of mind.
I’m especially concerned about professionalism: coming to work or class on time, coming to work or class prepared to work, completing work on time, planning one’s time, giving work one’s best effort, self-evaluating one’s work, etc. Employers very much want these skills, but they get short shrift in Bloom’s.
So what do I recommend instead? In my workshops I suggest five categories of learning goals:
- knowledge and understanding
- career-specific thinking and performance skills
- transferrable thinking and performance skills (the kinds developed in the liberal arts)
- attitudes and values
- habits of mind
But I also like the taxonomies developed by Dee Fink and by Marzano et al.
I wouldn’t expect every course or program to have learning goals in all five of these categories, of course. But I do suggest that no more than half of a course or program’s learning goals be in the knowledge and understanding category.
For more information, see Chapter 4 (Learning Goals: Articulating What You Most Want Students to Learn) in the new 3rd edition of my book Assessing Student Learning: A Common Sense Guide.
|Posted on September 23, 2018 at 10:35 AM|
A recent Inside Higher Ed piece, “The Contamination of Student Assessment” by Jay Sterling Silver, argued that behaviors such as class attendance and class participation shouldn’t be factored into grades because grades should be “unadulterated measurements of knowledge and skills that we represent them to be—and that employers and graduate admissions committees rely on them to be.” In other words, these behaviors are unrelated to key learning goals.
He’s got a point; a grade should reflect achievement of key learning goals. (That’s what competency-based education tries to achieve, as I discussed in a blog several years ago.) But I behaviors like coming to class, submitting work on time, giving assignments one’s best effort, and participating in class discussions are important. They fall under what I call professionalism: traits that include coming to work on time and prepared to work, dependably completing assigned work thoroughly and on time, giving one’s work one’s best effort, and managing one’s time.
Surveys of employers confirm that these are important traits in the people they hire. Every few years, for example, Hart Research Associates conducts a survey for AAC&U on how well employers think college graduates are prepared on a number of key learning outcomes. The 2018 survey added two learning outcomes that weren’t in previous surveys:
- Self-motivated: ability to take initiative and be proactive
- Work independently: set priorities, manage time and deadlines
Of the 15 learning outcomes in the 2018 survey, these were tied for #4 in importance by hiring managers.
So I think the answer is to add professionalism as an additional learning goal. Of course “professionalism” isn’t a well-stated learning goal; it’s a category. I leave it up to college and program faculty to decide how best to articulate what forms of professionalism are most important to their students and prospective employers.
Then an assignment like a library research paper might have three learning goals—information literacy, writing, and professionalism—and be graded on all three. Professionalism might be demonstrated by how well students followed the directions, whether the assignment was turned in on time, and whether the student went above and beyond the bare minimum requirements for the assignment.
Professionalism, by the way, isn’t just a skill and isn’t just an attitude. It’s a combination of both, similar to what Arthur Costa and Bea Kallick call habits of mind, which include things like persisting, managing impulsivity, taking responsible risks, and striving for accuracy. One of the reasons I’m not a fan of Bloom’s taxonomy is that it doesn’t really address habits of mind, which—as evidenced by Hart’s new survey—are becoming increasingly important learning goals of a college education.
|Posted on July 15, 2018 at 7:45 AM|
The word “demonstrate” in learning goals raises a red flag for me. Consider these (real) learning goals:
- Demonstrate fundamental business and entrepreneurship skills
- Demonstrate critical and creative thinking.
- Demonstrate information literacy skills.
- Demonstrate teamwork and collaboration.
- Demonstrate ethical self-awareness.
- Demonstrate personal responsibility.
Clearly the people who wrote these learning goals were told that they had to start with an action word. So they plopped the word “demonstrate” in front of a fuzzy goal. But adding “demonstrate” doesn’t make the goal any less fuzzy. What are “fundamental business and entrepreneurship skills”? What is “personal responsibility”? Until these concepts are stated more clearly, these learning goals remain fuzzy and therefore difficult to assess meaningfully.
Now consider these (real) learning goals:
- Demonstrate proficiency in analyzing work-related scenarios, taking appropriate action and evaluating results of the action.
- Demonstrate proficiency in the use of technology for collecting and analyzing information
- Demonstrate the ability to work cooperatively with others
- Demonstrate enhanced competencies in time management
Here the phrase “demonstrate proficiency/ability/competencies” are simply superfluous, making the learning goal unnecessarily wordy. Consider these restatements:
- Analyze work-related scenarios, take appropriate action, and evaluate the results of the action.
- Use technology to collect and analyze information.
- Work cooperatively with others.
- Manage time effectively.
Not only are they clearer but, because they’re shorter, they pack a punch; they have a better chance of engaging students and getting them enthused about their learning.
So should we abolish the word “demonstrate” from our assessment lexicon? Well, consider this (real) learning outcome:
- Demonstrate appropriate pitch, tone and demeanor in professional settings.
If we make clear what we want students to demonstrate, using observable terms, “demonstrate” may be fine.
Now consider these (real) learning outcomes:
- Demonstrate appropriate, professional conduct. (if you define it)
- Demonstrate professionalism and cultural sensitivity while interacting and communicating with others.
It could be argued that these learning outcomes are a bit fuzzy. What is appropriate, professional conduct, after all? What is cultural sensitivity? But if we clarified these terms in the learning outcome, we’d come up with a pretty long list of traits—so many that the learning outcome would be too cumbersome to be effective. In these cases, I’m okay with leaving these learning outcomes as is, provided that the rubrics used to assess them explicate these terms into traits with clear, concrete language that students easily understand.
So, no, I don't think we should abolish the word "demonstrate" altogether, but think twice--or even three times--before using it.
|Posted on May 27, 2018 at 7:40 AM|
When I help faculty and co-curricular staff move ahead with their assessment efforts, I probably spend half our time on helping them articulate their learning goals. As the years have gone by, I’ve become ever more convinced that learning goals are the foundation of an assessment structure…and without a solid foundation, a structure can’t be well-constructed.
So what are well-stated learning goals? They have the following characteristics:
They are outcomes: what students will be able to do after they successfully complete the learning experience, not what they will do or learn during the learning experience. Example: Prepare effective, compelling visual summaries of research.
They are clear, written in simple, jargon-free terms that everyone understands, including students, employers, and colleagues in other disciplines. Example: Work collaboratively with others.
They are observable, written using action verbs, because if you can see it, you can assess it. Example: Identify and analyze ethical issues in the discipline.
They focus on skills more than knowledge, conceptual understanding, or attitudes and values, because thinking and performance skills are what employers seek in new hires. I usually suggest that at least half the learning goals of any learning experience focus on skills. Example: Integrate and properly cite scientific literature.
They are significant and aspirational: things that will take some time and effort for students to learn and that will make a real difference in their lives. Example: Identify, articulate, and solve problems in [the discipline or career field].
They are relevant, meeting the needs of students, employers, and society. They focus more on what students need to learn than what faculty want to teach. Example: Interpret numbers, data, statistics, and visual representations of them appropriately.
They are short and therefore powerful. Long, qualified or compound statements get everyone lost in the weeds. Example: Treat others with respect.
They fit the scope of the learning activity. Short co-curricular learning experiences have narrower learning goals than an entire academic program, for example.
They are limited in number. I usually suggest no more than six learning goals per learning experience. If you have 10, 15, or 20 learning goals—or more—everyone focuses on trees rather than the forest of the most important things you want students to learn.
They help students achieve bigger, broader learning goals. Course learning goals help students achieve program and/or general education learning goals; co-curricular learning goals help students achieve institutional learning goals; program learning goals help students achieve institutional learning goals.
For more information on articulating well-stated learning goals, see Chapter 4 of the new 3rd edition of my book Assessing Student Learning: A Common Sense Guide.
|Posted on May 2, 2018 at 6:55 AM|
I look on learning goals as promises that we make to students, employers, and society: If a student passes a course or graduates, he or she WILL be able to do the things we promise in our learning goals.
But there are some things we hope to instill in students that we can’t guarantee. We can’t guarantee, for example, that every graduate will be a passionate lifelong learner, appreciate artistic expressions, or make ethical decisions. I think these kinds of statements are important aims that might be expressed in a statement of values, but they’re not really learning goals, because they’re something we hope for, not something we can promise. Because they’re not really learning goals, they’re very difficult if not impossible statements to assess meaningfully.
How can you tell if a learning goal is true learning goal—an assessable promise that we try to keep? Ask yourself the following questions.
Is the learning goal stated clearly, using observable action verbs? Appreciate diversity is a promise we may not be able to keep, but Communicate effectively with people from diverse backgrounds is an achievable, assessable learning goal.
How have others assessed this learning goal? If someone else has assessed it meaningfully and usefully, don’t waste time reinventing the wheel.
How would you recognize people who have achieved this learning goal? Imagine that you run into two alumni of your college. As you talk with them, it becomes clear that one appreciates artistic expressions and the other doesn’t. What might they say about their experiences and views that would lead you to that conclusion? This might give you ideas on ways to express the learning goal in more concrete, observable terms, which makes it easier to figure out how to assess it.
Is the learning goal teachable? Ask faculty who aim to instill this learning goal to share how they help students achieve it. If they can name specific learning activities, the goal is teachable—and assessable, because they can grade the completed learning activities. But if the best they can say is something like, “I try to model it” or “I think they pick it up by osmosis,” the goal may not be teachable—or assessable. Don’t try to assess what can’t be taught.
What knowledge and skills are part of this learning goal? We can’t guarantee, for example, that all graduates will make ethical decisions, but we can make sure that they recognize ethical and unethical decisions, and we can assess their ability to do so.
How important is this learning goal? Most faculty and colleges I work with have too many learning goals—too many to assess well and, more important, too many to help students achieve well in the time we have with them. Ask yourself, “Can our students lead happy and fulfilling lives if they graduate without having achieved this particular learning goal?”
But just because a learning goal is a promise we can’t keep doesn’t mean it isn’t important. A world in which people fail to appreciate artistic expressions or have compassion for others would be a dismal place. So continue to acknowledge and value hard-to-assess learning goals even if you’re not assessing them.
For more information on assessing the hard-to-assess, see Chapter 21 of the new 3rd edition of Assessing Student Learning: A Common Sense Guide.
|Posted on January 9, 2018 at 7:25 AM|
Just before the holidays, the Council of Graduate Schools released Articulating Learning Outcomes in Higher Education. The title is a bit of misnomer; the paper focuses not on how to articulate learning outcomes but on why it’s a good idea to articulate learning outcomes and why it might be a good idea to have a learning outcome framework such as the Degree Qualifications Profile to articulate shared learning outcomes across doctoral programs.
What I found most useful about the paper was the strong case it makes for the value of articulating learning outcomes. It offers some reasons I hadn’t thought of before, and they apply to student learning at all higher education levels, not just doctoral education. If you work with someone who doesn't see the value of articulating learning outcomes, maybe this list will help.
Clearly defined learning outcomes can:
• Help students navigate important milestones by making implicit program expectations explicit, especially to first-generation students who may not know the “rules of the game.”
• Help prospective students weigh the costs and benefits of their educational investments.
• Help faculty prepare students more purposefully for a variety of career paths (at the doctoral level, for teaching as well as research careers).
• Help faculty ensure that students graduate with the knowledge and skills they need for an increasingly broad range of career options, which at the doctoral level may include government, non-profits, and startups as well as higher education and industry.
• Help faculty make program requirements and milestones more student-centered and intentional.
• Help faculty, programs, and institutions define the value of a degree or other credential and improve public understanding of that value.
• Put faculty, programs, and institutions in the driver’s seat, defining the characteristics of a successful graduate rather than having a definition imposed by another entity such as an accreditor or state agency.
|Posted on July 21, 2015 at 6:55 AM|
Over the last few years there have been a number of studies of the knowledge, skills, and competencies that employers seek in new hires. The results have varied because many of the studies have asked employers to choose from a predetermined list, and those lists can vary considerably. But last year, when I was working on my book Five Dimensions of Quality: A Common Sense Guide to Accreditation and Accountability, I reviewed those studies and found persistent mention of a number of competencies.
Now some new surveys—one sponsored by AAC&U and others mentioned in a piece by Margaret Andrews in University World News—have reinforced what I found. The following competencies seem most valued by employers, in roughly the following order:
• Teamwork and collaboration
• Real-world problem solving
• Evaluating information and conclusions
• Flexibility and adaptability to change
• Creativity and innovation
• Working with people from diverse cultural backgrounds
• Ethical judgment
• Understanding numbers and statistics
How well are your curricula designed—and your faculty prepared and equipped—to help students develop these skills?
|Posted on February 5, 2015 at 7:55 AM|
A recent study by Hart Research Associates for the Association of American Colleges & Universities found, among many other things, that only about a quarter of employers are satisfied with the creative and innovative skills of recent college graduates. Why are college graduates so dismal in this respect? Throughout their education, from grade school through college, in most classes, the way to get a good grade is to do what the teacher says: read this assignment, do this homework, write a paper on this topic with these sections, develop a class presentation with these elements. Faculty who teach general education courses in the creative arts—art, theater, creative writing, even graphic design—have told me that students hate taking those courses because they have no experience in “thinking outside the box.”
How can we encourage creativity and innovative thinking? Simply building it into our grading expectations can help. The first time I used a rubric, many, many years ago, I gave it to my class with their assignment, and the papers I received were competent but flat and uninspired. I had to give the best papers A’s because that was what the rubric indicated they should earn, but I was disappointed.
The next time I taught the course, I changed the rubric so that all the previous elements earned only 89 points. The remaining points were for a fairly vague category I labeled “Creative or innovative ideas or insight.” Problem solved! The A papers were exactly what I was hoping for.
Now this was a graduate course, and just putting something on a rubric won’t be enough to help many first-year students. This is where collaborative learning comes into play. Put students into small groups with a provocative, inspiring question for them to discuss, and watch the ideas start to fly.
|Posted on July 10, 2014 at 5:35 AM|
I love Alison Head and John Wihbey’s piece, “At Sea in a Deluge of Data” in this week’s Chronicle of Higher Education. They talk about a particular skill that’s growing in importance in the 21st century, what I call seeing the 30,000-foot picture: taking a lot of information, seeing the big ideas from all that information, and communicating the big points clearly and understandably.
Many colleges have a hard time helping their students develop this skill. Traditional library research papers may help, but they don’t give students the real-world integrative skills that employers are looking for: separating the information wheat from the chaff (the relevant from the irrelevant and the credible from what I like to call the incredible) and communicating big points in short, succinct ways that people can quickly and easily understand (see my earlier blog on infographics).
One reason that I think we have a hard time helping students develop this skill is because so many of us struggle with this ourselves. Seeing the 30,000-foot picture doesn’t come naturally to most people. David Keirsey has found that only about 5-10% of the population has the inherent temperament for big-picture analysis; people are far more likely to be detail-oriented. (You can take the Keirsey Temperament Sorter at www.keirsey.com and see where you fit.)
I see this a lot in work on assessment and accreditation. People are good at saying, “We used this rubric and here are the scores,” “Students took this survey and here are their responses,” “Here are grade distributions from key gateway courses.” But people often struggle to connect those pieces. What do your rubric, survey, and grade distribution results each say about students’ writing skills, for example? What are they telling you overall about students’ writing skills? Are the survey results and grades helping you understand why you’re getting your rubric results? Accreditors are less interested in a table of results than in what the results are saying to you. What overall conclusions can you draw about your students’ writing skills?
We need both detail and 30,000-foot people working on assessment and accreditation activities. Make sure you’ve got both on your team.
|Posted on September 14, 2013 at 8:10 AM|
I've reviewed a number of surveys of employers, and the following crop up most often, in roughly the following order:
• Teamwork and collaboration, including listening
• Written and oral communication, especially articulating ideas clearly and effectively
• Real-world problem solving, especially complex problems, under pressure or “on the fly”
• Critical thinking and analysis, especially in evaluating information and conclusions
• Flexibility and adaptability to change, including the capacity to continue learning
• Creativity and innovation
• Intercultural knowledge and skills, especially working with people from diverse cultural backgrounds
• Ethical judgment
• Quantitative and computer skills, especially understanding numbers and statistics
This list is similar to but not the same as AAC&U's Essential Outcomes (LEAP goals) and Lumina's Degree Qualifications Profile.
How do your college's program and general education learning outcomes compare to this list? Are your students graduating with the skills they need for success in the 21st century?