trisha poole 31/1/20 trisha poole 31/1/20

How do I know whether my SoTL-based intervention worked?

In this post, we look deeper at how to determine whether the approach used to address the identified problem, challenge, or opportunity had impact. That is, how can we evidence the impact that is (or is not) being had? A critical aspect of our SoTL practice is investigating our intervention in order to determine whether it had an impact and what that impact was. This means that we need to collect appropriate evidence (i.e. data) that will speak to the impact. It might be worthwhile revisiting the understandings of evidence and impact in relation to SoTL…

In the last post, the ideas around reflection as a key part of the SoTL cycle were discussed. In this post, we look deeper at how to determine whether the approach used to address the identified problem, challenge, or opportunity had impact. That is, how can we evidence the impact that is (or is not) being had? For the purpose of this post on evidencing impact, the approach taken to address the problem, challenge or opportunity will be referred to as the intervention.

A critical aspect of our SoTL practice is investigating our intervention in order to determine whether it had an impact and what that impact was. This means that we need to collect appropriate evidence (i.e. data) that will speak to the impact. It might be worthwhile revisiting the understandings of evidence and impact in relation to SoTL.

What are evidence and impact?

In SoTL, as in research, evidence is the data that we can collect to support our claims of impact and evaluation of the SoTL intervention. However, a difference is that the evidence typically focuses on students, their learning, and our teaching. Therefore, the evaluations investigate the experiences of and changes in students and their learning in relation to the implemented approach. The SoTL focus on student learning is often examined through asking students about their experiences or through examining their interactions, artefacts (e.g. assessment items), and other objects or observable behaviours (Kelly, 2008).

When thinking about impact, it is useful to return to the original SoTL problem, challenge, or opportunity, and the research question that was developed for this. Impact is broadly the difference your intervention has made in relation to both the research question and the stakeholders involved, e.g. students, academics (IMPEL, 2016). While the impact occurs both during and after the SoTL project, it is important to be able to identify it clearly so that you can talk about it with others and disseminate your work. (Remember: Dissemination is a key part of SoTL and distinguishes SoTL from scholarly teaching.)

What can I do to evidence my impact?

Much like any research project, the evidence that you collect needs to be fit-for-purpose. That is, it needs to be able to talk to the impact that your intervention is having and thus it must align. The research questions, their objectives, indicators of impact, and data collected must be well-aligned in order to evidence the impact. Without this alignment, it will be difficult to demonstrate the impact that you are having and can see in your students and learning and teaching.

To start with, you need to clearly identify the research question, objective(s) of the intervention, indicators of when you will know that the objectives are being met, and what could evidence achievement of that objective(s). That is, what is your intervention aiming to achieve? What will be an indicator of having achieved that? What will be able to evidence that indicator? When you conceptualised your SoTL project, what were the key research questions and how were you going to know if they were achieved? What were the objectives? What indicators did you have in mind? It may be helpful to use a table to visualise this alignment. The table below is an example of how this might appear.

Table 1. Example of alignment between research question, objectives, indicators, and data collection.

Now that you have the alignment clearly visualised, you need to draw out the impact that the intervention is having during the SoTL project implementation and beyond. This will support the demonstration of the value of your intervention, so it is important to clearly articulate the impact, based on the evidence that has been gathered, and to present it in a concise way. One way to do this is by using the Impact Management Planning and Evaluation Ladder (IMPEL) model.

The IMPEL model can be used to identify different levels of impact that can result from SoTL projects (Department of Education & Training, 2016). The model starts by identifying key stakeholders from those closest to the project and then moves outward to students, the university, and then more broadly. The IMPEL table supports the clear identification of the impact at various times as well, starting from the current point in time, project end, six months post-project completion, 12 months post-project completion, and 24 months post-project completion.

Impact vs Outcomes

Now is a good time to check in about the differences between impact and outcomes. These terms are often confused or conflated, so clarity is important, particularly when disseminating your findings and impact to others. As above, impact refers to the difference that is being made and it focuses on doing things whereas outcomes are typically things (nouns) that have been achieved or developed (OALT, 2019). For example, an impact might be increased student engagement but an outcome could be a research paper written about the increased student engagement. It is important to keep these two areas clear and distinct, even though they can (and should!) talk to each other.

Connection to your SoTL project

It's time to bring this back to your SoTL project and how you can connect what you are observing with your intervention and how you can ensure that you have robust evidence to support it. Thinking about your original SoTL problem, challenge, or opportunity, and considering where it is at now, what are your objectives and what are the indicators that will evidence the achievement of these objectives? What data will support the evidencing of these indicators? Take time to draw your own table similar to that above with your research question, objectives, indicators, and data collection ideas. Do they align?

Now thinking about the impact that the intervention is having on you and your students. Start an IMPEL table with three columns: the first has the key stakeholders, the second has the current point in time, and the third has project completion. Can you fill in any of the table cells with ideas of what the current impact is? Remember that impact is actions (doing), not things (outcomes).

References

Department of Education and Training. (2016). The Impact Management and Planning Evaluation Ladder. https://docs.education.gov.au/documents/impact-management-planning-and-evaluation-ladder-impel

Kelly, D. (2008). Evaluating teaching and learning: enhancing the scholarship of teaching by asking students what they are learning. In Murray, R. (Ed.), Scholarship of Teaching and Learning in Higher Education (pp. 80-90). Online: McGraw-Hill Education.

OALT (2019). Applying for Academic Promotions Workbook: The Learning & Teaching Component. University of Southern Queensland: Toowoomba.

trisha poole 29/10/19 trisha poole 29/10/19

How do I know if I'm doing research or SoTL?

Building on the previous post introducing the scholarship of teaching and learning (SoTL), this post looks into the similarities and differences between SoTL and research. To recap, SoTL focuses on how we, as academics and teachers, use research to support our learning and teaching endeavours and examine our practice with the view to furthering the domain knowledge. It sounds simple, doesn't it? Just like research! But it is different, so let's explore…

What are the key similarities and differences between SoTL and research?

If we understand research to be a structured approach to informing a response to a question, then SoTL refines this further and narrows the bounds. SoTL is a structured approach to informing response/s to question/s about students and their education (including achievement, retention, transitions, and success); using these responses to make improvements; and sharing process and outcome knowledge with other educators (OALT, 2019). Through these two definitions, it is easy to see that SoTL is a more focused approach to research and includes specific elements such as reflection and feedback. The following table (OALT, 2019, p. 31) provides a brief comparison of the elements of research and SoTL.

OALT 2019 p 31 - SoTL vs Research table.png

One element listed in this table that often comes up for discussion is the reduction or buy-out of teaching because academics often want to reduce their teaching load so that they can focus on their research. Due to the nature of SoTL, i.e. that it focuses on one's teaching practice, it is not logical to reduce or buy-out teaching. Therefore, a key difference is the ability to use research funding to buy-out/reduce teaching and to "shift" some of the research work onto a research assistant. However, this is not possible in SoTL: in SoTL, it is expected that the academic/teacher will be conducting this research on their teaching in their context using their experiences and knowledge. This is not practical if the academic/teacher is not teaching.

So, how do we "do" SoTL?

Conducting SoTL has a broad yet consistent approach, regardless of the context. The process is similar to that of research, but also slightly different. Typically, an academic/teacher identifies a student problem, opportunity, or challenge (often simply referred to as the "problem"), and then uses the literature to support an approach to addressing this. That is, the proposed approach to the problem is evidence-based and grounded in research. Part of determining the approach to the problem is also deciding on the appropriate types of evidence that will be collected in order to demonstrate effectiveness and impact. Because a critical aspect of SoTL is dissemination, ethics approval is typically required before collecting the evidence, so this goes into the cycle as well. While implementing the approach, the evidence is collected and analysed. There can be iterative design approaches used where the approach is iterated and refined through a number of teaching cycles. After the approach is implemented, it is evaluated, and the results (positive or negative) are disseminated. Then the process can start again. Throughout the process, there is reflection and refinements. These parts all come together to build a picture and tell the story of how we are improving the student learning journey through our active, engaged learning and teaching practice. The following image is an attempt at describing the SoTL process, but it is typically not as linear and smooth as the image makes it appear.

Remember that the SoTL cycle depicted here is the smooth, flowing version, not necessarily the reality. SoTL can be an exciting process in which we can become caught in a loop of iteration and refinement because we always want to improve what we are doing. At times, this iteration and refinement loop may seem never-ending, but we need to disseminate so that we can receive feedback from peers and then continue to refine and iterate our approach. Keep in mind that our teaching contexts rarely stay constant, so our approach also needs some flexibility and adaptability.

How does this connect to your GELTS?

Through your GELTS project, you should be working through this SoTL cycle. In the application phase, you have already identified your student problem, the related literature, and your proposed approach. Now, you are working on implementing your proposed approach, which should also include ethics approval to collect the data that you are using to evidence your approach.

With that in mind, consider the following questions and let's start a discussion about your GELTS project in response to this thread.

In brief, what is your SoTL project?
How is it similar to or different from your typical discipline research focus?
How have you engaged with the SoTL cycle so far?
Have you considered reflection activities as part of your SoTL project?

Note that if you want to share a SoTL focus that you have outside of your GELTS, then feel free to do that, too.

Delving further

The field of SoTL is ever-expanding, so a quick Google Scholar or Library search will result in different articles in different disciplines for further reading and consideration. I have included two articles below if you would like to start with these. The first article provides an overview of SoTL from Hutchings' perspective (which is the introduction to his edited book on SoTL, Opening Lines), and the second article is a write up of a SoTL project in the science discipline. (I will endeavour to include a SoTL article from different disciplines as we go through the GELTS Coffee Course.)

Hutchings, P. (2000). Approaching the scholarship of teaching and learning. In: P. Hutchings (Ed.) Opening Lines: Approaches to the Scholarship of Teaching and Learning, (pp. 1-10) Princeton, NJ: Carnegie Foundation for the Advancement of Teaching.

Rowland, S. L. and Myatt, P. M. (2014). Getting started in the scholarship of teaching and learning: A “how to” guide for science academics. Biochem. Mol. Biol. Educ., 42,6-14. doi: 10.1002/bmb.20748

References

Office for the Advancement of Learning & Teaching [OALT] (2019). Applying for Academic Promotions Workbook: The Learning & Teaching Component. University of Southern Queensland: Toowoomba.