While studies can be quantitative or qualitative , keep in mind that projects may collect both types of data. Also, a quantitative study could have an open-ended question on a survey that would collect some interesting qualitative data. Likewise, information collected from a qualitative study may also include some information summarized into a quantitative format; for example, the proportion of the people interviewed that mentioned a certain topic.
Mixed Methods. A mixed methods approach involves integrating methodologies, traditionally quantitative and qualitative , but arts-based methods can also be integrated with quantitative and qualitative. The benefit of mixing methods is that it can help you overcome the limitations and weaknesses that arise when each method is used alone and allow you to ask a broader set of questions.
If different data sources reveal the same findings, or findings that are coherent with each other, this can lend credibility to your evaluation. A mixed method evaluation can also deepen understanding of your program, its effects, and context. Visit the Examples page to see examples of projects using mixed methods approaches. What stage s of the evaluation to mix methods? The design is considered much stronger if mixed methods are integrated into several or all stages of the evaluation. Will methods be used: sequentially the data from one source inform the collection of data from another source , or concurrently triangulation is used to compare information from different independent sources?
Will qualitative and quantitative methods will be given relatively equal weighting? Arts-Based Evaluation. Arts-based evaluation ABE utilizes art or artistic processes in the evaluation process itself. It is important to distinguish between ABE methods and the evaluation of arts-based projects. ABE methods conversely, can be very effective in gathering data on projects that have little to do with art or social change. As such, the arts-based evaluation methods should be planned for and prepared in advance of implementing the project, as with other evaluation methods.
ABE methods do not necessarily have a social change component. For example, adolescents attending a course on financial accounting may be asked to draw a picture or write a poem to indicate how they felt about the course; this would be an example of ABE, but the intervention had nothing to do with social change.
At the same time, this exercise can have great evaluative or research value that also sheds light on the effectiveness of the ASC intervention at achieving its desired goals, including the challenges faced. The photographs can be interpreted to derive themes and narratives for further research. Some may reflect directly on what a community-based project means to them. For example, persons living with a disability could be asked to photograph what is meaningful to them - some participants may photograph the barriers they face on a daily basis; others show various accommodations implemented.
The photos are analyzed to identify common barriers as well as to reflect on how participants feel about measures offered. Visit the Examples page to see examples of projects using arts-based evaluation methods. One commonly-used ABE method is photovoice. In photovoice, participants are asked to photograph images that represent the research subject, and then the photographs can be interpreted to derive themes and narratives. For examples, persons living with a disability are asked to photograph the barriers that they face on a daily basis.
Then the photos are analyzed to identify common barriers. Social Context. Social context is also fundamental to program implementation:. Finally, social context shapes the production and utilization of evaluation findings:. Advantages of arts-based methods include:. Challenges of arts-based methods include:. Boydell, K. The production and dissemination of knowledge: A scoping review of arts-based health research.
It is always useful for an organization to review what they are hoping to accomplish and how they are doing. We'd like to see the numbers. What percent improvement do you have for the indicators you are evaluating? Who is initiating the evaluation? On the other hand, a self-driven evaluation may focus more on qualitative data , such as examining the experience of the participants. If the evaluation is funder-driven, it can be helpful to discuss with the funder the evaluation process and how the results will be used.
Keep in mind that research does not always have to be quantitative; it may be preferable to use qualitative or arts-based methods, or even a combination of quantitative and qualitative methods. While we can provide some numbers, we'd also like to examine the experience of the participants.
The social challenges and problems we face today are often at the intersection of many systems, contextual factors and the relations between them- in other words, the social processes involved. Breilh, J. Latin american critical 'social' epidemiology: New settings for an old dream. International Journal of Epidemiology , 37 4 , En: Passos R, organizador.
Krieger, N. Epidemiology and the people's health: Theory and context. New York: Oxford University Press. Spiegel, J. Community Development Journal. Global Public Health. Kania and Kramer focus on shared measurement, which goes hand-in-hand with authentic partnerships. As with other types of evaluation, it is important to consider the theory of social change underlying the program as this will inform the focus of the evaluation.
In shared measurement, this is especially important to establish among the collective impact partners, as the partners may otherwise approach social change at different levels — for example, at the micro individual level versus the meso group level — and this would influence the respective choice of measures or indicators to collect and analyze within the shared measurement system. One example of a method relevant for collective impact and its shared measurement is outcome mapping.
In outcome mapping, the focus shifts from measuring direct causal outcomes to focus on behavioural changes and the subsequent ripple effect as those changes move into the wider community and into other domains. This is particularly relevant for collective impact, where the overall goals of social change may be too complex and longstanding to attribute as direct causal outcomes to individual programs. Cabaj, M. Tamarack Institute. The Philanthropist , 26 1.
Earl, S. International Development Research Centre. Collective Impact Forum. Kania, J. Collective impact. Stanford Social Innovation Review , 9 1 , Kramer, M. Breakthroughs in Shared Measurement and Social Impact. Patton, M. Utilization-Focused Evaluation 4th ed. Thousand Oaks: Sage Publications. Guiding Principles. Evaluating community-based social change initiatives often entails working within inequities, such as those caused by systemic discrimination and unequal distribution of power.
In working within inequities, a set of principles may be helpful to guide ethical and equitable work in this area. The principles are:. However, traditional evaluation methods are not always appropriate for art for social change projects. These reasons may relate to:. From Evaluation to Research. It may be wise to see this as a continuum rather than a dichotomy either one or the other. If the intention is to publish in a peer-reviewed journal, this would be to the right of the continuum and would require research ethics approval a requirement for publishing in a peer-reviewed academic journal.
Alternatively, if the evaluation is being conducted for the organization's own internal use and not intended for a broader audience, this would be to the left of the continuum.
Hiring an evaluator. We designed this tool so any organization can conduct evaluation in-house. However, if your funding allows it, you may want to hire an evaluator. It is important to consider this early in the planning stages, as an evaluator will likely have valuable input. Keep in mind that the evaluator your group selects will have an impact on the type of evaluation they conduct.
It's important to select someone who shares your vision. It may also be the case that you want to conduct the evaluation without an external evaluator. This could be beneficial as nobody will know your program better than your own team. How do you see your art for social change project evaluation? Reveal how the ASC project can empower people to effectively mobilize for social change.
Reveal the multiple realities to challenge dominant discourses about ASC projects such as this one. What does the project hope to accomplish? The answer to this question will determine what are suitable method s to use for your evaluation.
For instance, a particular outcome may be best evaluated with quantitative methods whereas a different outcome may be better suited to a qualitative evaluation, or even an arts-based approach. Below are a few of very many examples of desired outcomes from art for social change projects. The list above provides only a few examples. Visit the Examples page to see more examples of ASC project outcomes.
Logic Model. Projects and programs should try to plan their activities with as clear as possible idea of the outputs, outcomes and impact they wish to have — and how each of these will be assessed.
We have compiled a searchable list of examples. In addition to filtering by outcome, you can filter the list by:. In other words, the authors were likely funded as researchers not only to evaluate the specific program, but to contribute new knowledge to the body of scholarship on the impact of ASC project. Please do not feel that these kind of in-depth undertakings are necessarily needed for routine ASC evaluations. Click here to go to examples.
Who should you engage with during the evaluation? It's important to involve your partners throughout the evaluation process. Across the field, ASC practitioners emphasize the value of long-term partnerships and projects both as crucial to deepening impact and as an ethical imperative in community-engaged work.
As an integral part of this community-engaged work, partners should be actively involved in the evaluation process, including involvement in:. Throughout the project and evaluation process, key success factors in developing partnerships include:.
There are various ways to organize and structure partnerships, both at the governance and day-to-day operational levels. Navigating partnerships effectively can be complex and challenging, so ongoing monitoring and evaluation of the partnership itself can be beneficial to help ensure alignment and mutual benefit among partners.
The Canadian Coalition for Global Health Research CCGHR offers a Partnership Assessment Tool to guide and support the joint evaluation of health research partnerships, although it can be applied to other sectors including art for social change evaluations.
The tool consists of a series of questions and exercises, leading through four stages:. Take a quiz, proceed to the next section, or go back to the main menu:.
Quiz Your Knowledge. Proceed To Part 2: Ethics. Return to Main Menu. End of Part 1: Ethics. Proceed To Part 3: Methods. End of Part 3: Methods. Proceed To Part 4: Techniques. End of Part 4: Techniques. Proceed To Part 5: Analysis. End of Part 5: Analysis.
Proceed To Part 6: Dissemination. End of Part 6: Dissemination. Proceed To End of Course. End of Mini-Course. You have completed our mini-course on evaluative research. Hopefully this has aided in providing some valuable information about some of the more complex issues and concepts involved in doing a more nuanced research project.
Quantitative evaluation is outcome-oriented. You will need to have predefined outcomes for your project. You will then test to see how your program is doing with respect to these outcomes using numerical data. For example, you may want to design an evaluation that tests to see if participation in your program promotes self-esteem.
It is even better to compare these changes in your program participants with changes in a similar group that didn't participate in the ASC project. For more information on comparison groups, see the Evaluative Research Mini-Course. When To Use Quantitative Evaluation.
Quantitative evaluations are useful for an ASC project that aims to impact a specific, predefined outcome. For example, change in use of alcohol or drugs, improvement in school grades or completion, or even constructs such as self-esteem or social inclusion. The project should also have a large enough group of participants to have the statistical power to be able to detect a meaningful change related to the program if indeed it did occur. There are several factors which influence the calculations for sample size and statistical power.
Online sample size calculators can help you quickly calculate the sample size needed based on the population size, confidence level, and margin of error. This type of evaluation works less well when an ASC project prefers to:.
Photo from Jumblies Theatre, used with permission. Primary or Secondary Data. The data for analysis can be information gathered specifically for this study, also called primary data. This is usually done with surveys. An analysis can also be cone on data already available. This is called secondary data. For example, an ASC project concerned with whether engaging in community-based arts increases grades for at-risk youth can look at youth transcripts with informed consent , or decreased school drop out rates in the community as a whole.
This would be secondary data. An ASC project concerned with whether engaging in community-based arts decreases the crime rate for young offenders could look at criminal records. This would also be secondary data. Cross Sectional vs. Longitudinal Evaluations.
A cross sectional evaluation will gather data from a single point in time. A longitudinal evaluation will gather data more than once, to observe how certain variables change over time. Typically a longitudinal evaluation will take place over a long period of time, often many months or even years. However, a longitudinal study could take place over a short period of time. For instance, a survey could be taken at the start of a weekend workshop, and then retaken at the end of the weekend to observe any changes.
Note that a retrospective post-pre evaluation can also be done. This is when you ask people to answer various questions according to how they feel now, as well as a comparison question according to how they remember feeling about this question when they started the program.
Comparison Group. If you are going to conduct a longitudinal evaluation, it could be beneficial to have a comparison group. That is, an extra group will participate in the pre-post survey, but will not participate in the art intervention. This can make for what is considered to be a stronger study; it can show the changes occurring over time between the two groups. Keep in mind that the comparison group needs to be similar to the group that is receiving the art intervention. This may not always be feasible.
If participants self-select into their own groups, you may find that each group is very different to begin with. Additional Reading. Ashburn, A. Project Length: Cross-sectional. A cross-sectional analysis, whether qualitative or quantitative, takes place at a single point in time. For example, doing a single survey of program participants after a group art project.
A comparative cross-sectional analysis also occurs at a single point in time. However, it is more comprehensive than a descriptive analysis of just one program, and will consider the viewpoints or experience of those in both the community that is, as well as one that is not, in the ASC project.
A longitudinal analysis will evaluate the same group of people over a period of time, and will monitor how their views and opinions change over this time. If possible, it is valuable to conduct a follow up evaluation in the medium- or long-term after the program ends. For example, you could conduct a follow up evaluation with the participants one year after the program finishes to assess the longer term effects of the program for the participants. While it can be challenging to complete such longitudinal evaluation e.
If it is possible to also consider a comparison group, you can determine if any changes in your main group are also occuring in the community outside of your ASC project. The type of evaluation that your project will adopt largely depends on the length of your ASC project and the resources available to you. Qualitative evaluations seek to explore issues, contexts, interactions, experiences, processes and reasons for outcomes.
Qualitative evaluations are better suited for ASC projects that seek to explore the experience of participants , rather than have predefined outcomes. The value of qualitative research lies in understanding rather than measuring differences. These approaches tend to view the world not as one truth, but believe in multiple realities and in socially constructed meanings.
Ethnographic approaches seek to understand the social world of a specific culture being studied through immersion in the community to produce detailed descriptions of the culture and beliefs. Ethnography is guided by the assumption a specific culture guides members to view the world and guide their experiences in a specific way, and the researcher aims to learn from the members of the cultural group.
In the context of ASC evaluation, ethnography investigates the experience of a specific group for example, street-youth engaged in an ASC activity. The techniques used to obtain this information include in-depth interviews , review of records or charts, observations and other types of evidence, and may also include arts-based methods.
Types of Ethnographic Evaluation. Performance ethnography uses performance to generate data for the ethnographic study. The program offers a Master of Science M. For more information about faculty, admissions, courses, policies, and research opportunities, please visit the program website. Most of the educational and social science research that takes place today relies on the expertise of those who develop data collection instruments such as assessments, questionnaires, and interview protocols, plan research and evaluation studies, design sampling frameworks, collect and analyze data, and develop new statistical models and methods.
Observations may help explain behaviors as well as social context and meanings because the evaluator sees what is actually happening. Interviews may be conducted with individuals alone or with groups of people and are especially useful for exploring complex issues. Interviews may be structured and conducted under controlled conditions, or they may be conducted with a loose set of questions asked in an open-ended manner.
It may be helpful to tape-record interviews, with appropriate permissions, to facilitate the analysis of themes or content. Some interviews have a specific focus, such as a critical incident that an individual recalls and describes in detail. Focus groups are run by a facilitator who leads a discussion among a group of people who have been chosen because they have specific characteristics e.
Focus group participants discuss their ideas and insights in response to open-ended questions from the facilitator. The strength of this method is that group discussion can provide ideas and stimulate memories with topics cascading as discussion occurs Krueger et al. Doctoral degree studies include advanced graduate coursework, a research apprenticeship, a Ph. Candidacy Examination, and the completion of a doctoral dissertation that represents an independent and significant contribution to knowledge.
The research apprenticeship provides students with an opportunity to collaborate with a faculty sponsor on an ongoing basis and to participate in field research leading to a dissertation. For information about courses and requirements, visit the Quantitative Methods Ph. Eric T. Bradlow K.
Graduates go on to careers as university professors, researchers and psyshometricians for government agencies, foundations, nonprofits organizations, and corporations.
Please visit our Admissions and Financial Aid pages for specific information on the application requirements , as well as information on tuition, fees, financial aid, scholarships, and fellowships. Christine P. Lee Program Manager cplee gse. Please view information from our Admissions and Financial Aid Office for specific information on the cost of this program. All Ph.
0コメント