Thursday, December 26, 2019

Students Transitioning from Section 23 To Community-Based School Programs

Final Evaluation Design:

Students Transitioning from Section 23 To Community-Based School Programs


Background
    
The Ontario Education Act enshrines the right of youth to have access to an education, regardless of whether or not they are physically present in a community-based school. This type of programming include homeschooling, but more often refers to in-risk students whose needs cannot be met by a traditional school setting. These specific programs are referred to as Children and Youth In Care (CYIC), or Care, Treatment, Custody and Corrections (CTCC) programs. The common umbrella term for these programs is Section 23, which is the aforementioned section of the Education Act. 

A 2016-2017 review found that in Toronto, there are 63 different Section 23 programs with 734 spaces, generally operating at 80% capacity (East Metro Youth Services, 2017). These programs, whether providing short- or long-term care, aim to reintroduce their students into community-based schools whenever possible. Part of this process involves creating transition planning that focuses on gradual reintegration, as well as providing information and support to educational staff at both ends of the transition (Ministry of Education, 2016). 

Research shows that these transitions are not always effective. Youth drug treatment programs, for example, find that 47% of their students resume full drug use within one year of their transition, and that the transition time is when they are at highest risk for relapse (Finch, Lawton-Krupp, and Moberg, 2014). Children’s aid programs found that students considered the transitions “jarring” and did not make connections with their community school, especially those transferred to a new school boards (Directions, 2017). Regardless of the type of programming, many transitioning students felt hindered in their progress by stigma and a lack of preparation for post-secondary education. (Directions, 2017)

Situation
       At any given time in Toronto, there are hundreds of children and youth who are transitioning between community-based schools and Section 23 programs. These transitions vary in terms of length, rationale, number of instances, quality of supports offered, and rate of success. For a variety of reasons, including the high turnover rate of CYIC programs, there is little data available to indicate how successful these transitions are in both the short- and long-term. 

Policy Problem
       There are several reasons, at the institutional level, that could contribute to poor transitioning for students in Section 23 programs. One is that there is a lack of research that specifically focuses on children and youth in care and treatment programs. The result is that policy for these children is often designed around research for in-risk students in general, as opposed to in-risk students who cannot be housed in a community-based school (Directions, 2017). Another potential reason is the perceived lack of communication between the different agencies involved in programming for these students. The program goals of the various ministries involved are not aligned, and there is significant ambiguity around who is responsible for the implementation and success of these programs. (East Metro Youth Services, 2017) Finally, there are limited resources available for both the sending and receiving programs. Staff in community-based schools feel they are not equipped to handle the needs of these transitioning students (Directions, 2017), which can lead to the students being kept longer in the Section 23 program not for any discernible need, but simply because the receiving school needs more time to prepare. (East Metro Youth Services, 2017) This puts an additional strain on the Section 23 system and, more importantly, the in-risk student.

Literature Review
       Leonard and Gudino (2016) hypothesize that students who move schools frequently - in this population sample, due to being involved in foster home care - experience social stress which can negatively impact their academic success and social networks (Leonard and Gudino, 2016). Of the 420 students in the research sample, taken from the National Survey of Child and Adolescent Well Being (NSCAW), 72% had been in at least 2 different schools during the 36 month study period; 46% had been in a minimum of 3 different schools (Leonard and Gudino, 2016). Interestingly, the authors found that there was no statistically significant correlation between home instability and school engagement with these students, findings which Leonard and Gudino note do not match the existing literature around the subject (Leonard and Gudino, 2016). Another possibility is that the act of removal from a home is not necessarily a negative factor if that home is a source of trauma for the student. The authors note that “if youth removed from home can have the benefit of remaining with familiar teachers, established social networks, and familiar settings, children may have the [ability] to have their emotional needs met.” (Leonard and Gudino, 2016) This study shows that moving between programs is not necessarily inherently detrimental to student academic success and mental health, but reinforces the idea that the transition must be managed carefully.

       Blake, et al. (2016), examine the Bridge for Resilient Youth in Transition (BRYT) program model at Brookline High in Brookline, Mass. In essence, this program provides a school-within-a-school transition for students coming in to or out of school due to not only mental health issues, but also concussions or any other serious health issue requiring hospitalization. (Blake, et al., 2016) The authors found 6 similar programs across the state. Collectively, three-quarters of the students enrolled in these BRYT-model programs were concurrently in out-patient hospital programs, while 73% had at one point been fully or partially hospitalized for psychiatric concerns. The expectation for students in the BRYT-model programs was that their transition period would last between 8-12 weeks; the median stay in the program was 10 weeks. (Blake, et al., 2016) The study found that these programs were highly successful. Student participation in the program began on average at 75% in the first week of admission but was down to 17% in the final week. The authors frame this as a positive, as it indicates students relying less on the program and being able to integrate themselves back into the school over time. (8) This would, however, as Blake, et al. state, require a team of teachers-slash-mental health practitioners who have not only clinical expertise “but also a caseload and schedule that allows them the flexibility and time necessary to help students navigate the academic, emotional, and social challenges they will inevitably face” with transitioning back to school. (2016)

       Simone (2017) provides a qualitative counterpoint to the more quantitative Leonard and Gudino article by interviewing multiple students who were housed in psychiatric facilities, then transferred to BRYT-style programs across Massachusetts. These interviewees, ranging in age from 15 to 19, each identified key issues vis-a-vis reentry: concern over the perception of their peers during their attempt to reintegrate, and their worry that they would be unable to catch up with their courses. (Simone, 2017) Interestingly, these students all reported that having intake meetings with hospital and school staff helped ease these stresses. Completing their school work gave them a positive sense of accomplishment. This was facilitated through a variety of means, including: smaller class settings in the BRYT program; teachers giving accommodations or modifications to the late assignments, or simply removing them; and a continual focus from staff on reintegration, not staying permanently in BRYT. Simone also notes that these students had a better rapport with their teachers once the cause of their “poor behaviour” had been demystified. In line with this is the fact that although these students continued to struggle with mental health issues, they eventually saw school as a place where they could practice coping mechanisms and management strategies, rather than a place to be avoided at all costs. (Simone, 2017)


Lens - Sense Making & Co-Construction 
(Datnow and Park, 2009)
Foregrounding
       When viewing this issue through the sense making and co-construction lenses, the issues that are foregrounded are whether or not a student can successfully transition back into the school community, and the ability of the staff and receiving program to facilitate this transition. Another issue that is foregrounded is how individual schools navigate the return of individual students; each will have their own strengths, weaknesses, and needs that must be addressed on a case-by-case basis.

Backgrounding
       What is least important in this lens is the role of the prescribed curriculum in meeting the needs of transitioning students. The pressures on both staff and students of meeting the curriculum requirements can be more of a hindrance than a help in the case of students with mental health issues.
In The Frame / Out Of The Frame
       With this policy problem, what is “in the frame” is the attempts of the school program to successfully facilitate a transition, and its ability to adapt to changing situations (the student relapsing, changes of medication, etc.) What is “out of the frame” are the decisions made by medical staff in the care and treatment programs - medication, therapeutic techniques, etc. However the sense making and co-construction lens suggests that these hospital contexts need to be considered as well, regardless of their place “higher” or “lower” on the policy chain.

Arena of Action
       This lens indicates that top-down policy is not the only motivating factor in organizational behaviour, and that different interpretations of implementation can occur in different settings. Thus, the arena of action is focused on how individual school programs can design their own policies, based on staff experience, on a school-by-school basis, to best meet the myriad needs of this fragile student population.

Locus of Control
       The locus of control ultimately rests with the school administration, who work with hospital staff to determine the pace and schedule of the students’ transition back to school.

Potential Policy Solution
       One possible policy solution to this issue is to provide funding, resources, training, and staffing to schools specifically for the purpose of creating classrooms devoted solely to transitioning students back from extended hospital stays, particularly for psychiatric hospitalization, into the mainstream school community.

Who has the authority to make changes?
       The authority to provide additional funding lies with the Ministry of Education. Individual school boards would then identify which schools in their area would best put this funding to use based on the needs of their student population. At the school level, administrators would be responsible for staffing the programs, and classroom teachers would be responsible for delivering the programming to support the transitioning students.

Who needs to know about this?
       It could be argued that the Ministry of Education, the Ministry of Children and Youth, and the Ministry of Health and Long-Term Care all have a vested interest in solving this issue; a successful implementation of a solution would reduce the number of youth moving multiple times between hospitals and schools, which reduces the strain on those services. School boards would also need to know, as mental health is an increasingly pressing need in school communities across Ontario which teaching staff find themselves under-equipped to deal with. Parent councils at individual schools may also be a potential audience, particularly if their school has a high rate of recidivism amongst transitioning students; the mental health and successful transition of a student is something that reverberates well outside the classroom, into the home and the hospital program as well. Finally, Section 23 programs and their ilk would need to know as they are at the front lines of balancing the educational and medical needs of students.

Conversations to Consider

       Mental health is no longer the stigmatized subject it once was, although there is still work to be done in this area. It is not a “third rail” issue that any particular political party or level of government would want to avoid; indeed, politicians from across the political spectrum now only speak about their own struggles with poor mental health. Thus, regardless of whichever party is in power or whatever their goals are, this is a policy issue that can gain traction. In a climate of austerity the idea of putting more funding into education, for something that benefits an often-hidden cohort of students, may not be palatable, however, without some societal pressure.

       It is also important to consider that multiple agencies and ministries could potentially be involved in crafting this policy as it cuts across a variety of sectors - namely, education and health care. Recognizing the needs, expectations, and demands of those separate actors would be critical in ensuring a successful implementation of this policy. Teacher unions should also be consulted to ensure that the work asked of teachers, and the conditions of the transition classrooms in terms of number of students, hours worked, personal safety, etc., do not violate the terms of their collective agreements.


Next Steps
       While this policy is geared towards providing assistance for students and staff at the secondary level, there is a considerable amount of work to be done in this area at the post-secondary level as well. A 2017 MacLean’s article found that 15 universities across Canada - including Queen’s, Ryerson, and York - reported students feeling “overwhelmed” between 50 - 60% of the time on a daily or weekly basis. (Hutchins, 2017) The University of Toronto recently found itself embroiled in controversy in June 2018 when it enacted a policy that would allow them to place students with mental health issues on mandatory leave if the school deems it necessary. This policy was met with criticism from multiple student groups, as well as the Ontario Human Rights Commission. (CBC, 2018) Although post-secondary institutions increased their mental health budgets by 35% in 2017, (Bowden, 2018) it is increasingly students who are taking the lead in meeting these needs. One example is Waterloo student Tina Chan, who created kits to help students experiencing panic attacks. The university purchased 7,100 of her kits, which were given to first-year students in the fall of 2018. (Bowden, 2018) This suggests that, at the post-secondary level, a variety of policy approaches ranging from mandates to capacity-building will be necessary to ensure students struggling with these issues can return to their studies in a supportive, structured environment. 


REFERENCES

Bowden, Olivia. (2018, September 3rd). "Students push for mental health support." The Hamilton Spectator. Retrieved from https://www.thespec.com/living-story/8866484-students-push-for-mental-health-support/

Directions Evidence and Policy Research Group, LLP (2017). Evaluation of the Innovative Programs for Students in the Care of, or Receiving Services from, Children’s Aid Societies: Final Report. Toronto, ON.

East Metro Youth Services (2017). Section 23 Working Group Summary: Final Report. Toronto, ON.

Finch, Lawton-Krupp, and Moberg (2014). Continuing Care in High Schools: A Descriptive Study of Recovery High School Programs. Journal of Child & Adolescent Substance Abuse, Vol. 23, 116-129.

Hutchins, Aaron. (2017, April 20th). Are universities doing enough to support mental health?MacLean’s Magazine. Retrieved from https://www.macleans.ca/education/depth-of-despair/

Leonard, Skyler S. and Gudino, Omar G. (2016) Academic and Mental Health Outcomes of Youth Placed in Out-of-Home Care: The Role of School Stability and Engagement. Child Youth Care Forum. New York. 45:807-827.

Ontario Ministry of Education. (2016-17) Guidelines For Educational Programs for Students in Government Approved Care and/or Treatment, Custody and Correctional (CTCC) Facilities. Ottawa, ON: Queen’s Printer for Ontario.

Simone, Daniel J. (2017) “Getting Back To School - Understanding Adolescents’ Experience Of Reentry Into School After Psychiatric Hospitalization”. Doctoral thesis. 50-93. Retrieved from ERIC.

The Canadian Press. (2018, June 27th). U of T approves policy that could place students with mental health issues on leave. CBC News. Retrieved from https://www.cbc.ca/news/canada/toronto/university-toronto-mental-health-mandatory-leave-1.4725104

White H, LaFleur J, Houle K, Hyry-Dermith P, Blake SM. (2017) Evaluation of a school-based transition program designed to facilitate school reentry following a mental health crisis or
psychiatric hospitalization. Psychol Schs. 54:868–882.

Tuesday, December 17, 2019

The Impact of Regional Differences in Italy on Operationalizing PISA Scores in Education Policy



The Impact of Regional Differences in Italy on Operationalizing PISA Scores in Education Policy


The political climate of post-WWII Italy has traditionally been marked by constant instability, a trait which extends to its education system. Between 2006 – 2016, Italy had five Ministers of Education, two of whom had terms which lasted less than two years (Damiani 532). This constant turnover leads to political deadlock and inefficient policy implementation. For example, a 1974 law decreeing that all teachers must have a university education was not supported by offering education courses at the post-secondary level until 1996 (533). Italy has consistently performed at or below the OECD rankings in literacy, mathematics, and science. In 1999, the government created the INVALSI national testing system to bring Italian students in line with OECD standards (Engel and Rutowski 775) based on PISA test results. However, the relationship to date between Italy and PISA has been difficult to discern. Damiani (535) argues that the evidence of PISA’s effect on education policy in Italy can only be indirectly observed, as there is not enough direct evidence to suggest a correlation; although national tests in Italy use language similar to that found in PISA, education policy itself does not reference PISA. Engel and Rutowski (778) suggest that the OECD heavily influenced Italian policymakers to develop a national assessment system, colloquially known as ValSIS, to rectify the country’s traditionally low PISA scores. 

While multiple reports (Damiani 532) have shown that low PISA scores can be associated with the overall low level of investment, poorly paid teachers, and weak monitoring of the Italian system, the literature suggests that regionalism is a larger issue impacting on quality of education. This is particularly evident in the differences between the Northern and Southern regions (Damiani 538; Giambona and Porcu 76). Considering the historical weakness of Italian policymaking, the unclear relationship between PISA results and educational policy in Italy, and the significant differences that exist on an regional level, Italian policymakers would be advised not to use the national testing of PISA to base future policies on.

This argument is supported by world systems theory, which posits the existence of a global education reform movement. Smith (3) cites evidence showing that “between the years 1995 and 2006 the number of countries worldwide that participate in an annual testing program more than doubled from 28 to 67”, and that as of 2006, 81% of developed countries have conducted at least one national test, as well as over half of all developed countries. This global reform is led by the World Bank, UNESCO, and the OECD; the latter in particular using its PISA testing to define the previously nebulous concept of “educational quality”. Although the OECD recognizes that PISA is not the only indicator of educational quality, many countries, including Italy, interpret PISA scores to mean just that (Engel and Rutowski 779-780).

               Italian policymakers should not use PISA data to inform policy because the strong regional differences in the country make the results from a national test, like PISA, not as relevant. For example, Giambona & Porcu conduct a study using PISA data on the relationship between class size in Italy and student success. Realizing that a linear regression model would produce biased samples due to the homogenous clusters in PISA, they instead use a multilevel random intercept model utilizing PISA data gathered every three years from 2000 to 2015. Data was collected from 31,073 secondary students clustered in 1,194 schools and examined five Plausible Variables (PV) for three competencies on PISA, as well as student characteristics – such as economic status and home access to educational resources – and regional dummies for territorial differences. (68) The dependant variables are the PV scores in mathematics, science, and reading. The authors hypothesize that smaller class sizes, represented by the proxy of teacher-student ratio, will result in better PV scores. 

        At first, their results indicate that student achievement actually increases with increased school size, perhaps due to greater funding allocated to larger schools (72). However, after considering regional sub-groupings of students, the authors found that there is an inverse u-shape relationship between class size and success, suggesting an optimal range. They note that “the relationship is different considering the territorial area: positive (with decreasing increases) in Centre-Northern area and negative (with increasing decreases) in the Southern one” (76). Giambona & Porcu had originally hypothesized that smaller class sizes would result in better quality (67), but the strong variance across regions suggests that hypothesis to not apply in all regions equally. They do not indicate which point in the inverse u-shape is the optimal size range but suggest that policymakers should conduct further study to determine this for themselves (75).
              
Another area in which PISA data is insufficient to inform Italian policy is in the debate between public versus private education. Checchi and Verzillo note that in Italy, the public education system is not only much more popular than the private system, but outperforms it in PISA testing (127). However, Italy has a two-tiered private system: non-confessional, i.e. secular, and confessional, i.e. Catholic private schools, as opposed to the secular public system (128). Their data comes from PISA scores from 2009, 2012, and 2015, with a representative sample of 302 public schools and 18 private schools, both secular and Catholic. (132) Checchi and Verzillo find that secular private schools, on average, score 29 points lower in math and 21 points lower in reading than public schools, with no statistically significant difference between Catholic private schools and public schools. However, these findings are only consistent with the 2009 PISA scores – not those in 2012 or 2015, which had a much smaller sample size (133). The authors note that this evidence is only descriptive and should not be considered causal, due to the inherent bias from sorting students into schools; therefore, they conduct another study using an instrumental variable, or IV, approach. This is uncorrelated with test scores but is instead correlated with reasons for selecting private schools (138). 

One of their dummy variables includes schools located in villages (pop. < 30,000), towns (pop. 3,000 – 15,000), cities (15,000 – 100,000) or large cities (> 100,000), noting that living in a large city puts students at higher odds of attending either a Catholic or secular private school. Considering these variables and using the 2012 PISA scores, the authors find that Catholic private schools are indistinguishable from secular public schools vis-à-vis­ student performance; using the 2015 PISA scores, the Catholic schools performed better than the public schools in science and reading, and the secular private schools had essentially the same results as the public schools. (142) Checchi and Verzillo conclude that the geographic location of a student, as well as the available family resources, have an effect on which type of schooling they attend, and is therefore connected to the quality of their education. One can extrapolate that the more economically prosperous Northern regions would have more large cities, and thus more students with access to better education, while the more economically disadvantaged Southern regions less access and thus a poorer quality of education. However, the sample size of this study is quite small which calls into question the validity of the authors’ findings. Considering the small sample size of this study, and the inconsistent results between years of PISA analyses, it is unlikely to settle the public-private debate.
               A larger issue with PISA data is brought up by Komatsu and Rappleye. They point out the seminal study from Hanushek and Woessmann which, based on their findings, extrapolated that a 3.5% GDP increase in education leads to 0.5 standard deviation points of improvement in international testing, which then translates to an increase of 5% in the GDP – effectively making investment in education something that pays for itself (169). While this has been the causal claim that underpins the rise in global testing, Komatsu and Rappleye seek to refute it by suggesting that the relationship between test scores and economic growth should be based on the explanatory power of test scores, not statistical significance [p], since if the sample size is large enough, any relationship can be considered quote-unquote “statistically significant” (173). They use the same data from the original Hanushek and Woessmann study: international test scores in math and science [T] from 1964 – 2003 combined with the mean annual GDP growth rate per capita [G] recorded between 1960-2000 for 50 countries. However, they also include GDP per capita [Y] for 2010 – 2014, which the original study did not account for. (175) Further, they changed the calculating period from 1960 – 200 to 1980 – 2000, 1985 – 2005, 1990 – 2010, and 1995 – 2014. 

     They found that the relationship between T and G was strong for the first reporting period of 1980 – 2000, but became weaker with each subsequent reporting period (176). In Italy, during the first reporting period, the T value rose by 20% and the GDP rose by 0.586, but declined after that by -1.08 in 1990 – 2010, and -1.46 in 1995 – 2014 (177). While the relationship between the T and G values are still statistically significant at 5%, the explanatory power R2 value shows the relationship declining over time. This discrepancy suggests that the concept of test scores being linked to economic growth could be faulty, thereby undercutting the purpose of using PISA as a benchmark to determine educational policy.
               One counterargument to these claims is found in Ramirez, et al.  They analyze the relationship between the total number of international standardized tests, such as PISA, that a nation has completed over the past 10 years, and the quality of educational outcomes in that country. As a proxy for “quality”, they use secondary and tertiary education enrollment, evidence of progressivism in pedagogical content, female participation in schooling, and primary and secondary school repeat rates. Their data range extends from 1970 – 2012 and uses a panel regression model, while controlling for GDP (350). While Ramirez, et al. do not find any negative corollary between number of international tests and educational quality, their evidence for a positive corollary, however, is not strong. They acknowledge that their findings cannot equate a causal relationship between national and international testing and positive outcomes; further, they also acknowledge that their samples cannot suggest inferences about specific cases where the number of international tests directly influences any of their dependent variables (358).


Any policy recommendation that references PISA and the Italian education system needs to take into account the complexities inherent in both. Singer and Braun comment that with international tests like PISA, it is difficult to know “whether the observed variation in student achievement across countries is attributable to any policy or set of policies […] most methodologists agree that ILSAs are ill-equipped to provide unambiguous answers.” (79-80) An example of this can be surmised from the recommendations of Checchi and Verzillo. The findings of their study on secular and religious private schools compared to secular public schools are, as they admit, difficult to assess; while private schools reduce costs to local municipalities, the self-sorting of students into different groups runs counter to positive socialization practices. (142) Thus, they essentially have no recommendations. The Italian education system clearly needs guidance; students regularly score below the OECD average in reading and science, and scores have been declining since 2012 (OECD 1) Of note are the distinct regional differences between Northern and Southern regions. The regions of Trento and Bolzano, both in the North, scored above the national average in reading, science, and math; additionally, fewer than 40% of students reported that they skipped one or more days of school in a week. In the Southern region of Sardegna, however, the scores were all below the national average and 67% of students reported skipping once or more in a given week (2-3). 
Therefore, rather than relying on national or international tests such as PISA from which to gather data, the government should focus on empowering regional offices to pursue pedagogy, teacher training, and assessment tools that are relevant to those regions. Data should be collected from assessments that take into consideration the historical, cultural, linguistic, and socio-economic disparities between North and South, as opposed to generalized testing that does not reflect this reality. This would allow targeted policies for specific regions that have specific needs, as opposed to blanket policies that are too general to be effective where the needs are greatest, such as in the impoverished South. Italian policymakers should disentangle themselves from PISA results and move towards this model to truly represent their citizens.
                                                             WORKS CITED

Braun, Henry I. and Singer, Judith D. (2019) Assessment for Monitoring of Education Systems:
International Comparisons. The Annals of the American Academy, AAPSS, no. 683, 75 – 92


Checchi, Daniele and Verzillo, Stefano. (2017) “The Role of PISA in Regional and Private/Public Debates
in Italy.” The PISA Effect on Global Educational Governance, 1st Edition. Ed. Louis Volante. Routledge: 127-147.


Damiani, Valeria. (2016) Large-scale assessments and educational policies in Italy. Research Papers in
Education, vol. 31, no. 5, 529-541.


Engel, Laura C. and Rutowski, David. (2014) Global Influences on National Definitions of Quality
Education: examples from Spain and Italy. Policy Futures in Education, vol. 12, no. 6, 769-783.


Giambona, Francesca and Porcu, Mariano. (2018) School size and students’ achievement. Empirical
evidences from PISA survey data. Socio-Economic Planning Sciences, vol. 64, 66-77.


OECD (2016), PISA 2018 Results (Volume I): Excellence and Equity in Education, PISA, OECD Publishing,
               Paris, https://doi.org/10.1787/9789264266490-en.


Ramirez, F., Schoefer, E., & Meyer, John. (2018) International Tests, National Assessments,
and Educational Development (1970-2012). Cooperative Education Review, vol. 62, no.
3, 344-364.


Smith, William C. (2014) The Global Transformation Toward Testing for Accountability.
Education Policy Analysis Archives/Archivos Analíticos de Políticas Educativas, vol. 22, 1-34




NFB: Film & Bullying

Originally published at NFB Education (February 2023) Chances are, you were bullied in school. Or maybe you bullied someone. Or, even more l...