The Impact of Regional Differences in Italy on Operationalizing PISA Scores in Education Policy

The Impact of Regional Differences in Italy on Operationalizing PISA Scores in Education Policy

The political climate of post-WWII Italy has traditionally been marked by constant instability, a trait which extends to its education system. Between 2006 – 2016, Italy had five Ministers of Education, two of whom had terms which lasted less than two years (Damiani 532). This constant turnover leads to political deadlock and inefficient policy implementation. For example, a 1974 law decreeing that all teachers must have a university education was not supported by offering education courses at the post-secondary level until 1996 (533). Italy has consistently performed at or below the OECD rankings in literacy, mathematics, and science. In 1999, the government created the INVALSI national testing system to bring Italian students in line with OECD standards (Engel and Rutowski 775) based on PISA test results. However, the relationship to date between Italy and PISA has been difficult to discern. Damiani (535) argues that the evidence of PISA’s effect on education policy in Italy can only be indirectly observed, as there is not enough direct evidence to suggest a correlation; although national tests in Italy use language similar to that found in PISA, education policy itself does not reference PISA. Engel and Rutowski (778) suggest that the OECD heavily influenced Italian policymakers to develop a national assessment system, colloquially known as ValSIS, to rectify the country’s traditionally low PISA scores. 

While multiple reports (Damiani 532) have shown that low PISA scores can be associated with the overall low level of investment, poorly paid teachers, and weak monitoring of the Italian system, the literature suggests that regionalism is a larger issue impacting on quality of education. This is particularly evident in the differences between the Northern and Southern regions (Damiani 538; Giambona and Porcu 76). Considering the historical weakness of Italian policymaking, the unclear relationship between PISA results and educational policy in Italy, and the significant differences that exist on an regional level, Italian policymakers would be advised not to use the national testing of PISA to base future policies on.

This argument is supported by world systems theory, which posits the existence of a global education reform movement. Smith (3) cites evidence showing that “between the years 1995 and 2006 the number of countries worldwide that participate in an annual testing program more than doubled from 28 to 67”, and that as of 2006, 81% of developed countries have conducted at least one national test, as well as over half of all developed countries. This global reform is led by the World Bank, UNESCO, and the OECD; the latter in particular using its PISA testing to define the previously nebulous concept of “educational quality”. Although the OECD recognizes that PISA is not the only indicator of educational quality, many countries, including Italy, interpret PISA scores to mean just that (Engel and Rutowski 779-780).

               Italian policymakers should not use PISA data to inform policy because the strong regional differences in the country make the results from a national test, like PISA, not as relevant. For example, Giambona & Porcu conduct a study using PISA data on the relationship between class size in Italy and student success. Realizing that a linear regression model would produce biased samples due to the homogenous clusters in PISA, they instead use a multilevel random intercept model utilizing PISA data gathered every three years from 2000 to 2015. Data was collected from 31,073 secondary students clustered in 1,194 schools and examined five Plausible Variables (PV) for three competencies on PISA, as well as student characteristics – such as economic status and home access to educational resources – and regional dummies for territorial differences. (68) The dependant variables are the PV scores in mathematics, science, and reading. The authors hypothesize that smaller class sizes, represented by the proxy of teacher-student ratio, will result in better PV scores. 

        At first, their results indicate that student achievement actually increases with increased school size, perhaps due to greater funding allocated to larger schools (72). However, after considering regional sub-groupings of students, the authors found that there is an inverse u-shape relationship between class size and success, suggesting an optimal range. They note that “the relationship is different considering the territorial area: positive (with decreasing increases) in Centre-Northern area and negative (with increasing decreases) in the Southern one” (76). Giambona & Porcu had originally hypothesized that smaller class sizes would result in better quality (67), but the strong variance across regions suggests that hypothesis to not apply in all regions equally. They do not indicate which point in the inverse u-shape is the optimal size range but suggest that policymakers should conduct further study to determine this for themselves (75).

              

Another area in which PISA data is insufficient to inform Italian policy is in the debate between public versus private education. Checchi and Verzillo note that in Italy, the public education system is not only much more popular than the private system, but outperforms it in PISA testing (127). However, Italy has a two-tiered private system: non-confessional, i.e. secular, and confessional, i.e. Catholic private schools, as opposed to the secular public system (128). Their data comes from PISA scores from 2009, 2012, and 2015, with a representative sample of 302 public schools and 18 private schools, both secular and Catholic. (132) Checchi and Verzillo find that secular private schools, on average, score 29 points lower in math and 21 points lower in reading than public schools, with no statistically significant difference between Catholic private schools and public schools. However, these findings are only consistent with the 2009 PISA scores – not those in 2012 or 2015, which had a much smaller sample size (133). The authors note that this evidence is only descriptive and should not be considered causal, due to the inherent bias from sorting students into schools; therefore, they conduct another study using an instrumental variable, or IV, approach. This is uncorrelated with test scores but is instead correlated with reasons for selecting private schools (138). 

One of their dummy variables includes schools located in villages (pop. < 30,000), towns (pop. 3,000 – 15,000), cities (15,000 – 100,000) or large cities (> 100,000), noting that living in a large city puts students at higher odds of attending either a Catholic or secular private school. Considering these variables and using the 2012 PISA scores, the authors find that Catholic private schools are indistinguishable from secular public schools vis-à-vis­ student performance; using the 2015 PISA scores, the Catholic schools performed better than the public schools in science and reading, and the secular private schools had essentially the same results as the public schools. (142) Checchi and Verzillo conclude that the geographic location of a student, as well as the available family resources, have an effect on which type of schooling they attend, and is therefore connected to the quality of their education. One can extrapolate that the more economically prosperous Northern regions would have more large cities, and thus more students with access to better education, while the more economically disadvantaged Southern regions less access and thus a poorer quality of education. However, the sample size of this study is quite small which calls into question the validity of the authors’ findings. Considering the small sample size of this study, and the inconsistent results between years of PISA analyses, it is unlikely to settle the public-private debate.

               A larger issue with PISA data is brought up by Komatsu and Rappleye. They point out the seminal study from Hanushek and Woessmann which, based on their findings, extrapolated that a 3.5% GDP increase in education leads to 0.5 standard deviation points of improvement in international testing, which then translates to an increase of 5% in the GDP – effectively making investment in education something that pays for itself (169). While this has been the causal claim that underpins the rise in global testing, Komatsu and Rappleye seek to refute it by suggesting that the relationship between test scores and economic growth should be based on the explanatory power of test scores, not statistical significance [p], since if the sample size is large enough, any relationship can be considered quote-unquote “statistically significant” (173). They use the same data from the original Hanushek and Woessmann study: international test scores in math and science [T] from 1964 – 2003 combined with the mean annual GDP growth rate per capita [G] recorded between 1960-2000 for 50 countries. However, they also include GDP per capita [Y] for 2010 – 2014, which the original study did not account for. (175) Further, they changed the calculating period from 1960 – 200 to 1980 – 2000, 1985 – 2005, 1990 – 2010, and 1995 – 2014. 

     They found that the relationship between T and G was strong for the first reporting period of 1980 – 2000, but became weaker with each subsequent reporting period (176). In Italy, during the first reporting period, the T value rose by 20% and the GDP rose by 0.586, but declined after that by -1.08 in 1990 – 2010, and -1.46 in 1995 – 2014 (177). While the relationship between the T and G values are still statistically significant at 5%, the explanatory power R² value shows the relationship declining over time. This discrepancy suggests that the concept of test scores being linked to economic growth could be faulty, thereby undercutting the purpose of using PISA as a benchmark to determine educational policy.

               One counterargument to these claims is found in Ramirez, et al.  They analyze the relationship between the total number of international standardized tests, such as PISA, that a nation has completed over the past 10 years, and the quality of educational outcomes in that country. As a proxy for “quality”, they use secondary and tertiary education enrollment, evidence of progressivism in pedagogical content, female participation in schooling, and primary and secondary school repeat rates. Their data range extends from 1970 – 2012 and uses a panel regression model, while controlling for GDP (350). While Ramirez, et al. do not find any negative corollary between number of international tests and educational quality, their evidence for a positive corollary, however, is not strong. They acknowledge that their findings cannot equate a causal relationship between national and international testing and positive outcomes; further, they also acknowledge that their samples cannot suggest inferences about specific cases where the number of international tests directly influences any of their dependent variables (358).

Any policy recommendation that references PISA and the Italian education system needs to take into account the complexities inherent in both. Singer and Braun comment that with international tests like PISA, it is difficult to know “whether the observed variation in student achievement across countries is attributable to any policy or set of policies […] most methodologists agree that ILSAs are ill-equipped to provide unambiguous answers.” (79-80) An example of this can be surmised from the recommendations of Checchi and Verzillo. The findings of their study on secular and religious private schools compared to secular public schools are, as they admit, difficult to assess; while private schools reduce costs to local municipalities, the self-sorting of students into different groups runs counter to positive socialization practices. (142) Thus, they essentially have no recommendations. The Italian education system clearly needs guidance; students regularly score below the OECD average in reading and science, and scores have been declining since 2012 (OECD 1) Of note are the distinct regional differences between Northern and Southern regions. The regions of Trento and Bolzano, both in the North, scored above the national average in reading, science, and math; additionally, fewer than 40% of students reported that they skipped one or more days of school in a week. In the Southern region of Sardegna, however, the scores were all below the national average and 67% of students reported skipping once or more in a given week (2-3). 
Therefore, rather than relying on national or international tests such as PISA from which to gather data, the government should focus on empowering regional offices to pursue pedagogy, teacher training, and assessment tools that are relevant to those regions. Data should be collected from assessments that take into consideration the historical, cultural, linguistic, and socio-economic disparities between North and South, as opposed to generalized testing that does not reflect this reality. This would allow targeted policies for specific regions that have specific needs, as opposed to blanket policies that are too general to be effective where the needs are greatest, such as in the impoverished South. Italian policymakers should disentangle themselves from PISA results and move towards this model to truly represent their citizens.

                                                             WORKS CITED

Braun, Henry I. and Singer, Judith D. (2019) Assessment for Monitoring of Education Systems:

International Comparisons. The Annals of the American Academy, AAPSS, no. 683, 75 – 92

Checchi, Daniele and Verzillo, Stefano. (2017) “The Role of PISA in Regional and Private/Public Debates

in Italy.” The PISA Effect on Global Educational Governance, 1^st Edition. Ed. Louis Volante. Routledge: 127-147.

Damiani, Valeria. (2016) Large-scale assessments and educational policies in Italy. Research Papers in

Education, vol. 31, no. 5, 529-541.

Engel, Laura C. and Rutowski, David. (2014) Global Influences on National Definitions of Quality

Education: examples from Spain and Italy. Policy Futures in Education, vol. 12, no. 6, 769-783.

Giambona, Francesca and Porcu, Mariano. (2018) School size and students’ achievement. Empirical

evidences from PISA survey data. Socio-Economic Planning Sciences, vol. 64, 66-77.

OECD (2016), PISA 2018 Results (Volume I): Excellence and Equity in Education, PISA, OECD Publishing,

               Paris, https://doi.org/10.1787/9789264266490-en.

Ramirez, F., Schoefer, E., & Meyer, John. (2018) International Tests, National Assessments,

and Educational Development (1970-2012). Cooperative Education Review, vol. 62, no.

3, 344-364.

Smith, William C. (2014) The Global Transformation Toward Testing for Accountability.

Education Policy Analysis Archives/Archivos Analíticos de Políticas Educativas, vol. 22, 1-34