Full paper for 10th IFIP World Conference on Computers in Education
WCCE 2013 Torun, 2–5 July 2013
"learning while we are connected"
The presentation is accessible here.
At the Hungarian Institute for Educational research and Development an R+D team developed an online self-evaluation asset for schools which also serves as a country-wide monitoring tool for assessing the educational use of ICT in compulsory education. It collects soft data in four fields, learning, teaching, operation and infrastructure; and concentrates on pedagogical issues with a whole school approach. 2427 schools have already registered in the system. In 2011 362, in 2012 and this year also 723 full self-evaluations have been done. There are 83 schools doing so in 2011 and 2012, no data yet on their number in 2013. We have already had three evaluation periods and are working on the data set of the third one. The yearly report puts the schools in four categories according to their level of ICT usage, the category names are based on the UNESCO/IFIP categories described in Approaches to ICT Development in Schools. The paper is to introduce the system itself and the data we have already collected, - in comparison with an European survey carried out in the autumn in 2011, called Survey of Schools: ICT in Education.
At the Hungarian Institute for Educational research and Development an R+D team developed an online self-evaluation asset for schools which also serves as a country-wide monitoring tool for assessing the use of ICT in compulsory education. The project was funded by the European Union within the framework of the 3.1.1. Social Renewal Operative Programme.
The tool collects soft data in four fields, learning, teaching, operation and infrastructure and concentrates on pedagogical issues with a whole school approach. There are 100 claims in the system and the school has to decide the extent to which they are true for them, using the four-point Likert scale, so they must decide if the statements are rather true or rather false in their case.
The schools enter their data once a year voluntarily, based on teacher and students questionnaires and their school documents as proofs. The system is able to produce the results in a textual evaluation format along with diagrams that show all the fourteen sub-areas of evaluation in comparison with the country results or the last year results of the school. The format is compatible with an ICT strategy template, where the results are used as the state of art. In the same format the system is also able to produce the country results.
2427 schools have already registered in the system: in 2011 362, in 2012 and 2013 723 full self-evaluations have been done. There are 83 schools doing so both the first years, we do not know yet, how many repeated the process in 2013. The tool is available for the schools all over the year but there is a campaign in each February, since at the end of February a country snapshot is done from all the data entered since the previous snapshot. The yearly report puts the schools in four categories according to their level of ICT usage, the category names are based on the UNESCO/IFIP categories in Approaches to ICT Development in Schools.
The paper aims at introducing the system (tool) itself and the philosophy behind, places the Hungarian solution into an international context. It refers to the English Self-evaluation Framework, the Microsoft Innovative Toolkit, the IFIP categories of school status regarding their level of development, also to the results of the EU/EUN survey on educational use of ICT (ESSIE, 2011-2012), with its final name as Survey of Schools: ICT in Education. It presents the general picture of the ICT usage in the Hungarian public education system, the strengths and weaknesses in general and highlights some of the best practice schools. The presentation ends with some plans for the future, like establishing a kind of reward system for the for-runners.
Reasoning and context
In many countries lots of money has been spent on introducing ICT into public education. There has been debate everywhere if the gorgeous spending was worth to be done and very few countries can boast that they have the proof for either side of the debate. Until we do not have the right question it is hard to find the answer. There are different expectations; it is popular to say that learning results must get better when using ICT in which I do not believe. I believe that the way of learning outside and inside school must get closer; I believe that young people get more able to answer the expectations of modern life if they are taught how to use ICT purposefully. I believe that ICT cannot do anything about better learning if the methods are not appropriate, if the teachers are not comfortable with using it and if they do not know why and how they should use it. ICT is not the magic weapon to create a brilliant knowledge society generation. Learning results cannot get better until ICT is not fully and professionally integrated into teaching and learning. And even after, we do not know.
In order to find the right questions, to help the schools find their ways, the politicians formulate their measures at the Hungarian Institute for Educational research and Development an R+D team designed and produced an online self-evaluation asset for schools. It also serves as a country-wide monitoring tool for assessing the use of ICT. The system is called eLEMÉR which means e-metre in the sense of measuring the use of digital device for school purposes. The concept was based on national and European policy documents, and on the review of the existing solutions of different organisations and countries, like the English Self-evaluation Framework by the former (BECTA 2009), now NAACE, the Microsoft Innovative Toolkit (Microsoft 2009), the IFIP categories of school status (UNESCO/IFIP 2009) regarding their level of development, also to the results of the EU/EUN survey on educational use of ICT named (STEPS 2009) and an Evaluation framework that is the result of a peer learning activity, lead by the European Schoolnet (EFW 2009), also a Handbook by the World Bank (InfoDev 2005).
The eLEMÉR self evaluation system
The tool and the project have a website at http://ikt.ofi.hu/english where some information is also available in English. Visitors can try the self-evaluation tool without registering, they can use the system even for self-evaluation but the results must be printed immediately since the next user overwrites the data. eLEMÉR collects soft data in four fields, learning, teaching, operation and infrastructure and concentrates on pedagogical issues with a whole school approach. Each field consists of 3 or 4 subcategories, and there are 100 claims in the system. The school has to decide the extent to which the claims are true for them. A four-point Likert-type evaluation scale is used so that they must declare if they are below or above the so-called average or medium level regarding each claim.
The claims are also for orientation, they can suggest directions and ideas for using ICT to develop the school, and they give hints for building the school’s ICT strategy. There is a review process each year, some of the questions are modified or changed in a way that the sub-category results still can be compared year by year on both school and country levels. Some examples of the claims:
- Learners regularly work on home assignments where the use of ICT tools is necessary.
- Learners are aware of what intellectual property is and what copyright rules are.
- Learners aim at producing nice design when using ICT tools.
- The school management set an example while using ICT tools for communication.
- Teachers use digital communication tools for participating in wider professional networks.
- Teachers offer possibilities for decision making and creative ICT use during classes.
- The school management support and motivate teachers to use ICT tools in the teaching and learning process.
- Digital resources, timetables, log books, e-registry books, attendance sheets and results are available for teachers, students and parents from their homes.
- At least 75% of the stuff has their voice in the process of the school ICT self-evaluation.
The methods of self evaluation and data collection
The schools enter their data once a year voluntarily, based on teacher and students questionnaires and school documents as proofs. They are asked to do the self-evaluation as a common task, not to transfer the job to the system administrator or the IT teacher. There are 3 ways suggested, like using the questionnaires provided for collecting the data from at least 75% of the staff and 50% of learners above the age of 10. They can use a data projector in the staff room and enter the data they have collectively agreed on. It means that only one questionnaire is filled by a single school, entering overall, agreed data. By the time of the recent year campaign, we developed the method of online personal questionnaires with an automatic calculation system that shows how many teachers and how many students have filled in the questionnaire and what culminated result belongs to a sub-category. The result is listed among the proofs and must be considered when choosing the right grades of the scale. The system is able to produce the results in a textual evaluation format along with diagrams that show all the fourteen sub-areas of evaluation in comparison with the country results or the last year results of the school. The format is compatible with an ICT strategy template, where the results are used as the state of art.
The tool is available for the schools all over the year but there is a campaign in each February, since at the end of February a country snapshot is done from all the data entered since the previous snapshot. It shows the tendencies and in case of schools that repeat the self-evaluation also compares the yearly results.
Results of the monitoring process
By now we have three sets of data and the tendencies of a three-year period. We can say that there is a positive tendency over the three years. All the claims got better average scores in 2013 than in 2011. Regarding the stages of development UNESCO/IFIP 2009) a positive shift has been observed. There are much less schools in the first category of schools where ICT is just ‘Emerging’ (from 51% to 30,2% and 27,7% in 2013). In 2012 there was significant growth in the second and third ones where the schools are Applying or already ‘Integrating’ ICT into their work, but in 2013 these categories remained almost untouched. Transforming with ICT shows a significant tendency of growing, still well under 10% (figure 1).
Figure 1: Shift in development categories (%)
The strongest area is always the infrastructure, neither teaching nor learning, nor does the operation exploit the given ICT infrastructure at its full potential. The organisational ICT usage itself scores the lowest, since in Hungary there are very few schools that apply any solutions for outside access either for the staff or the students, families (figure 2). Schools are not aware as organisation how they can better organise their work with the help of ICT and they are not conscious enough when planning. (The scores can range from 1.00 to 4.00.)
Figure 2: Changes in the four fields of measurement
Relation to the results Survey of Schools: ICT in education
The European Commission ordered a survey that has been carried out through the collaboration of the European Schoolnet and the University of Liege, Belgium. 1200 schools were asked to participate in each country, involving teachers and head teachers of grads 4, 8, 11, also students of 8 and 11. Hungary produced quite a good participation rate, more than 50%.
In most areas of the survey we are somewhere close to the European average. There are a couple of issues that need closer attention, and these are in accordance with the findings of eLEMÉR. Hungary is the last regarding the use of VLE, we have no ICT coordinators just system administrators and IT teachers, teacher networking within and outside of the school for CPD is not in practice, and the broadband connection is not good enough for real classroom use. Teachers are not aware of the educational value of the social media, and students must be taught for more responsible internet use.
The report ends with general and area-specific suggestions every year. These suggestions did not change a lot from 2011 to 2012 or 2013. The general suggestions say that distance access would strengthen the link between school and families, it would support any time anywhere learning, and schools would become more open by using some digital log book of presence and performance, also a virtual learning environment. They could use ICT not just more but more creatively, in a better integrated way to support learning, to improve the digital skills and the 21st century skills in general. They could be better at these tasks by cooperating with other school either on national or international levels, also within the school, among each other.
We would like to establish a reward for the for-runners, a kind of mark (Becta, 2009), similar to the Eco-school system that has been working in Hungary for years. Schools are to be supported by experts, so we developed a training course for eLEMÉR advisors who can go and explain what, how and why schools can and should do with eLEMÉR, and they would be the first cohort of people who could do the validation process for the reward. We are proud of the system, it was designed all conceptualised by a handful devoted experts spending very little. The most involved ones deserve to be named. Nora Nemeth, Tartsayné was a team member for two years as an ICT pedagogy expert, Éva Tibor was the only IT expert in the project, and she is still helping me with maintaining the system. Thank you for them and all the others, teachers and experts that gave us invaluable support.
eLEMÉR as an innovation practice has been chosen and published in the VISIR project Micro innovation practices a couple of weeks ago.
(UNESCO/IFIP, 2009) Approaches to ICT development in schools. UNESCO/IFIP, 2009. Retrieved from http://wwwedu.ge.ch/cptic/prospective/projets/unesco/en/approaches1.html
(EFW 2009) Evaluation Framework for the Use of ICT in Education, EUN 2009. retrieved from http://peerlearning.eun.org/ww/en/pub/peerlearning/inspection/tools/evaluation_framework.htm
(InfoDev 2005) Monitoring and Evaluation of ICT in Education Projects: A Handbook for Developing Countries (InfoDev Handbook), World Bank, Washington DC, 2005. Retrieved from http://www.infodev.org/en/Publication.9.html
(STEPS 2009) STEPS: Study of the impact of technology in primary schools, EUN http://insight.eun.org/ww/en/pub/insight/minisites/steps.htm
(Microsoft 2009) innovative school toolkit retrieved from http://www.is-toolkit.com