Author: Michael Kenny

Enhancing evaluation and valorisation practices of AE policies at regional and local level

CREATE2Evaluate is a follow-up of the previous CREATE project which intends to reinforce its aims and goals enhancing the performance and efficiency of Adult Education (AE) by intervening at systemic level at the nexus between overarching EU Policy (EU2020) and implementation on the ground, i.e., regions tasked with policy formulation and implementation of education programmes.

The aim of CREATE2Evaluate project is to identify reliable tools for the evaluation of adult education at various layers of governance. Identifying these tools will advance the formulation of effective adult education policies and programmes.

Now that the project is approaching its formal conclusion, partners are finally consolidating the last two main deliverables foreseen by implementation:

1. Training Suite for the target groups of policy makers to guide them with step-by-step procedures on the use and implementation of the tools to evaluate policy interventions in the domain of AE. The Training Suite includes user-friendly and flexible training resources for policy makers, among which:
   1. Guidelines on the policy evaluation tools to fully describe the functionalities of the tools, their field of application, scale and scope. This is crucial to clarify the real validity of the CREATE 2 Evaluate tools, manage expectations from users of what the evaluation tools and deliver and under which circumstances, and also depending on the scenarios
   1. Scenario setting and profiling tools: in this instance, the training Suite will include guidelines on the type and extent of data/information/statistics and input requirements for the policy evaluation tools to provide meaningful insight. Users need to be aware of the operational implications and data requirements of the policy evaluation tools
   1. Users’ manual: this is a crucial element of the training suite, as it provides hands-on advice on how to implement the evaluation tools in diverse operational settings
   1. Introduction guide to the CREATE2Evaluate ToolBox

• A Green Paper to advance the debate and stimulate the discussion on policy formulation, implementation and evaluation of AE. The AE Policy Evaluation Green Paper will collect and collate all the findings and results stemming from the implementation

These two resources provide for an operational set of tools that allows for the upscale of the CREATE2Evaluate project results while igniting the policy dialogue and exchange on how to advance the discourse on the importance of adult education as a whole for socio-economic development and integration. Indeed, the project as a whole is devoted to the cohort of policy makers for their adoption of the tools in operational environments, providing capacity building elements.

To know more about the project, the organisations involved and all resources available, please feel free to consult the Open Education Resource Platform of Create2Evaluate: www.create2evaluate.eu

A multidimensional and cross-disciplinary resource for local AE policy making

Mapping the Impact, Validation and Evaluation of Adult Education Policies has identified that EU wide AE Evaluation is highly fragmented and lacks common definitions and standards. This makes it very difficult for policy makers to control and assess the effectiveness of current AE policies. Plain coherent and straightforward policy development is needed. Any AE evaluation framework should ensure that cross organisational policy coherence occurs in vertical and horizontal terms. Vertical coherence requires that the different levels of government – national, regional, and local – follow common policy objectives and align systems of funding, accreditation and quality assurance. Horizontal coherence implies that there is understanding and coordination across policy areas within national or regional or local government.
The challenge for the wider adult learning community is to articulate targets and indicators for adult learning that capture the humanistic conceptions and practices of nonformal adult learning. The articulation of such a metric would ensure that this domain of adult learning can no longer be considered as ‘invisible’ by the state.
With reference to findings and evidences provided by partners, the CREATE 2 Evaluate ToolBox is conceived to ensure that local and regional policy makers from across Europe will be able to use the policy tools to better plan, design, implement and monitor Adult Education policies with a clear vision of sustainability of public funding in AE.
The CREATE and CREATE 2 EvaluateToolBox comprises a series of practical tools for policy makers to ensure evidence-based policy making and better align AE policies and programmes with local socio-economic development. Resources elaborated and published by partners are available in free, open and multilanguage version so as to ease and sustain the visibility and dissemination of training and education opportunities stemming from the model.
The ToolBox is structured in six different areas, each area includes specific resources and references that sustain local policy makers in better strategizing the alignment, consistent and coherence of local lifelong learning plans to EU long-term objectives and horizons.
Consistency of the objectives and outcomes
• Checklist to assess consistency of the objectives and outcomes of the AE trainings and programmes with the overarching strategies and agendas adopted by the EU

Programme creation at the policy/public administration level
• Checklist for providers: Evaluation of provider inclusion in Adult Education Policy Making
• Evaluation of Creativity in Adult Education Policy Making Matrix
• Focus Group Questions
• Survey for providers
• Survey for Public Administration Staff

Inclusivity of AE policies and availability of AE programmes
• Checklist to assess: Inclusivity of AE policies and availability of AE programmes in geographically excluded areas
• Dimensions of Diversity in Adult Learning

AE trainings and programmes delivery
• Individual Interview with participant: Methods, equipment and personnel in Adult Education and adjustment to the target group.

Value added stemming from the participation in AE
• Measuring the Wider Benefits of Adult Learning – Focus Group Guidelines for Learners and Training Providers
• Value added stemming from the participation in AE.

Continuity of programme evaluation and use of its results to improve AE policies
• Valorisation and Follow-Up to Evaluation. Some Success Factors
• Critical Success Factors: A Case Study
• Tips for policy makers seeking stakeholder engagement

All resources developed by the CREATE2Evaluate partnership are available via the official OPEN EDUCATION RESOURCE Platform of the project: www.create2evaluate.eu

Findings from the transnational analysis conducted by CREATE2’s partners

Evaluation and Valorisation are crucial steps in creating viable strategies with long-term benefits. Yet many AE policies on the local and regional levels are not systematically evaluated. This is partly due to policy makers lacking the appropriate tools. CREATE2Evaluate addresses this gap by designing a set of useful tools and templates to help AE policy makers and agents active in the AE policy processes evaluate existing strategies, draw actionable conclusions and design better strategies in the future.

The rationale for our project CREATE2Evaluate stems from the thorough investigation of the state of the art and the needs analysis that builds a multi-pronged approach:

1. robust investigation of the field through the analysis of relevant public policy documents at both national and EU levels that confirm how at present there are no effective means and tools to monitor and assess AE policies and programmes; and where those exist, they are used in a very fragmented way.

• direct experience of the partners: the proposal brings together AE providers (both formal and non-formal, both public and private), umbrella AE organisations, AE practitioners from research & academia, the third sector and the public sector (represented by local authorities tasked with policy formulation and monitoring & evaluation).

• the results of another Erasmus+ Project KA2 funded in Germany that was recently (2019) and successfully completed (final evaluation score from the National Agency of 85), the CREATE Project that developed tools for appropriate policy formulation in the domain of AE – in the 24 months of implementation and constant involvement of local policy makers, the clear need for monitoring and evaluation tools and mechanisms emerged.

During the first half of 2021, the CREATE2 Consortium finalised a very detailed cross-dimensional and transnational mapping activity aimed at identifying current methodologies, frameworks and/or models applied at local level for policy evaluation.

The final report is available for free by consulting the section of the CREATE2 Open Educational Resource Platform of the project.

Results, takeaways and findings suggested that the primary focus of current evaluation policy is quantitative outputs and student specific learning outcomes. Within any quantitative outcome evaluation partners identified a range of gaps and inconsistencies in target group statistical data, and in private and non-formal evaluation provision. Whilst significant qualitative evaluation is conducted in adult education centres according to project partners, the valuable data emerging remains inaccessible due to lack of transparent systematic overarching analysis and learner protection requirements. This makes it particularly difficult for policy makers to assess the effectiveness of their adult education policies.