Sophisticated, systematic evaluation

Are training providers interested in evaluating employer satisfaction beyond basic measures?

John Mitchell's 'Inside VET' column in Campus Review, 7 July 2009

As of 1 July 2009, and as part of the Australian Quality Training Framework, training providers are required to collect and use data on three quality indicators. Those indicators are learner engagement, employer satisfaction and competency completion. While the battle still rages in the national media about similar national data collection strategies in the schools sector, and the publication of leagues tables of schools, this compulsory survey requirement in VET has generated little public discussion.

To assist training providers collect the data, ACER was commissioned to develop handy questionnaires, but as the questions are meant to suit every employer they are necessarily very broad. For instance, the mandatory employer satisfaction questionnaire asks employers to rate providers on basic measures such as "overall we were satisfied with the training", "we would recommend the training to others" and "the training was flexible enough to meet our needs".

Given such broad questions, one can imagine there will growing concern by training providers over the next twelve months about how the survey data they provide to their registering bodies will be interpreted and used. If bureaucrats are tempted to publish the data, depending on how they interpret and release the data, there could be uproar.

Putting aside this potential controversy, the main strategic challenge associated with these questionnaires provided to training organisations is that they are only a start on the complex process of understanding and responding to the needs of learners and employers. Many of the items in the questionnaires, such as "the training focused on relevant skills", are only designed to measure the surface-level reaction to the training by learners or employers. Such questions are often satirised as 'happy sheet' questions.

The documentation supporting the questionnaires acknowledges that they are only the beginning of a process of dialogue with students and employers, and sensibly leaves it to individual providers to develop the next layers of customised evaluation and continuous improvement strategies. Strategically, this is the most important work.

Measuring multiple impacts

Challenger TAFE in Western Australia has decided to exceed the basic requirements of the AQTF and to actively develop unique tools and strategies for measuring the level of effectiveness, and the multiple impacts, of the training it delivers to specific enterprises. With funding provided by the Western Australian Department of Education and Training, and with specialist services provided by John Mitchell & Associates, the college recently commenced a project to pilot and implement the tools, in collaboration with selected clients.

The customised tools will provide both qualitative and quantitative data upon which decision making can be based, resulting in a systematic approach to satisfying individual enterprise clients.

Challenger's Jill Jamieson, the general manager of training, research and development, explains that the impetus for the project is the college's future success in a more competitive VET.

"It's a path we've been on and it's about us moving into a more competitive environment. I think we all understand that we have to be able to demonstrate to our major clients that we are adding value in the services that we're providing and that we are very targeted in how we're doing that.

"The tools will give us a more sophisticated way of targeting the clients' expectations of us. The tools will enable us to be much more proactive and much more able to engage the client at the level of sophistication that they will really be wanting in future."

The tools will be part of "a continuous circle of engaging the client, understanding their expectations, and targeting what we do with clients much more effectively. In the long run, that's going to build a stronger relationship with the client."

Jamieson expects a number of benefits to flow to such enterprises, both in the short term and the long-term. "We will be more able to provide for them what they are wanting. It might be an increase in productivity or it might be the development of a different culture within the workplace."

She is also excited about the benefits flowing to her staff. "I think it's going to be a wonderful learning opportunity for our staff. I think we'll have much more confident staff to be able to work in this space.

"We will have a better understanding of where we need to be moving our resources to be able to compete better, and we will continually be looking at the strategies that we're developing and refining our strategies to be more effective.

"I hope we have an increase in commercial revenue as well. It's about the sustainability of our business."