Analytical Methods for Evaluations

Analytical evaluations differ from empirical evaluations in that analytical evaluations do not include user observations. Reviewers, most often experts, rely on data and quantitative criteria when conducting evaluations. Internal and external financial auditors, prototype developers and business process analysts all conduct analytic evaluations. Available analytic evaluation methods focus on determining how close data values come to benchmark parameters.

Regardless of the method used, the goal of an analytical evaluation is to establish relationships between actual and benchmark data to determine whether variations exist. For example, financial auditors use analytic evaluation methods during the planning stages of an audit. The objectives are to identify variations in relationships, such as unusual transactions, ratios and trends that indicate financial data requires greater scrutiny, a longer audit time frame and procedures that are more detailed.

Software developers commonly use cognitive walk-through evaluations in the design phase of development. The goal is to identify strengths and weaknesses in a prototype design and how users will understand it. Data sources include a user interface mock-up, a user profile that assumes a specific knowledge level, task lists and action sequence diagrams. A cognitive walk-through starts with analysis of the steps and actions required to accomplish a task, and system responses to user actions. Evaluators, typically designers and developers, then walk through the steps as a group, gathering usability data along the way. Analysis determines whether tasks or actions require a redesign.

Unlike the team approach used in a cognitive walk-through, a heuristic evaluation is actually a series of independent evaluations. It is useful in analyzing operational processes, developing standard operating procedures and writing an instructions manual. Data sources include established guidelines and performance measurements. During the evaluation, two or three analysts compare current procedures against pre-established guidelines or principles, with each looking for and ranking a specific issue such as unsafe, erroneous, and duplicate or redundant actions. A post-evaluation meeting and analysis determines which instructions require modification.

Point-factor evaluations are common in job evaluations. Goals typically focus on ranking different jobs within a company and establishing a pay-grade or structure. Data sources include role profiles, job descriptions and a numerical ranking system. In a point-factor evaluation, reviewers -- who most often are human resources staff members -- identify and break the key elements of each job into separate components. Evaluators then compare these factors to role profiles and allocate points according to the skills, expertise or level of difficulty of each specific job. Most often, the more demanding a job is, the higher the point value and the higher its pay grade.