View source for Simplifying user-centric evaluation

== Reducing complexity ==
Aside from being time and resource intensive, user-centric evaluation of recommender systems can be a complicated endeavor. This especially holds for [[quantitative user experiments or field trials]] using one or more [[subjective evaluation measures]]. For quick iterations on an existing system, researchers may instead use [[qualitative user-studies]], but such [[formative evaluation]] methods cannot be used to generate statistically conclusive research findings.

Some work has been done to simplify quantitative evaluation by reducing the complexity of measurement. Instead of creating a questionnaire with 5-7 items per concept, one can ask only the one or two most accurate ones. Taking this one step further, one can forego asking questions altogether, and instead measure certain behavioral aspects that are proven to correlate with specific subjective constructs. Previous research can be used to select questions or behavioral measures that robustly measure certain concepts. [[Standardization of user-centric evaluation|Standardized evaluation frameworks]] are helpful sources of robust metrics in this respect.

Once measurement has been simplified, the next step is to simplify evaluation. By only selecting one or two questions per concept, one avoids the need for Structural Equation Modeling or Factor Analysis. Instead of path models, simple linear regression or correlation can be used to evaluate the effects, provided that previous work has considered possible mediation effects.

== Existing simplifications ==
Two simplifications have been proposed. Pearl Pu and Li Chen offer a simplified version of their [http://sunsite.informatik.rwth-aachen.de/Publications/CEUR-WS/Vol-612/paper3.pdf user centric evaluation framework] that is basically a subset of their questionnaire. Similarly, Knijnenburg et al. propose the idea of simplifying their [http://www.usabart.nl/portfolio/framework.pdf evaluation framework] as an evaluation toolbox.
[[category:User-centric evaluation]]