Objective evaluation measures

Aside from the typical recsys measures such as accuracy and precision, in live experiments several other objective evaluation measures can be taken. These typically concern the users' behavior with the system: number of log-ins, session length, clicks, item views, and purchases. Objective evaluations provide a ground truth for the effect of the system on its users. However, it is sometimes hard to interpret differences in user behavior. For example, if users of a video recommender system click on more clips to watch, does this mean that the user experience is better (more consumption) or worse (more browsing)? In this case, the number of clips watched from beginning to end would be a better measure. Better yet, one can combine objective and subjective evaluation measures. In quantitative user experiments or field trials, this entails correlating the objective and subjective measures. In qualitative user-studies (specifically observational studies), this entails asking the user to think aloud while performing tasks/work.