EvalSmart — Research Consultation Service — University Library — Sample Evaluation Brief
STATUS: DRAFT — REQUIRES HUMAN REVIEW
Planning artifact — not an authorization. This plan does not authorize new data collection, participant-level linkage, cross-border data transfer, analysis of identifiable or minor-participant data, or any compliance certification. The prerequisites it names must be resolved before execution.
Illustrative sample — a fictional program, AI-drafted and stamped DRAFT for human review.
The questions this evaluation answers
The Research Consultation Service has a clear program theory and strong stakeholder commitment, but currently lacks the measurement infrastructure to answer its own key questions. The immediate priorities are: confirm the program review timeline, establish a baseline outcome measure, resolve IRB requirements, and audit the tracking system. Once these prerequisites are met, the mixed-methods plan can generate defensible evidence of patron outcomes, service consistency, and equity of reach.
q1 — Do consultations actually improve patrons' research outcomes and information-literacy skills, beyond self-reported satisfaction? — The program cannot yet show that consultations produce measurable improvements in patron research ability or information-literacy skills, because no standardized outcome instrument or baseline data currently exist — satisfaction counts cannot be distinguished from evidence of learning or skill gain.
q2 — Is the consultation experience and its outcomes consistent across subject librarians, and does variation in delivery matter for patron outcomes? — The program cannot yet show whether outcome differences across librarians reflect genuine variation in service quality or other factors, because session delivery practices are undocumented and no librarian-level outcome data have been collected.
q3 — Is the service reaching the campus equitably — across disciplines, academic level (undergraduate vs. graduate), and modality (in-person vs. online)? — The program cannot yet show whether service reach is proportionate to campus population composition, because data linkage between consultation records and institutional enrollment data has not been confirmed as feasible, and tracking-system completeness varies across librarians.
q4 — How can the assessment system be strengthened to produce defensible evidence of impact for program review? — The program cannot yet show what a defensible assessment system would look like in practice, because the program review timeline, IRB requirements, data-linkage feasibility, and librarian capacity for new data collection activities have not yet been established.
q5 — Are there patron subgroups (e.g., by discipline, academic level, or modality) who are underserved or who experience different outcomes? — The program cannot yet show whether specific patron subgroups experience systematically different outcomes or access barriers, because subgroup-level outcome data do not exist and the specific equity-relevant subgroups of local concern have not been identified.
The decisions only you can make
EvalSmart names these; your team owns them. A few must be resolved before any data collection begins — the full checklist is in the executive brief.
Confirm program review timeline and submission deadlines before finalizing any method choices
Obtain IRB review or exemption determination for all new data collection (outcome surveys, focus groups, librarian interviews) before beginning data collection
Audit appointment-tracking system for data completeness and export capability before designing equity-of-reach analysis
Confirm data-linkage feasibility with Institutional Research / Registrar before designing subgroup outcome analyses
As a first step, confirm with the Head of Research Services and Assessment Librarian whether any prior outcome data have ever been collected. If not, plan to establish a baseline in the earliest feasible data-collection window this cycle. Evaluate candidate outcome instruments (e.g., a standardized patron outcome survey, a locally developed post-consultation instrument, or a pre/post design) for fit with the service's goals, response-rate feasibility, and IRB requirements before selecting one
Confirm the program review schedule and key reporting deadlines immediately — this is the single most important constraint on method selection. Use the confirmed timeline to determine which methods (e.g., follow-up surveys, focus groups, longitudinal tracking) are feasible within the available window
Priority measures — and what blocks them
Status — Ready: computable now; Needs setup: you supply a value or data first (a baseline, threshold, identifier, or linkage); Blocked: a decision or approval must happen first (IRB / privacy / consent, a scope or design decision, or instrument mapping).
Measure (quant): M-1 — Post-consultation patron-reported improvement in ability to find and evaluate sources independently (knowledge/skill dimension) (status: Blocked)
Explain (qual): QQ2 — How do patrons perceive the durability and transferability of what they learned in the consultation — do they feel they can apply the skills independently in future research tasks?
M-1 (patron-reported improvement in ability to find and evaluate sources) is the primary post-consultation outcome measure; QQ2 (patron perceptions of skill durability and transferability) explains whether reported gains are experienced as lasting and applicable beyond the immediate session.
Measure (quant): M-10 — Variance in patron-reported outcomes attributable to subject librarian (inter-librarian consistency of outcomes) (status: Blocked)
Explain (qual): QQ3 — How do subject librarians describe their approach to conducting a consultation — what goals do they prioritize, what strategies do they use, and how do they adapt to different patron needs?
M-10 (variance in patron-reported outcomes attributable to subject librarian) quantifies inter-librarian consistency; QQ3 (librarian accounts of their consultation approach, priorities, and adaptive strategies) explains the sources and nature of that variation.
Measure (quant): M-4 — Patron-reported application of research skills to subsequent coursework or projects (delayed / follow-up outcome) (status: Blocked)
Explain (qual): QQ2 — How do patrons perceive the durability and transferability of what they learned in the consultation — do they feel they can apply the skills independently in future research tasks?
M-4 (delayed patron-reported application of research skills to subsequent coursework) is the closest available measure of durable IL skill transfer — the service's stated intermediate outcome; QQ2 probes whether patrons experience that transfer in their own terms.
Measure (quant): M-7 — Service reach ratio: proportion of consultations by patron subgroup relative to that subgroup's share of the campus population (status: Blocked)
Explain (qual): QQ5 — What barriers — perceived or experienced — prevent certain patron groups from booking or fully engaging with the consultation service?
M-7 (service reach ratio by patron subgroup relative to campus population share) is the primary equity-of-reach indicator; QQ5 (patron-reported barriers to booking or engaging with the service) explains why certain groups may be underrepresented.
Measure (quant): M-8 — Documentation / appointment-tracking completeness rate: share of consultation records with all required fields populated (patron role, department/discipline, modality, topic) (status: Needs setup)
Explain (qual): QQ9 — How do patrons who did not respond to the post-consultation survey differ in their experience or outcomes from those who did — and what would make them more likely to respond?
M-8 (appointment-tracking completeness rate) directly measures the quality of the primary administrative data source; QQ9 (perspectives of non-respondents on their experience and on what would increase survey response) addresses the non-response bias that undermines the defensibility of all outcome data.
Top gaps
No baseline outcome data — cannot measure change or improvement
Program review timeline unknown — constrains all method choices
IRB status unconfirmed for all new data collection
Tracking-system data quality and export capability unverified
Data linkage between survey responses and tracking records unconfirmed
Produced by EvalSmart — draft for human review.
This EvalSmart starter package converts a program description into an evaluation-ready brief: priority evaluation functions, quantitative indicators, qualitative follow-up questions, evidence gaps, and prerequisite decisions requiring human review.
ACRL is a professional standards body, not an accreditor; where ACRL's SLHE principles or Project Outcome are referenced, they are organizing context, not a compliance verdict.
Standards and frameworks referenced above are included as contextual alignment prompts and must be verified by your institution before any accreditation or compliance use. EvalSmart cites them as organizing context, not as certification of compliance.
This is the short sample. A full EvalSmart run also produces a fuller executive brief and a comprehensive technical plan — every item tagged stated, inferred, or gap, and reviewed by a human.