Evaluating effectiveness will be different for each organization, department, and program.  An evaluation needs to focus on a specific set of outcome measures chosen by leadership.  A good beginning to evaluation includes reading the white paper on evaluation to understand the foundation of any effective evaluation process. A second step would be to do an academic search to see if any other programs have done similar evaluations.  It is important to understand no two evaluations are the same, because each program is different and have different evaluation needs.  However, in order to help “jump start” an evaluation, we have created a list of common variables programs tend to evaluate. 

1.      Health Outcomes

a.      Disease management

b.      Comfort at end of life

c.      Correctness of diagnosis made

d.      Effectiveness of treatment plan 

e.      Reduced visits and emergencies

f.        Medication usage

g.      Reduction in medical errors

2.      Cost Outcomes

a.      Cost benefits

b.      Cost analysis

c.      Return on Investment (ROI)

3.      Patient and/or Provider Perception

a.      Satisfaction

b.      Utilization

c.      Adoption/diffusion     

4.      Health Systems Delivery

a.      Resources

b.      Workflow

c.      Referrals

d.      Organizational structures

e.      Coordination among disparate health orgs

5.      Policy Issues

a.      Regulatory issues

b.      Licensure

c.      Reimbursement

After selecting the category you would like to evaluate you will need to decide on measurement procedures.  Below are examples of different methods commonly used.  Keep in mind that several methods can be utilized for the same evaluation.

1.      Pre-existing surveys

These surveys have the benefit of being tested for validity and reliability, which will add credibility to a research endeavor.  Surveys generally produced quantitative data.

a.      Short Form 12 (SF12)

b.      Beck’s Depression Inventory (BDI) scale

c.      Quality of Life Index

d.      Other validated surveys for specific outcomes can be found through searching Internet engines and past literature.

2.      Surveys from other studies

a.      These surveys are often good places to begin, especially if the past evaluation is similar to what you are trying to measure.  However, it is important to note they may not have been tested rigorously for validity and reliability.

b.      If you read an article that developed a survey for a study that you think might be helpful for your evaluation, contact authors and ask for a copy.

3.      Develop your own survey

a.      If you are unable to find any existing surveys from the literature that meet your evaluation needs you can borrow ideas from others or develop your own questions to build a survey. Be careful to seek consultation from a person with survey design expertise.

4.      Interviews

a.      Interviews have the benefit of letting the researcher explore issues in greater depth than a survey would normally allow. They tend to be qualitative in nature.

5.      Focus Groups

a.      Focus groups are similar to an interview, except you are questioning several people at the same time.  This method generally requires a moderator and a note taker or a tape recorder for transcription.  The moderator has a list of set questions to ask a small group of individuals (around 6-12) in order to facilitate discussion about concepts of interest. Focus groups allow more flexibility in topics and can provide insight unavailable through interviews.  Focus groups tend to produce qualitative results.

6.      Observation

a.      This method involves observing behavior as it naturally happens, where it occurs. The observer should be unobtrusive and not interfere with the subjects who are being observed. An advantage to this method includes its ease, however, it does not enable you to explain why a subject behaved a certain way.

7.      Archival Data

a.      Archival data is information that already exists.  If interested in the diffusion of a technology, you may want to collect information regarding the launch, training materials, emails of the technology’s introduction, etc.  This type of data is useful in monitoring a program over time.

8.      Chart Data

    a.      Chart data comes directly from a patient’s medical record and enables you to access visits, changes in physiological states, and medication usage.