Format of Our Evaluations

Each of our evaluations begins with the name of the information source we’re reviewing, our overall rating, and an overview. Our overall rating is based upon the weighted average of five key attributes. Our overview contains a few basic pieces of information like location and purpose (for evaluating hospitals or doctors, etc.). We rate each information source along five dimensions and then offer a “take-away” entitled “What You Need to Know.”

Overall Rating

These five attributes are important, but not equally so; our overall rating reflects these differences. Everything begins with usefulness, followed by methodology. Timely disclosure is crucial. Coverage and accessibility are what make these values personally useful.

As you read, you’ll find – unlike almost any other guide to health care – that no source has achieved the highest possible overall score. We don’t grade on a curve – we grade on what consumers need.

We assign one of the following overall ratings to every health care information source that we evaluate:

Overall Rating
EXCELLENT
VERY GOOD
GOOD
NEEDS IMPROVEMENT
FLAWED

We note especially worthy data sources which serve as benchmarks for the rest of the industry with an "editors' choice" designation. We designate very good – but not quite as good as we would like – data sources “Commended.”

Specific Attributes

We have developed a five-point rating scale for each of the five different health care data attributes (along with our proprietary overall rating algorithm).   These will help you better understand the quality of the available information; how to use the information appropriately; and what to avoid.

Usefulness

Here's the single most important question you can ask yourself when it comes to usefulness: can this information lead me - with confidence - to the decision that I need to make?

It’s surprising what is relevant and what isn’t when it comes to healthcare health care information. Your doctor loves to ski? This may turn out to be the difference between recommending that you learn to live with reduced mobility or referring you to the orthopedic surgeon for the Pennsylvania Ballet who does nothing but knees.

We rate objective outcomes data higher than subjective data. We also consider the scope, or depth, of the reported metrics.

We use the following 1-5 scale (5 being the most useful and therefore the best) in evaluating usefulness:

Usefulness
5 Can be used to make a decision without consulting other data sources
4 A good source of helpful information
3 A comprehensive directory – helpful to identify candidates, but inconclusive
2 Not much more helpful than a telephone book
1 Misleading information – not to be trusted

Methodology

Neutral, independent, unbiased – whichever word you like best, this is critical. Put aside the fact that a particular study shows wonderful results. First you need to know: who published it? Who wrote it? Who sponsored it?

The most important criterion for evaluating methodology is objectivity. The health care industry is more susceptible to conflicts of interest in creation and dissemination of information than most other industries.

Unfortunately, the media, academia and the Internet are clogged with an astounding amount of biased and flawed data.

We use the following 1-5 scale (5 being the most objective and therefore the best) in evaluating methodology:

Methodology
5 Statistically appropriate and unbiased outcomes data
4 Outcomes data - but a potential conflict of interest or a biased methodology
3 Approximate data - some indication of the provider's ability, but not outcomes based
2 Directory information - appropriate measures and unbiased
1 Directory information - flawed or biased

Timeliness

Sometimes old data are worse than no data. The most recent state data available for Bayshore Medical Center in Pasadena, Texas, for example, reported that Bayshore had the highest coronary artery bypass graft (CABG) mortality rate in greater Houston during 2000. By 2002, the Bayshore CABG death rate had dropped to zero with the addition of two new, renowned surgeons. Unfortunately, this new statistic has not yet been widely published. Anyone relying on the available state data would draw the wrong conclusion.

We use the following 1-5 scale (5 being the most timely and current and therefore the best) in evaluating timeliness:

Timeliness
5 Multiple updates per year
4 Annual updates - one year old data reported
3 Annual updates - reported data more than one year old
2 Less than annual updates
1 No fixed update schedule

Coverage

Coverage is the most personal attribute in selecting care: no matter how good the data may be in another state, all else being equal you want data about your own back yard. But coverage is critical for a second reason: it introduces a discussion about standards. When every state has the same standards for health care information and disclosure, the quality of care in this country will rise dramatically.

We use the following 1-5 scale (5 being fairly complete national coverage and therefore the best) in evaluating coverage:

Coverage
5 Fairly complete national coverage - detailed information about most providers available
4 Fairly complete regional coverage - detailed information about most providers available
3 Incomplete national coverage - fairly complete provider list but lacks detailed information or has gaps
2 Incomplete regional coverage - fairly complete provider list but lacks detailed information or has gaps
1 Incomplete provider list for intended coverage area

Accessibility

Private companies and government agencies that restrict public access collect some of the best health care information. If you can’t see the information, you can’t use it.

We use two criteria to evaluate accessibility: availability and cost.

We use the following 1-5 scale (5 being the most accessible and free and therefore the best) in evaluating accessibility:

Accessibility
5 Complete data available on web site free of charge
4 Complete data available on web site for a fee
3 Reports or portions of data available free for charge
2 Reports or portions of data available for a fee
1 No public access to data