CHIEF Properties

Development of CHIEF

Using multiple methodologies, pools of qualified persons were identified to participate in four separate advisory panels. Methods for identifying and selecting participants assured that a wide and varied range of abilities, disabilities, attitudes, philosophies, knowledge and skills were reflected in the panel meetings. Each group included a diverse array of 32 participants with expertise in the four areas of disability: mobility, self-care, learning, and communication limitations. Four draft instruments resulted, one from each group. Each draft instrument was designed for use in a telephone or 'paper pencil' survey that would be appropriate for general population use, as well as applying to the full range of disability categories. After review and critical assessment, a synthesis of the vital elements, conceptualizations, and spirit of each draft was conducted, resulting in a fifth or "next generation" survey that identified 25 key elements of the environment.

Two forms of the draft instrument were proposed. Both had the same item content, but two different metrics were used to assess environmental impact. In one form, individuals were asked to indicate "how often" a barrier is encountered using response categories of "daily, weekly, monthly, less than monthly, and never". In the other form, individuals were asked to assess the degree to which the environmental element "facilitates or hinders" participation using response categories of "big barrier, little barrier, no impact, little help, and big help." The first form had the advantage of easier response categories, while the second form had the advantage of identifying facilitators as well as barriers.

Initial Pilot Testing

Both forms were tested on a group of 97 people, 50 with disabilities and 47 who indicated they did not have a disability. Results of that pilot testing indicated:

1. The "frequency" response categories were strongly preferred by participants over the "extent of barrier/facilitator" response categories.

2. The "frequency of barrier" response categories better differentiated people with and without disabilities than the "extent of barrier/facilitator" response categories.

3. Correlations between the two response categories, while significant, were relatively low.

Discussions of the results from the comparison of response categories with project staff and representatives who attended advisory panel meetings, yielded a consensus that all 25 items should be retained in the draft instrument, but that a follow-up impact question needed to be added since the correlation between frequency and impact was not particularly high. These discussions led to adding a follow-up question, "When this problem occurs, is it usually a big problem or a little problem?" This question was added after each item where the respondent indicated the frequency of the problem to be anything other than never. The final draft instrument, the Craig Hospital Inventory of Environmental Factors (CHIEF) was distributed to all advisory panel members for review.

Testing of the Environmental Instrument

Validation - "CHIEF 400 Dataset"

A convenience sample of 409 individuals with disability was recruited for a validation study to test the psychometric properties of the CHIEF (CHIEF 400 Dataset). In total, the sample included 124 participants with spinal cord injury, 120 participants with traumatic brain injury and 165 participants with a wide variety of other disabilities. This included 55 persons with multiple sclerosis, 35 persons with amputations, and others with auditory and visual impairments, developmental disabilities, cerebral palsy and some with multiple impairments resulting in disability. While the group of individuals with spinal cord injury was 80% male with an average age of 41 and the group of individuals with traumatic brain injury was 61% male with an average age of 41, the group of individuals with a variety of other impairments was 62% female with an average age of 48.

All 409 study participants were administered the CHIEF. In addition, 103 of the total 409 participants (46 with spinal cord injury, 44 with traumatic brain injury, and 13 with other impairments) were interviewed using CHIEF a second time, approximately two weeks after the first administration in order to assess test-retest reliability. Finally, family members or friends of 125 subjects (46 with spinal cord injury, 54 with traumatic brain injury, and 25 with other impairments), not included in the test-reliability sub-study, were successfully recruited and asked to complete the CHIEF as a proxy for the subject in order to assess subject-proxy agreement.

Analysis of this data began by defining three methods of scoring each item:

1. A frequency score on a scale of 0-4 indicating the frequency with which barriers were encountered (0=never, 1=less than monthly, 2=monthly, 3=weekly, and 4=daily).

2. A magnitude score on a scale of 0-2 indicating the size of the problem which a barrier typically presented (0=no problem since the barrier was never encountered, 1=a little problem, and 2=a big problem).

3. A frequency-magnitude product score on a scale of 0-8 calculated as the product of the frequency score and the magnitude score, indicating the overall impact of the barrier.

Total scores across the 25 items were calculated as the average frequency score, the average magnitude score, and the average frequency-magnitude product score across all of the non-missing scale items.

Test-Retest Reliability

Test-retest reliability of individual items and the total scale were calculated using both the intra-class correlation coefficient (ICC) and the percent of cases with exact agreement between both tests. Mean difference scores between the test and retest were also calculated and significance assessed. This process was repeated for frequency scores, magnitude scores, and frequency-magnitude product scores. In general, the product scores showed slightly higher reliability co-efficient and they became the focus of additional psychometric analysis. These data indicated a total scale score ICC reliability of .926, indicating acceptable reliability for the instrument.

Subject-Proxy Agreement

Across all disability sub-groups subject proxy interclass correlations ranged from 0.494 to 0.618 with a total scale ICC of 0.618. These data indicate that subject proxy agreement is marginal and result in the recommendation that proxies not be asked to complete CHIEF when subjects are unavailable to do so.

Discriminant Validity

Differences in response patterns were compared across impairment groupings in an effort to determine if the instrument differentiated among impairment groups in expected ways. The majority of items and sub-scales produced statistically significant differences among the impairment groups. For example, cases with traumatic brain injury scored dramatically lower on physical barriers than the other groups. These data lend support to the validity of CHIEF by indicating that the tool differentiates scores among different disability groups in ways that are consistent with the unique barriers faced by those groups.

Further Evaluation of the CHIEF

In 1999, the CHIEF was added to the Behavioral Risk Factor Surveillance System (BRFSS) survey in Colorado in order to: 1) examine the underlying dimensions that might exist within the context of the 25 items; 2) demonstrate its applicability to large-scale disability surveillance; 3) establish scoring norms; and 4) develop a CHIEF Short Form. The survey was administered via telephone to 2,259 individuals. The BRFSS data was weighted using the standard BRFSS weighting formula. In addition, post-stratification weighting has been applied to account for differences in age and gender between the sample and the population of Colorado. All analyses have been performed using the weighted data; therefore, the results can be generalized to the entire population of Colorado, 18 years or older.

Identification of CHIEF Subscales

Factor analysis was used to identify underlying dimensions, or subscales, within CHIEF. This analysis was performed on the 25 CHIEF items with five factors accounting for 48% of the cumulative variance across the 25 items. After varimax rotation, each item was assigned to the factor with the highest positive loading. This resulted in five factors with 3-7 items included in each factor. Descriptive labels for the factors were assigned including "policy barriers", "physical and structural barriers", "work and school barriers", "attitude and support barriers", and "services and assistance barriers".

The following indicates which items are contained in each subscale of the CHIEF:

Policies Subscale: Policies businesses, policies government, policies employment/education & services community.

Physical/Structural Subscale: Surroundings, natural environment, design home, design community, design work/school, & technology.

Work/School Subscale: Attitudes work/school, help work/school & support work/school.

Attitudes/Support Subscale: Attitudes home, discrimination, support community, attitudes community & support home.

Services/Assistance Subscale: Transportation, medical care, help home, information, education/training, help community & personal equipment.

Scoring Differentiation Between Groups

Across items, subscales and total scores, the CHIEF was able to show differences in reported frequency and magnitude of environmental barriers between groups with a variety of impairments and activity limitations. Both people with and without disabilities experienced environmental barriers. However, those with disabilities reported an overall higher frequency and magnitude of environmental barriers. Further, individuals with traumatic brain injury reported greater barriers than those identified as having a disability from the BRFSS data*, but fewer than individuals with spinal cord injury. Individuals with other types of impairments (i.e., multiple sclerosis, amputees, other auditory, visual and multiple impairments, developmental disabilities, cerebral palsy) reported the greatest barriers.

In an examination of the mean scores for each CHIEF subscale and the items on that subscale by disability status, the same general trend is seen, however some items and subscales do vary by disability status. This analysis confirms that the CHIEF has the ability to differentiate between those with and without disability and between different impairment groups.

*defined as having an impairment or health problem that: limits ability to work, learn, remember, concentrate, or any other activity or requires special equipment or help

Development of the CHIEF Short Form (CHIEF-SF)

Several criteria were used to determine which items should be retained for a "short form" version of the CHIEF. In general, these criteria included items which: 1) had the highest frequency of barrier mean scores; 2) had the highest magnitude of barrier mean scores; 3) had the highest item score-subscale score correlations (using the mean frequency-magnitude product score); 4) had the highest item score-total score correlations (using the mean frequency-magnitude product score); 5) were the most frequently reported barriers; and 6) best differentiated between people with and without disability. In addition, taking all of the criteria into consideration, if an item was to be excluded, but it was felt, conceptually should be in the scale, it was retained. The following 12 items within the original five subscales were retained:

Policies Subscale: Policies businesses & policies government

Physical/Structural Subscale: Surroundings & natural environment

Work/School Subscale: Attitudes work/school & help work/school

Attitudes/Support Subscale: Attitudes home & discrimination

Services/Assistance Subscale: Transportation, medical care, help home & information

The CHIEF-SF retained the ability of the long form CHIEF to discriminate among groups with differing disabilities (CHIEF 400 and BRFSS data).

This has been an abbreviated description of the properties of the CHIEF. For a more detailed description, please download the PDF file (CHIEF_Prop.pdf, 133K) that includes additional information on the development of the CHIEF and the PDF file (CHIEF_Figs.pdf, 2416K and SF_Figs.pdf , 278K) that have the supporting charts and figures.

(What is PDF and How do I use PDF Files?)