HiMAT Properties

The psychometrics properties of the HiMAT were investigated throughout the development phase.

Validity

Content validity and Unidimensionality
The content of the HiMAT was initially generated from a review of existing mobility scales and by surveying expert opinion (Williams et al., 2005a). Rasch analysis was then used to establish content validity and unidimensionality of the items that were generated (Williams et al., 2005b).

In the initial stages, a literature review was conducted on adult and paediatric neurological mobility scales to identify which items had previously been reported as “high-level” mobility items. The review identified 18 different items from 31 mobility scales. To further extend the pool of high-level mobility items, a consensus method was used to survey the opinions of expert physiotherapists and physical educators. Expert clinicians generated 157 items that were collated and condensed to 88 items for ranking on a questionnaire by excluding items with excessive equipment demands or items that were too dependent on upper limb or cognitive involvement. This meant that items such as running whilst throwing and catching a ball or climbing a ladder were excluded. Fifteen items on the questionnaire were rated as very important by 80% of expert clinicians. These included walking forwards, on slopes and different surfaces, changing direction as well as walking long distances and stair use. Running items included forwards, backwards, on slopes and different surfaces, changing direction, stopping and starting as well as running long distances. Balancing in single limb stance was also included. Some items from the literature review that were not rated very important by 80% of expert clinicians were added to the 15 ‘very important’ items, resulting in a group of 20 high-level mobility items that were prepared for testing on TBI clients.

To investigate the content validity and unidimensionality of the high-level mobility items that would ultimately form the HiMAT, a three-stage process was used. Firstly, internal consistency was investigated using Cronbach’s alpha. Internal consistency, which provides an estimate of the extent to which the items measure the same domain, was very high (Cronbach’s alpha of .99). Second, principal axis factoring was performed to identify items that were linearly correlated (indicating unidimensionality) with each other. Principal axis factoring led to the identification and exclusion of the balance items and two hopping items on the less-affected leg. Finally, the remaining items were further scrutinized using Rasch analysis.

Rasch analysis is a powerful tool in the initial stages of scale development, particularly for establishing content validity and supporting construct validity. Rasch analysis compares individual response patterns to those of the entire sample to estimate both an individual’s ability and item difficulty (Williams et al., 2005b). Rasch analysis provides a framework, with guidelines, for verifying rating scale categorization to ensure efficient measurement. Misfitting items are identified, and excluded, by high weighted mean square fit statistics. In the development of the HiMAT, the more stringent upper limit of 1.3 was applied. Items with weighted mean square fit statistics exceeding 1.3 were excluded from further analysis.

Discriminability
Once misfitting items were excluded, Rasch analysis was used to investigate the discriminative ability of the remaining items (Williams et al., 2005b). The item estimates assigned to each item identify the level of difficulty of each task. These estimates establish a hierarchy of difficulty. Clustering is a term used to describe items with similar levels of difficulty. Where items clustered, redundant items were excluded. Redundant items were identified and excluded and the more practicable and feasible item was retained. Item practicability and feasibility were assessed by considering the time, equipment and location required to test each item. For example, walking, walking on grass and the 6-minute endurance test all had a very similar level of difficulty. In this case, the walking item was retained as it does not require access to a flat grassed surface and is quicker to administer than a 6-minute endurance test. This process led to the exclusion of seven items, leaving 13 items on the final version of the HiMAT. Both the easiest and most difficult items were retained during the exclusion process so that the range of ability quantified by the HiMAT was maintained. Only redundant items that clustered in the middle of the scale, providing little discriminative information, were excluded.

An investigation was performed to determine if summed raw HiMAT scores provided a valid measure of motor ability (Williams et al., 2005b). Summed raw scores were compared to the logit scores obtained for each individual. The correlation between the summed raw scores and the logit scores was very high (r =.98). This result means that summed raw scores adequately reflected true change for the majority of the range of ability that the HiMAT quantifies.

Concurrent validity
Concurrent validity refers to the comparison between measures that assess the same construct. This approach to validation is used to compare a new measure, such as the HiMAT, against existing measures. The HiMAT was compared to the motor component of the Functional Independence Measure (FIM) and the gross function component of the Rivermead Motor Assessment (RMA). The FIM was chosen because it represents the most frequently used measure of physical performance in TBI rehabilitation. Despite its intended use as a ‘burden of care’ measure for inpatient rehabilitation, it is frequently used to report long-term outcome following TBI. The RMA contains more high-level items, such as running, stair climbing and hopping, than existing adult mobility scales. This scale was originally developed in the stroke population, but is yet to be validated in TBI.

To investigate concurrent validity, 103 patients were concurrently scored on the HiMAT, motor FIM and gross function RMA. Correlations, using Pearson’s r, were calculated between the HiMAT, the motor FIM and the gross function component of the RMA to investigate concurrent validity. The correlation between the HiMAT and the motor FIM was only moderately strong (r = .53, p<.01) due to a substantial ceiling effect the motor FIM suffers when compared to the HiMAT. More specifically, the motor FIM was unable to discriminate motor performance for 90 (87.4%) of the 103 patients, yet these patients had a mean score on the HiMAT of only 32.6/54 (SD = 13.8, range 5-54) (Williams et al., 2006b).

The HiMAT and gross function RMA had a much stronger correlation (r = .87, p < .01), but the gross function RMA also had a substantial ceiling effect when compared to the HiMAT. Fifty-three patients (51.5%) scored the maximum score of 13/13 on the gross function RMA, yet had a mean score of only 41.7/54 on the HiMAT (SD = 8.8, range 24-54) (Williams et al., 2006b).

Reliability

Inter-rater reliability
Inter-rater reliability was investigated by three physiotherapists concurrently and independently scoring the performances of 17 people with TBI. The three physiotherapists had an average of nine years of clinical experience in neurological physiotherapy. Since the HiMAT was developed as a simple and easy to use clinical tool for physiotherapists, two of the three physiotherapists used to investigate inter-rater reliability had no prior knowledge or training on the HiMAT, but were provided with an instruction sheet just prior to testing. Investigation of live inter-rater reliability, rather than rating from videotaped recordings, is essential as it reflects the real-life conditions experienced by clinicians in the workplace.

An intraclass correlation coefficient (ICC: 2,1) was used to assess inter-rater reliability for each of the items. The ICCs calculated to determine the inter-rater reliability of the three examiners were very high (ICCs = .99) for each of the individual items on the HiMAT. Total scores on the HiMAT were also calculated from the coded scores for each of the physiotherapists and assessed using an ICC(2,1). The inter-rater reliability of the total HiMAT scores was also very high at .99 (Williams et al., 2006a). This result demonstrates that the HiMAT has high inter-rater reliability and supports the user friendliness of the HiMAT as this result was achieved even though two of the therapists had no prior knowledge or training.

Retest (intra-rater) reliability
To investigate intra-rater reliability 20 people with TBI were asked to return for repeat testing two days following their initial test. To ensure that natural recovery was unlikely to occur between the initial and repeat tests, only participants who had sustained their TBI more than 18 months prior to testing were asked to return for repeat testing.

Three different calculations were performed to investigate retest reliability.

1) An ICC(2,1) was calculated on the total HiMAT scores. The retest HiMAT ICC was very high (ICC = .99).

2) To determine if a systematic error had occurred, a paired t-test was used on the groups’ total HiMAT scores. A paired t-test showed a mean improvement of 1.0 point (t =3.82, p<.001, range –1 to +3) on the HiMAT at retest, indicating a systematic improvement had occurred. This means that on average, people with TBI improve by 1 point when retested two days later. Although this improved score is significant, it is small and well within the confidence intervals for detecting clinically important change.

3) The standard error of measurement (SEM) was also calculated to determine the 95% confidence intervals (CI) for determining the minimal detectable change (MDC95) on the HiMAT according to the formula:

MDC95 = mean difference ± 1.96 x SEM

Minimal detectable change values are a reflection of clinically important change and the likelihood that true change has occurred. To calculate the SEM, the standard deviations from the pre-test and post-test scores were pooled according to the equation outlined by Mendenhall, McClave, and Ramey (1977). The MDC95 for the HiMAT was calculated at +/- 2.66 points. Considering the mean difference between test and retest (1 point), this means that to be 95% confident that clinically important change (improvement or deterioration) has occurred, participants have to improve by 4 points or deteriorate by at least 2 points (Williams et al., 2006a).

The HiMAT is highly reliable, but clinicians need to be aware that people with TBI tend to achieve improved HiMAT scores when retested after two days. This improvement is most likely to be from improved confidence in attempting challenging tasks gained from the first testing session, and most unlikely to be attributed to neural recovery. This highlights the importance of the familiarization trials before testing so a lack of confidence does not impact on motor performance.

Internal consistency
Internal consistency of the 13 HiMAT items was very high (Cronbach’s alpha of .97) (Williams et al., 2006a).

Floor and Ceiling effects
The HiMAT has specifically been designed to quantify high-level mobility. It is only appropriate for patients who are able to walk independently without a gait aid for at least 20 meters. Due to the floor effect, the HiMAT is more appropriate in the latter stages of inpatient rehabilitation and throughout outpatient and community integration rehabilitation. To date, the HiMAT has no demonstrable ceiling effect. In comparison to other scales of motor performance it is much less susceptible to a ceiling effect and is better able to discriminate high-level mobility (Williams et al., 2006b).

Responsiveness
Responsiveness refers to the ability of a measure to detect clinically meaningful change over time, and provides a means for determining if an individual’s score changes are related to true recovery, or to natural variation in repeated performances. Multiple methods have been proposed for investigating responsiveness. Responsiveness indices can be grouped into two main types. The first type evaluates the amount of change relative to measurement error. Examples include the method described by Guyatt, Walter and Norman (1987), and that suggested by Goldie, Matyas, and Evans (1996). The second type of index of responsiveness evaluates change in a group of patients in relation to the variability of change scores in the same group. This type of responsiveness index, such as that described by Liang, Fossel, and Larsen (1990), is influenced by varying rates of patient recovery and response to treatment. Scale responsiveness is an important concept for clinicians in this age of evidence-based practice where funding bodies may base payment on demonstrable change. Understanding and interpreting the responsiveness of a scale enables clinicians to discriminate true change from measurement error.

To investigate responsiveness, 14 people with acute TBI who were initially tested less than 12 months post-accident returned for repeat testing 3 months later. People less than 12 months post-TBI are still in their acute recovery phase and are therefore likely to change over a 3 month period. The 14 people with TBI were tested on the HiMAT, motor FIM and gross function RMA to compare the ability of each of these scales to detect change in high-level mobility. Three different methods were used to evaluate the responsiveness:

1) The method initially described by Guyatt et al. (1987). It was modified by Goldie et al. (1996) to take into account the systematic change that may occur with repeated measures.

2) The method described by Liang et al. (1990).

3) The third method, suggested by Goldie et al. (1996), calculates the proportion of patients that changed by at least as much as the minimal detectable change (MDC95) score.

Investigation of the responsiveness of the HiMAT, motor FIM and gross function RMA showed that the HiMAT was more responsive to change in high-level mobility, regardless of the index used to calculate it (Williams et al., 2006b). This is most likely due to the ability of the HiMAT to quantify mobility to a much greater extent, indicating it is less susceptible to a ceiling effect than existing scales.

Feasibility and Practicality
The HiMAT was specifically developed to be quick and easy to use with minimal time, equipment and training requirements. Testing typically requires only 5-15 minutes depending on the patients ability. Measurements are made with a stopwatch and tape measure which are routinely available and utilised in clinical settings.

Goldie PA, Matyas TA, Evans OM: Deficit and change in gait velocity during rehabilitation after stroke. Archives of Physical Medicine and Rehabilitation. 1996; 77: 1074-1082.

Guyatt G, Walter S, Norman G: Measuring change over time: assessing the usefulness of evaluative instruments. Journal of Chronic Diseases. 1987; 40: 171-178.

Liang MH, Fossel AH, Larsen MG: Comparisons of five health status instruments for orthopaedic evaluation. Medical Care. 1990; 28: 632-642.

Williams, G., & Goldie, P. (2001). Validity of motor tasks for predicting running ability in acquired brain injury. Brain Injury, 15, 831-841.

Williams, G., Robertson, V., & Greenwood, K. (2004a). Measuring high-level mobility after traumatic brain injury. American Journal of Physical Medicine and Rehabilitation, 83, 910-920.

Williams, G.P., Morris, M.E., Greenwood, K.M., Goldie, P.A., Robertson, V. (2004b). The High-level Mobility Assessment Tool For Traumatic Brain Injury: User Manual. Melbourne: La Trobe University. ISBN:1920948724.

Williams, G., Robertson, V., Greenwood, K., Goldie, P., & Morris, M. E. (2005a). The High-level Mobility Assessment Tool (HiMAT) for traumatic brain injury. Part 1: Item Generation. Brain Injury, 19 (11), 925-932.

Williams, G., Robertson, V., Greenwood, K., Goldie, P., & Morris, M. E. (2005b). The High-level Mobility Assessment Tool (HiMAT) for traumatic brain injury. Part 2: Content Validity and Discriminability. Brain Injury, 19 (10) 833-843.

Williams, G., Greenwood, K., Robertson, V., Goldie, P., & Morris, M. E. (2006a). High-level Mobility Assessment Tool (HiMAT): Inter-rater Reliability, Retest Reliability and Internal Consistency. Physical Therapy, 86 (3) 395-400.

Williams, G., Greenwood, K., Robertson, V., Goldie, P., & Morris, M. E. (2006b). The concurrent validity and responsiveness of the High-level Mobility Assessment Tool (HiMAT) for measuring the mobility limitations of people with traumatic brain injury. Archives of Physical Medicine and Rehabilitation, 87 (3) 437-442.