The High Dimensionality of Students' Individual Differences in Performance in EPGY's K6 Computer-Based Mathematics Curriculum

1 Introduction
2 EPGY Trajectories
    2.1 Computer-Time Trajectories
    2.2 Calendar-Time Trajectories
    2.3 Trajectory Parameters
3 Individual Differences
    3.1 The Multidimensionality of Individual Differences
    3.2 The Robustness of Individual Differences
4 The Undominated Set
5 Conclusion
6 References

Appendix A: Distributions and Fits of Computer-Time Parameter Values
Appendix B: Distributions and Fits of Calendar-Time Parameter Values
Appendix C: Joint Distributions and Fits of Trajectory Parameter Values
Appendix D: Further Discussion of Student Ages in EPGY
Appendix E: Distributions and Fits of the Five Ability Parameter Values
Appendix F: Sample Partial Orderings of Students Above Dominated Students
Appendix G: Partial Orderings for Various Dimensions in Each Grade

1 Introduction

A major focus of current educational research is the quest to understand students' motivation and engagement in their studies. The rationale for this is that many students are not motivated by their studies and do not show signs of being engaged in what they are learning. Curriculum designs are often motivated chiefly by the goal of understanding how to produce such motivation and engagement among students who do not exhibit it.

Overlooked in such curriculum designs are students who are already intrinsically motivated by the content of instruction. These students are often bored with curriculum that does not cover content rapidly or deeply enough to keep their interest. They are labeled `gifted students' or `high achievers.' That they will succeed academically is taken for granted, so the problem of designing a curriculum optimally to challenge this particular group of students is rarely treated in educational research. In this study, data from almost two thousand elementary- and middle-school students in the mathematics courses of Stanford's Education Program for Gifted Youth (EPGY) are analyzed for the purposes of better understanding gifted students' individual differences and refining curriculum to help such students to realize their full potential.

In traditional classrooms, many of the different learning needs of gifted students are suppressed in the interest of keeping all students in step with each other. Indeed, the range of individual ability differences among students is a daily pedagogical challenge for teachers; a 1999 government case study of the educational system in the United States found that many teachers deal with this discrepancy by 'teaching to the middle level of the classroom' in hopes of reaching a maximum number of students (Trefla 1999). However, the effectiveness of learning is a complex function of a student's individual aptitude as well as the instruction received. Cronbach and Snow's theory of Aptitude Treatment Interaction examines the effects of both on a student's learning; their findings imply that while this area is poorly understood, 'teaching to the middle level of the classroom' can be detrimental to both high and low ability students (Cronbach and Snow 1969; Snow 1988).

Elementary schools use age-graded classrooms as the primary way to limit individual ability differences within a single classroom, but many teachers and parents express dissatisfaction with this sytem, as age often misrepresents ability (Trefla 1999). At the middle- and high-school levels, many schools use tracking to limit the skill range within a classroom, and such policies have themselves generated considerable controversy. For example, in her influential 1985 book Keeping Track: How Schools Structure Inequality, Oakes argues that the social structure of ability groupings, concomitant with differential teacher enthusiasm for the various ability levels, creates a heirarchy within schools that is psychologically detrimental to students in the groups labeled 'low-ability' or 'slow' (Oakes 1985). Moreover, ability tracking does not provide a satisfactory solution to the underlying problem; enough heterogeneity remains within the ability groupings that many teachers feel the pedagogical issues are unresolved (Trefla 1999).

2 EPGY Trajectories

In the self-paced, distance-learning EPGY setting, students' individual differences can emerge and be fostered, and this happens in an environment that has minimal impact on their schools' social structures. The high degree and dimensionality of these differences is often surprising, especially since students are selected for the program only if they score within a narrow band of standardized test results (Cope and Suppes 2000). In order to allow expression of learning differences, the EPGY curriculum is highly individualized. For example, the courseware uses movement algorithms designed to allow students who demonstrate understanding of the material quickly to progress through that material rapidly, while students who are slower to comprehend a given concept are given hints and supplementary problems until they demonstrate mastery (Suppes and Tock 2002). As a result, individual students have very different experiences in the course. For example, the numbers of exercises that a student might do, when plotted versus average grade-placement, describe a cone of remarkable breadth, as shown in Figure 1 for Grades 3 and 6, which are taken to be representative. 'Average grade-placement' is calculated as the average placement in the five strands of the EPGY mathematics curriculum: Logic, Integers, Fractions, Measurement, and Geometry. Students' motion through the course within each strand is independent of motion within other strands.

Figure 1: Average grade-placement versus number of exercises completed: grades 3 (top) and 6 (bottom)

Of course, students' progress versus number of exercises done is rarely as smooth as Figure 1 implies. Different concepts are more or less difficult for different students to master. A student might speed through the lesson on 'Using the If-then-not Rule' at grade-placement 5.020 of the logic strand in only four exercises, but might require 10 or more exercises to demonstrate mastery of 'Concave and Convex 2D Figures', which occurs at grade-placement 5.260 of the geometry strand. As a result, students' trajectories through the courses are never a straight line. Indeed, the bumpiness and curvature is often quite pronounced. When average grade-placement is plotted versus "computer-time" spent working exercises on the EPGY course software, curves such as the representative trajectories of Figure 2 are the result.

Figure 2: Sample computer-time trajectories

A straight line is inadequate to fit these trajectories. Prior research has shown that a student's trajectory through a computer-based course can be described well by the power function

where t is time, y(t) is grade-placement as a function of time, c is the student's starting grade-placement in the course, and b and k are fit parameters (Suppes et. al. 1976). Suppes and Zanotti (1996) have applied this analysis to more than a thousand students working through similar mathematics courses from the Computer Curriculum Corporation, and have shown that this function usually describes students' progress through a computer-based curriculum rather well.

The parameter c is fixed at the grade-placement position at which the student started the course. The performance parameters b and k are both measures of the student's pace through the curriculum. When k is close to 1, b is the approximate fraction of the course completed per computer-time hour. In turn, k measures how curved a trajectory is. The average value of k for the EPGY computer-time trajectories in this study is 0.798, indicating that the students' rates of progress typically become slower as the course progresses toward more advanced material, which is more or less the expected result for most students in standard mathematics courses. Ninety-two percent of the computer-time trajectories in this study had a computer-time k parameter less than 1. By contrast, values of k higher than 1 mean that the student's rate increased during the course. The trajectory shown at the top in Figure 2 is taken from a student with k less than 1, and the trajectory at the bottom is taken from a student with k greater than 1.

2.1 Computer-Time Trajectories

Setting the course back.   Fitting values for k and b is complicated by the fact that students sometimes set their course back on the recommendation of their parents or, less often, their EPGY tutor. Even after fully completing a course, approximately 5% of students set their courses all the way back to the beginning to work through the course again. While the possibility of setting the course back is a distinctive strength of the program for students who wish to review, it does make trajectory fitting problematic. One such student's trajectory is shown in Figure 3. This student, an extreme case, repeated the sixth grade four times before deciding to move on to a more advanced level. In order to obtain meaningful trajectory parameter values, we use the first full trajectory to obtain k and b, as shown on the bottom of the figure. As the student's first pass through the course, this is most comparable to other students' trajectories.

Figure 3: Computer-time trajectory for a student who repeated grade 6 (top: raw data, bottom: fit of first trajectory)

Reviewing.   Other students review particular topics rather than revisiting the entire course. The student whose trajectory is shown in Figure 4 actively selected specific components of the course to review as she worked her way through the software. This is in addition to review exercises automatically presented by the software. The software-chosen review exercises are specially selected for the student, based on the student's estimated probability of having forgotten the relevant concept: they are sporadic and never constitute more than one or two exercises at a time.

Student-chosen mini-reviews, such as the one shown in Figure 4, are a relatively rare occurrence, affecting less than 10% of the trajectories in this study, but occasionally they do occur. The very slight tendency of females to engage in these mini-reviews more often than males is not statistically significant.

Figure 4: Computer-time trajectory for a student with false starts

Skipping.   Students occasionally skip portions of the course, as the trajectory of Figure 5 illustrates. This may occur when a student sees material that has been covered in the student's school curriculum or that the student is otherwise familiar with. Also, the possibility that a sequence of the student's email reports was lost or unsent cannot be overlooked. Students in EPGY communicate with their tutors a minimum of once per week to send the email reports, which are scrutinized by the tutors to track students' progress and ensure that they are benefitting from the program. Data are automatically attached to these emails by the EPGY software and put into the database. Occasionally, however, some report data may be lost due to computer errors. Because of this, it is difficult to diagnose trajectories with evidence of skipping. The number of skips per student averaged over all of the trajectories in this study is slightly higher for males than females (0.11 versus 0.08), but again, this result is not statistically significant.

Figure 5: Computer-time trajectories for students with skipping

Fitting algorithm.   It is found that fitting the trajectories by using a grid search results in closer fits than fitting them by using gradient descent algorithms, which are susceptible to local minima. Therefore, a grid search was used to obtain the fit values presented in this study. The algorithm searches the range of possible k's and b's on an increasingly fine grid: first with resolution 0.1 unit, then with resolution 0.01, and finally with resolution 0.001 in order to obtain fit values that are estimated to the nearest thousandth.

2.2 Calendar-Time Trajectories

It is often revealing to examine the trajectories not only in computer-time, but also in calendar-time. In other words, it is worthwhile to clock the grade-placement progress of students not only by how much time they have spent on the computer, but also by how many days have elapsed. The resulting calendar-time trajectories include the time between the day the student did the first exercise in the course and the day the student finished the last exercise. Once again, repeats of the grade, as in Figure 3, are removed from the trajectories.

In calendar-time, a student's progress is more complex due to the vagaries of scheduling and other schoolwork. Also, the trajectory parameters have a different meaning. Some students may be stimulated by the more difficult material that typically occurs at the end of a sequence and therefore be willing to put in extra hours per day, resulting in a high calendar-time k-value even though the corresponding computer-time k-value may be quite low. Others may be motivated to accomplish more per day by the desire to get to the end of a section before school vacation time or for other reasons. In general, the calendar-time b and k values are measures of student motivation, while the computer-time b and k values are more classical measures of student learning rates.

Figure 6: Sample calendar-time trajectories

Figure 6 displays two representative calendar-time trajectories. The top one has a value of k less than 1 and the bottom trajectory has k greater than 1. The horizontal distances between vertical groupings of points represent the numbers of days between sessions at the computer. Large horizontal distances with no data points often imply that the student was on vacation or busy with other coursework during that time, although there can be other reasons as well. Students who are accepted to EPGY typically have extremely full programs already, complete with academic and extracurricular activities that consume large amounts of time, so long horizontal stretches without activity are common (Suppes 1975). Indeed, such stretches often go together with periods of intense progress for students who prefer to complete their coursework in widely-spaced blocks. Figure 7 shows an extreme example of this.

Figure 7: Calendar-time trajectory for a student with intensive, widely-spaced blocks

3 Trajectory Parameters

Figure 8 shows an example of a computer-time and calendar-time trajectory for the same student in the same EPGY grade. The different units of time on the abscissa result in a drastic difference between the appearances of the two trajectories. This underlines the fact that the computer-time and calendar-time trajectory parameters are measuring constructs that have very different meanings. Roughly, the computer-time parameters measure the rate of learning in one sitting, while the calendar-time parameters measure motivation over time.

Figure 8: Computer-time trajectory and calendar-time trajectory for the same student

For either kind of trajectory, it is difficult to consider the k and b parameters in isolation from each other. The bt ^k term in Equation 1 links them inextricably. To understand how the parameters interact and how their subtle variations describe individual differences among students, the values of bt ^k for fixed values of t are tabulated below. This is the progress in units of grade-placement that can be expected over time t . Tables 1 and 2 show statistics for progress in the six grade levels with t fixed at 10 hours and 10 days, respectively. The approximate factor-of-two difference between the 75 ^th and 25 ^th percentiles highlights the strength of this individual difference in progress per time, and has been observed previously among EPGY students (Cope and Suppes 2000). Keeping students in lock step with one another, as is practiced in traditional school settings, suppresses these strong differences in individual learning rates.

It is also noteworthy that the magnitude of the differences is greater for the calendar-time data (Table 2) than for computer-time data (Table 1). This lends credence to the idea that student's varying schedules and priorities heavily influence the calendar-time trajectory. There is an approximate order-of-magnitude difference between the maximum and minimum in each category: again, this difference is greater in the rows of Table 2 than for all but one of the corresponding rows of Table 1.

Although it is problematic to discuss the k and b parameters separately because of their interaction, it is also somewhat artificial to hold time constant at 10 hours or 10 days. Looking at k and b separately yields a deeper perspective on the composition of the rate parameters of the preceding tables. The single and joint distributions of k and b are shown in Appendices A, B and C. Because of the large numbers of observations in these histograms, their chi² fit to any kind of a continuous distribution is quite poor, even when the observations are grouped so that there are approximately the same number in each bin. The distributions are almost all unimodal, and sufficiently asymmetric that a gamma distribution fits better than a normal for 25 of the 28 possible grade-variable combinations. That they are unimodal suggests that it is not possible to separate students into well-defined classes based on their k and b values. The asymmetry may be due to the impossibility of either variable taking on a value less than zero.

3 Individual Differences

3.1 The Multidimensionality of Individual Differences

Learning rates in computer-time and in calendar-time are not the only individual differences among the students of EPGY. In addition, students exhibit different error rates and average latencies for their coursework. Latency is defined as the interval in seconds between the presentation of an exercise and the student's response to that exercise. A student's average latency for correct answers is indicative of how many seconds, on average, it takes the student to arrive at the correct solution to an exercise. Tables 3 and 4 show statistics for average latency for correct answers and error rates, respectively. Once again, the characteristic factor-of-two differences between the upper and lower quartiles are observed, with the magnitude of the difference being in all cases greater for error rates than for latencies.

Latency and error rate have a complex relationship that appears to shift as grade in school changes. Students in the upper quartile of average latency are overall slower to respond to a given exercise than their peers in the lower quartile of average latency. If we plot the sample path of errors for students in the overall upper latency quartile versus the sample path of errors for students in the overall lower latency quartile, we find that the overall slower students make more errors than their peers in the early grades and fewer errors than their peers in the later grades, as shown in Figure 9.

Figure 9: Sample paths of errors for students in the upper quartile of overall latency (red) versus students in the lower quartile of overall latency (blue)

A fifth difference among students is age. While not an explicit measure of performance, age in a given grade is often taken as representative of intellect. In the Palo Alto Unified School District, the earliest age at which a student may begin the first grade is 5 years and 9 months old (PAUSD 2002). Most school districts in the United States have similar restrictions. As is evident from Table 2, however, gifted students left to their own devices can progress far more quickly than one grade level per year in mathematics. This generates the dissatisfaction with age-graded classrooms noted in the introduction. It has been remarked by several authors that consecutive years of early mathematics instruction can probably be compressed in time more easily than any other substantial part of the curriculum. Corroborating this, many EPGY students complete grade levels far ahead of the traditional age. As would be expected, the earlier a student begins the EPGY curriculum, the farther ahead that student is likely to be as time goes on. For a discussion of this point, see Appendix D.

Table 5 shows statistics for the ages of students in this study when they began the corresponding grade of the curriculum. The values of n in this table are occasionally smaller than in the previous tables because EPGY birthdate data are incomplete. As is clear from the data, many students are working far in advance of the grade level where they would be placed based on age alone. The factor by which the maximum and minimum age for each grade differ is strictly greater than two in all cases.

In summary, the five dimensions along which students can differ for this study are computer-time rate bt^k, calendar-time rate bt^k, error rate, average latency for correct answers, and age. The distributions of these parameters across the EPGY students in this study for each grade level are shown in Appendix E.

3.2 The Robustness of Individual Differences

Although the five variables that we have considered vary widely between students, they tend to remain relatively constant for a given individual, as demonstrated by the high correlation between these parameters from one grade to the next. In Figures 10-13, each student's deviation from the mean in grade n versus grade (n+1) is plotted. The preponderance of data in the first and third quadrants indicates that students' positions relative to the mean do not tend to change sign between grade levels. Especially in the cases of latency and error rate, the best predictor of future performance is past performance. It is unlikely that students with a high error rate in grade n will improve significantly relative to other students in grade (n+1) or that students with high latency in one grade will speed up in the next, despite substantial differences in course content and structure. In other words, not only are these variables predictive for a given student, but also the student's position in the pool of EPGY students with respect to these parameters tends to stay relatively fixed.

Figure 10: Relative latency from one grade to the next	Figure 11: Relative error rate from one grade to the next
Figure 12: Relative calendar-time progress from one grade to the next	Figure 13: Relative computer-time progress from one grade to the next

4 The Undominated Set

We have discussed five distinct individual differences that have been identified for students of EPGY: computer-time rate of grade-placement progress, calendar-time rate of grade-placement progress, error rate, average latency for correct answers, and age. To further probe these differences, a partially ordered list of students was generated. A partially ordered list is an asymmetric ordering of students such that Student B is listed after Student A if and only if Student B is weaker than Student A in all five difference dimensions and strictly weaker in at least one. Note that to surpass Student B, Student A must have a lower latency, error rate, and age, and a higher rate of progress with respect to computer-time and calendar-time.

A student is 'undominated' if and only if no other student surpasses that student in all five dimensions. The fractions of undominated students in each grade level are shown in Table 6. The third column shows the fraction of students who were undominated when age was not included as a dimension. Of the five dimensions, age was the one whose inclusion resulted in the most formerly dominated students becoming undominated. Thus, contrary to much popular belief, completing a mathematics curriculum at a young age does not imply, even probabilistically, that the student has a particularly low latency, error rate, or rate of progress.

The sizable fractions of undominated students in Table 6 illustrate that a strong relative showing in one dimension of mathematical ability does not necessarily imply an equally strong relative showing in other dimensions. Students have different strengths, and mathematical ability cannot be measured along one dimension alone. This is even more dramatically illustrated by the data in Table 7, which shows statistics for the numbers of students by whom a given student is dominated. Even the student who is dominated by the maximum number of other students in each grade is undominated with respect to about half of the other students. Also notice that the numbers for even the upper quartile are very low. Students who are dominated are typically dominated by very few other students. Moreover, the partially-ordered lists of students above individual students who are dominated are typically only few levels deep. An example is shown in Figure 14, and additional examples are displayed in Appendix F. In Figure 14, Student S is dominated by thirteen other students in three layers.

Figure 14: Partial ordering for students dominating student S

An example of a partially ordered list for an entire grade is shown in Figure 15. While this list is too complicated to invite a detailed visual interpretation, it is clear that it has only seven layers of students. A student might be dominated by more than six other students, but still the heirarchy is only seven layers deep. (For simplicity, only connections between adjacent layers are shown.)

Figure 15: Partially ordered list of students in grade 2

If we decrease the number of dimensions on which we allow students to differ, the depth of the heirarchy increases. Figure 16 is a plot of how the maximum depth of the heirarchy changes when additional dimensions are added. If students are rank-ordered in one dimension of ability, the maximum number of ranks is very close or equal to the number of students. Recognizing multiple dimensions of ability causes the number of possible ranks to fall dramatically; consequently, rank ordering becomes less meaningful as a procedure for separating students. Additional partial orderings for entire grades are shown for various numbers of dimensions in Appendix G.

Figure 16: Partially ordered list heirarchy depths

5 Conclusion

When gifted students' individual differences are allowed to emerge in an individualized curriculum such as EPGY's, the results are often surprising. A factor of two or more difference between students in the upper and lower quartiles is evident in many measurable aspects of learning, even for this restricted population of students. Permitting students to be at very different places in the curriculum at a given time, working through material at a pace and on a schedule most suited to their individual learning styles, results in students advancing beyond their years. This could not feasibly happen in a classroom; as we have shown, gifted students' individual differences are multidimensional and robust, and not amenable to classroom standardization or one-dimensional analysis. Careful use of technology can enable gifted students to learn according to their own optimal needs and strengths, and removal from the social hierarchy and gendered and racial expectations within a school environment may also function as a positive factor for some.

Thorough analysis of the large number of student records in EPGY's Oracle database supports the ongoing development and refinement of individual courses. We have focussed on showing that individual differences in mathematical ability stand out particularly when that ability is separated into components. In addition, such analysis increases our understanding of how students individually learn with computers, how best to challenge them, and how to insprire them to achieve.

6    References

] Palo Alto Unified School District website: "Registration Requirements". Retrieved June 23, 2002 from the World Wide Web: http://www.pausd.palo-alto.ca.us/attendance/registration.html.

Ravaglia, Raymond, Theodore Alper, Marianna Rozenfeld, and Patrick Suppes. "Successful Pedagogical Applications of Symbolic Computation." Reprinted from Kajler, N, Computer-Human Interaction in Symbolic Computation, Springer-Verlag, 1999. Retrieved July 2, 2002 from the World Wide Web: http://epgy.stanford.edu/research/index.html?papers.

Snow, Richard E. "Aptitude-Treatment Interaction as a Framework for Research on Individual Differences." Center for Education Research at Stanford. 88-CERAS-16. March 1988.

Stillinger, Constance and Patrick Suppes. "Gifted Student' Individual Differences in Computer-Based Algebra and Precalculus Courses." Retrieved June 23, 2002 from the World Wide Web: http://epgy.stanford.edu/research/index.html?papers.

Suppes, Patrick. "Impact of Computers on Curriculum in the schools and Universities. Computers in Education, O. Lecarme and R. Lewis (eds.) North Holland Publishing, p. 173. 1975.

Suppes, P. J. D. Fletcher, and M. Zanotti. "Models of Individual Trajectories in Computer-Assisted Instruction for Deaf Students." Journal of Educational Psychology, Vol. 68, No. 2, 117-127. 1976.

Suppes, Patrick and Kalee Tock. "EPGY Motion Algorithm." In Preparation.

Suppes, Patrick and Mario Zanotti. "Mastery Learning of Elementary Mathematics: Theory and Data." Foundations of Probability with Applications New York: Cambridge University Press, pp. 149-188. 1996.

Trefla, Douglas. "Individual Differences and the United States Education System," Chapter 3 in The Educational System in the United States: Case Study Findings, March 1999. National Institute on Student Achievement, Curriculum, and Assessment Office of Educational Research and Improvement. U.S. Department of Education. Retrieved May 6, 2002 from the World Wide Web: http://www.ed.gov/pubs/USCaseStudy/chapter3.html.