Unpacking TPACK in Mathematics Education Research: A Systematic Review of Meta-Analyses

: Teaching with technology is considered a necessity in the U.S. mathematics classroom. However, few studies have established explicit considerations to support technology-enhanced student achievement. The purpose of this study was to characterize the effectiveness of technology in the mathematics classroom by systematically reviewing meta-analytic research. An exhaustive literature search was conducted. After applying a prioi inclusion criteria the pool of 65 initial meta-analyses was reduce to 13 representative studies. Each study was reviewed and characteristics were coded in four categories: (1) sample, (2) measurement, (3) design, and (4) source. An inductive review of the coded studies produced five unique moderators that were the most salient across studies. Overall mean effect sizes were retrieved or calculated from available study data. Hedges g was used as the common effect size metric for comparison across studies. The Technological Pedagogical Content Knowledge (TPACK) framework was used to interpret the most salient moderators of effects across studies. Studies were categorized by didactical functionality and technology type. The results suggest that effects vary by didactical functionality from small to medium. The largest variations were observed for the didactical function of developing conceptual understanding. Implications for research and instructional praxis are provided.


Introduction
Technology is an essential element of instruciton across the world.Schools across the world have made considerable increases in their technology infrastructure, as well as the development of educational technology (Alavi & Leidner, 2001;Russell, Bebell, O'Dwyer, & O'Connor, 2003).These increases were made to support the achievement of students in the classrom.Substantial increases were made to support mathematics content learning.Despite these investments some students students remain underprepared in mathematics.
Research supports technology as a facilitator of instructional change (Roschelle, Pea, Hoadley, Gordin, & Means, & 2000;Sandholtz, Ringstaff, & Dwyer, 1999), but a closer investigation reveals that contextual factors may mediate some of these changes (Cuban, Kirkpatrick, & Beck, 2001;Windschitl & Sahl, 2002;Zhao, Pugh, Sheldon, & Byers, 2002).These factors include: training, adminstrative support, and teacher attitudes towards technology.The proliferation of educational technology in the United States has provided teachers with more electronic resources than ever before, but some teachers have not received sufficient training in the effective use of technology to enhance learning (Niess, 2005).A U.S. survey of technology implementation in mathematics classrooms found that almost half of American students are in classrooms where teachers lack access to district or school provided professional development on the use of computer technology for mathematics instruction (Mitchell, Bakia, & Yang, 2007).This lack of training could be attributed to the assumption that the technology tool was the primary factor in the integration of technology.The effects of technology on instruction can be dependent on contextual factors beyond the user of the tool.Dexter, Anderson, and Becker (1999) found that teachers cited reflection on experience, classes taken, and the context or culture of the school as the major catalyst of instructional change when technology is introduced.Assessing the effects of these and other factors on student achievement in technology enhanced mathematics classrooms is important.
The thoughtful integration of technology in the mathematics classroom is important because it supports student success in mathematics (National Council of Teachers of Mathematics, 2000).Thus, these effects should be maximized.The debate over the impact of technology in the classroom often isolates the technological tool as the primary catalyst for improved teaching and learning (Watson, 2001).Technology however, is not a catalyst for instructional change, but is a tool, much like a chalkboard or any of the other instructional tool.Because different technologies have different affordances and constraints, technology alone cannot be credited with improved teaching and learning (John & Sutherland, 2005;Webb, 2005).In order to improve the teaching and learning of mathematics with technology it is imperative that theoretical constructs are refined through empirical specification, which in turn should guide classroom applications.Examing the results of meta-analytic research is one means to examine these constructs and draw conclusions related to effective teaching and learning with technology in the mathematics classroom.The purpose of this study was to characterize the effectiveness of technology in the mathematics classroom by systematically reviewing meta-analytic research.

Literature Review
The Technological Pedagogical Content Knowledge (TPACK) framework provides practical, empirical, and theoretical considerations for the integration of technology in the mathematics classroom.Pierson (2001) suggest that effective technology integration can be defined as the intersection of technological knowledge (TK), pedagogical knowledge (PK), and content knowledge (CK).This type of knowledge is especially important for mathematics teachers because of the complex nature of mathematic content, pedagogy, and associated instructional technologies.The concept of TPACK is an extension of Pedagogical Content Knowledge (PCK) conceptualized by Shulman in the mid 1980's (Shulman, 1986).Technological knowledge, content knowledge, and pedagogical knowledge all afford and constrain one another.These affordances and constraints take place at the intersections of all these different types of knowledge.Thus, Mishra and Koehler (2008) suggest that the intersection of PCK, TCK, and TPK is TPACK and this type of knowledge is vitally important for teaching with technology.
There were several unique features of TPACK that suggested it should be considered in the development of a model for technology integration.The different sets of knowledge and skills that TPACK encompasses requires an understanding multiple representations of concepts using technologies; constructive pedagogical techniques that apply differentiated instructional technologies to meet the needs of all students; knowledge of nuances of particular content areas that make them difficult for students to comprehend and how technology can assist with student acquisition of the concepts; knowledge of scope and sequence of content and epistemological assumptions; and knowledge of how technologies can be scaffold student content knowledge (Harris, Mishra, Koehler, 2007).The potential of the TPACK framework is further supported by the U.S. Association of Mathematics Teacher Educators (AMTE), who have developed an explicit mathematics TPACK framework.This framework presents four components for enhancing mathematical learning.According to AMTE (2009) these components are: (1) designing and developing technologyenhanced learning experiences, (2) facilitating technology-integrated instruction, (3) evaluating technology-intensive environments, and (4) continuing to develop professional capacity in mathematics TPACK.This mathematics content specific framework guides the present study.In the next section a rationale is presented for reviewing the results of previous meta-analysis.
Reviewing previous meta-analyses can help identify best practices with technology in the classroom.Meta-analysis uses study effect sizes to generate empirical conclusions from similar studies.A meta-analysis has to common functions: (1) to summarize large numbers of studies in terms of effect sizes, and (2) to make inferences concerning the between group variances in studies (Kemery, Mossholder, & Dun lap, 1998).This process involves calculating the average effect size, testing homogeneity, and detecting moderators to explain the heterogeneity (Sanchez-Meca & Martin-Martinez, 1998).The detection of moderators represents the key feature of any met-analytic study; because this is where the difference in strength and direction in effect sizes is identified.Rosenthal expounds, "The search for moderators is not only an exciting intellectual enterprise but indeed…it is the very heart of scientific enterprise" p. 447).Thus, the use of a meta-analytic lens is often considered appropriate when evaluating prior research.
Meta-analytic allows researcher to make better decisions concerning technology integration in the mathematics classroom.Meta-analysis helps researchers identify specific variables that account for the variance in the effectiveness of technology integration in the mathematics classroom by assessing moderators.Moderators are quantitative of qualitative variables that influence the strength or direction of relationships in meta-analytic research (Shadish & Sweeney, 1998).Moderators are important because they cause statistical interactions, but associations do not mean causation (Cooper & Patall, 2009;Russell & Gilliland, 1995).Interpreting effect differences across studies is based on assessing the influence of moderators (Lipsey, 2003).Moderators are categorized as either: (1) methodological variations, (2) theoretical constructs, or (3) study characteristics (DeCoster, 2004).Methodological variations refer to components of the experimental design such as sample size, random assignment, or treat duration.Theoretical constructs are moderators grounded in theory or based on the application recognized theoretical trends.
The final category of moderator variables reflects study related artifacts such as publication status or publication year.Moderators are historically recognized for their ability to enhance theory development and increase the overall richness of empirical work (Sharma, Durand, & Gur-Arie, 1981).
Theoretical construct moderators are the focus of this systematic review.This study focuses on constructs related to the didactical functions of technology in the mathematics classroom.Drijvers, Boon, and Van Reeuwijk (2010) established three main didactical functionalities for digital technology use in the mathematics classroom: (1) doing mathematics, (2) practicing skills, and (3) conceptual understanding.Doing mathematics refers to the use of technology to complete tasks that could be done by hand.This enhances computational efficiency, which can free the student from arduous calculations to address more realistic problems.While practicing skills refers to more repetitive tasks that require immediate feedback to develop procedural or instrumental understanding.Finally, when technology is used to develop concepts it is associated with building deeper understandings and making connections.This is similar to relational mathematics understanding.Technological tools from previous meta-analytic research were categorized using these three didactical functionalities in the next section.

Calculators and Computational Efficiency
Calculators and other handheld computation devices afford teachers the ability to ask and students the ability to complete more complex questions.Before calculators were introduced into the mathematics classroom teachers were severely limited in their ability to present realistic problems, because the computations were often unnecessarily tedious or irrelevant to the instructional outcome.The ability to enhance computational efficiency in the mathematics classroom is one of the most significant instructional affordances.Thus, calculators are categorized as computational enhancement devices.
Calculator use in the mathematics classroom was once one of the most divisive issues in mathematics education research and policy.However, now calculators are recognized as indispensable pedagogical tools that continue to evolve in computational as well as representational prowess.For instance, different varieties of handheld calculators are emergent, ranging from simple arithmetic calculators to scientific calculators, graphing calculators, and symbolic calculators with a variety of calculating modes, including algebraic systems and spreadsheets (Close, Oldham, Shiel, Dooley, & O'Leary, 2012).Calculators foster students' higher-order thinking skills and motivation (Phillips-Bey, 2004), which historically has not been widely accepted.The National Council of Teachers of Mathematics (NCTM) added that calculators are fundamental technologies in mathematics classrooms that enrich student understanding (NCTM, 2000).Appropriately, the results of prior meta-analysis on the effects of calculators on mathematics achievement were instrumental in this switch in popular thought.

Computer Assisted Instruction and Instrumental Understanding
Increasing the ability of teachers to individualize instruction is a significant affordance of technology.Computerassisted instruction (CAI) and computer-based instruction (CBI) are two of the most well researched forms of technology-enhanced instruction present in the mathematics education literature.Together these tools allow teachers to provide individualized self-paced instruction.This type of instruction support practice and the development of instrumental understanding.For the purpose of this discussion these tools are categorized together as instructional delivery tools.Although many teachers use them as supplemental instructional modules, CAI and CBI are often utilized as individual instructional systems.However, they typically involve procedures rather than conceptual understanding.Consequently, for decades' studies have investigated the effects of CBI and CAI on student achievement in general and in mathematics education specifically (Azevedo & Bernard, 1995;Clark, 1985;Yung & Paas, 2015).Computer-based instruction is defined as the use of computers in the delivery of instruction (Coulson, 1968).Computer-assisted instruction is a more precise term, often referring to the use of computers in drill and practice, tutorials, or simulation activities offered in substitution or as a supplement to traditional, teacher-directed instruction (Hicks & Holden, 2007).Despite their nuances, within the academic literature CAI and CBI have been recognized as technology-enhanced forms of instruction, typically operationalized as learning delivered primarily by means of the computer, that typically incorporate drill and practice, simulations, and well-defined feedback mechanisms.Furthermore, in many instances CBI and CAI have been used interchangeably within prior meta-analyses in mathematics education research, and thus, they are discussed as one unit here.

Mathematics Specific Software and Relational Understanding
Mathematics specific software applications afford teachers with the ability to present abstract ideas in visually appealing virtual environments.These applications are often teacher directed or facilitated and tend to focus on relational understanding.Thus, the tools are primarily utilized to increase student conceptual understanding by mathematical modeling and simulation.This category of technology enhancement captures the mathematics specific software applications that are used by teachers to build connections and develop conceptual rather than procedural understanding.Mathematics software and applications have emerged in the specific forms of digital geometry software (DGS), virtual manipulatives, and more generally as mathematics-specific educational software.
Calculators, CAI/CBI, and mathematics specific software application all have unique affordances and constraints.These three categories of technologies and their didactical functionalities are examined in this study through the lens of the TPACK framework.The overall effect sizes and moderators of student achievement results are also examined to inform mathematics instruction with technology.

Research Goal
The synthesis of this research will help compare and contrast different conceptualizations of learning with technology, how it was measured in mathematics classrooms, and identifies common and generalizable findings across the metaanalyses with regard to the effectiveness of technology.
Our research questions were: 1. How are the mean effects of previous meta-analyses characterized by technology type and didactical functionality?
2. What TPACK related moderators are most salient in previous meta-analyses?

Data Collection
The primary list of meta-analysis studies was generated from a comprehensive literature search of articles written between 2001 and 2015.The year 2001 was chosen because Pierson wrote the first article that refers to TPCK/TPACK in 2001.Iterative electronic searches were made using educational databases (JSTOR, ERIC, EBSCO, Pych INFO, and Proquest).An initial pool of studies was located using different of keywords (meta-analysis, research synthesis, literature review, literature synthesis, mathematics, achievement, technology, instructional technology, Information Communication Technology).Citations from the initial pool of studies were then reviewed to identify any potential meta-analysis not previously located.There were 65 studies identified for preliminary review as a result of the aforementioned literature search procedures.Figure presents the entire inclusion and exclusion process based on the criteria presented below.
To be included in this research synthesis, the following inclusion criteria were established.
1.The study examined the effects of digital technology applications, including computer-assisted instruction, integrated tutoring systems, technology based programs, or the use of technological tools to improve mathematics achievement.
2. The studies employed meta-analytic methods to calculate mean effect sizes and identify moderator variables.
3. The studies involved students in K-12.
4. The primary dependent measure included quantitative measures of mathematics performance such as standardized test, researcher made test, or teacher made test.
5. Studies were conducted between 2001 and the present.The effect sizes for calculator use tend to converge at the moderate level of effectiveness, based on effect size benchmarks proposed by Cohen (1992): .20 (small), .50(medium), and (.80) large.The statistically significant moderators of calculator effects on mathematics achievement established in the literature are grade level and assessment.This is not surprising given that the early calculator use is controversial in U.S. For example, on the 2009 U.S. National Assessment of Educational Progress (NAEP), 66% of fourth graders said they never used a calculator for exams or quizzes, compared to only 28% of eighth graders on the survey who claimed to never use them on exams or quizzes (National Center of Educational Statistics [NCES], 2009).Grade level and assessment are considerations associated with the technological pedagogical affordances and constraints of calculators.
Hence, assessment and grade level should be strongly considered when implementing calculators in mathematics instruction.TPK helps teachers to design lessons and activities that use technology to assist in the acquisition of the content (Young & Young, 2012).In early mathematics classroom computation is a key learning objective and its assessment is drastically altered when calculators are used.However, there is a balance that must be maintained because realistic problems are often computationally rigorous, which can be a limitation of calculators are not introduced in the early grades.
The results of prior meta-analyses have suggested that the effects of CBI/CAI on mathematics achievement vary from small to medium.Studies were conducted across grade levels and various mathematics content strands.Prior metaanalyses of CBI/CAI studies have also consistently concluded that duration and instructional modality, were statistically significant moderators of effect size.Duration is a TPK consideration.The appropriate duration must be determined based on the development appropriateness and learning objective.The results of prior meta-analyses suggest that the duration of use can differentiate the effects of CAI/CBI.This is understandable given the primary didactical functionality is practice.Hence, the time a student spends practices mathematics skills influences achievement.Instructional modality is a more comprehensive moderator that reflect the larger TPACK construct.To determine the modality of CAI/CBI requires one to consider the intersections of PCK, TCK, and TPK.Unfortunately, TPACK is consistently the least stable construct to assess across in-service and pre-service teachers in prior research (Young, Young, & Hamilton, 2013).The instructional modality is vital to the use of CAI/CBI as a tool to enhance mathematics teaching and learning, but more work is needed to establish more stable in the across the measurement of the TPACK construct to support instruction.
The results of prior meta-analysis for this category of technology-enhanced mathematics learning are by far the most divergent.The overall effect sizes range from 0.09 to 1.02.One explanation is that unlike the other didactical functionalities, tools to support relational understanding are more subject-matter specific (algebra, geometry, etc.) which can contribute to the divergence.For example, mathematics software applications and virtual manipulatives are two common types of technology in this category.Software applications range from numeral modeling to modeling in calculus.While virtual manipulatives cover multiple subject matter areas and are used to address multiple concepts.Across this body of literature, the consistent statistically significant moderators of effect sizes were grade level, duration, and mathematics subject matter.Thus, indicating that the divergence in effect sizes across these studies may be attributed to these aforementioned moderators.The unique moderator for this category was mathematics subjectmatter.This category requires teachers to consider TCK.TCK is an important consideration because it supports teacher making and skills related to choosing appropriate technologies to support content learning.Although many of the aforementioned tools are content specific in nature, the teacher is still required to assess the affordances and constraints of the tools in relation to the learning objective and classroom context.

Limitations
The three didactical functionality and categories of technological tools represent the most common approaches to technology integration in the mathematics classroom.Summarizing these results by characterizing moderators and effects across meta-analyses does not come without some empirical limitations.Given that much of the moderator data reside at the individual study level, a precise estimation of the exact influences of all moderators assessed in prior meta-analyses would be difficult to feasibly examine through systematic review.Thus, a representative sample of moderators that could be assessed at the meta-analysis rather than study level was selected for inclusion in this study.Additionally, the didactical functionalities of the technological tools are not exhaustive nor mutually exclusive, as many of the tools have complex functions and multiple adaptations.This is an important consideration; however, the categories were applied to offer broad pedagogical characterizations based on the primary functionality as established in the literature.