The screening effectiveness of a chemical similarity search depends on a range of factors, includ... more The screening effectiveness of a chemical similarity search depends on a range of factors, including the bioactivity of interest, the types of similarity coefficient and fingerprint that comprise the similarity measure, and the nature of the reference structure that is being searched against a database. This study introduces the use of cross-classified multilevel modelling as a way to investigate the relative importance of these four factors when carrying out similarity searches on the ChEMBL database. Two principal conclusions can be drawn from the analyses: that the fingerprint plays a more important role than the similarity coefficient in determining the effectiveness of a similarity search, and that comparative studies of similarity measures should involve many more reference structures than has been the case in much previous work.
Kelley at al. argue that group-mean-centering covariates in multilevel models is dangerous, since... more Kelley at al. argue that group-mean-centering covariates in multilevel models is dangerous, since— they claim—it generates results that are biased and misleading. We argue instead that what is dangerous is Kelley et al.'s unjustified assault on a simple statistical procedure that is enormously helpful, if not vital, in analyses of multilevel data. Kelley et al.'s arguments appear to be based on a faulty algebraic operation, and on a simplistic argument that parameter estimates from models with mean-centered covariates must be wrong merely because they are different than those from models with uncentered covariates. They also fail to explain why researchers should dispense with mean-centering when it is central to the estimation of fixed effects models—a common alternative approach to the analysis of clustered data, albeit one increasingly incorporated within a random effects framework. Group-mean-centering is, in short, no more dangerous than any other statistical procedure, and should remain a normal part of multilevel data analyses where it can be judiciously employed to good effect.
It is claimed the hierarchical-age–period–cohort (HAPC) model solves the age–period–cohort (APC) ... more It is claimed the hierarchical-age–period–cohort (HAPC) model solves the age–period–cohort (APC) identification problem. However, this is debateable; simulations show situations where the model produces incorrect results, countered by proponents of the model arguing those simulations are not relevant to real-life scenarios. This paper moves beyond questioning whether the HAPC model works, to why it produces the results it does. We argue HAPC estimates are the result not of the distinctive substantive APC processes occurring in the dataset, but are primarily an artefact of the data structure—that is, the way the data has been collected. Were the data collected differently, the results produced would be different. This is illustrated both with simulations and real data, the latter by taking a variety of samples from the National Health Interview Survey (NHIS) data used by Reither et al. (Soc Sci Med 69(10):1439–1448, 2009) in their HAPC study of obesity. When a sample based on a small range of cohorts is taken, such that the period range is much greater than the cohort range, the results produced are very different to those produced when cohort groups span a much wider range than periods, as is structurally the case with repeated cross-sectional data. The paper also addresses the latest defence of the HAPC model by its proponents (Reither et al. in Soc Sci Med 145:125–128, 2015a). The results lend further support to the view that the HAPC model is not able to accurately discern APC effects, and should be used with caution when there appear to be period or cohort near-linear trends.
Urbanization has long been seen by scholars and policymakers as a disruptive process that can con... more Urbanization has long been seen by scholars and policymakers as a disruptive process that can contribute to social and political unrest, yet there is little cross-national quantitative empirical research on the topic. In this paper we provide a comprehensive analysis of the links between urban geography and the incidence of protests (i.e. demonstrations, riots and strikes) in African countries since 1990. In contrast to previous studies, we are careful to distinguish between urban population scale effects, urban population ratio effects, population rate-of-change effects and urban population distribution effects. We also provide an explicit test of the long-standing hypothesis that ‘over-urbanization’ increases the risk of civil unrest. Employing multilevel negative binomial models that control for key political and economic variables we find that urban population size and the number of large cities in a country are both positively and significantly associated protest incidence. By contrast, we find that a country's level of urbanization is negatively associated with protest incidence and reject the over-urbanization hypothesis: higher levels of urbanization are associated with less frequent protests at all income levels. We find no evidence that the pace of urban population growth or urban primacy significantly influence protest mobilization. In sum, our results provide a nuanced picture of the relationship between urban geography and protest incidence that challenges conventional wisdom and contemporary hyperbole about the dangers of ‘rapid urbanization’ in Africa in particular, and developing countries more generally.
This article challenges Fixed Effects (FE) modelling’s status as the ‘default option’ when using ... more This article challenges Fixed Effects (FE) modelling’s status as the ‘default option’ when using time-series-cross-sectional and panel data. We argue that understanding the difference between within- and between-effects of predictor variables is important when considering what modelling strategy to use. The downside of Random Effects (RE) compared to FE modelling – correlation between lower-level covariates and higher-level residuals - is a case of omitted variable bias, readily solvable using a variant of Mundlak’s (1978a) formulation. Consequently, RE modelling provides everything that FE modelling promises, and more. It allows time-invariant variables to be modelled, more parsimoniously than Plümper and Troeger’s (2007) suggested method. It is also readily extendable to Random Coefficients Models, allowing reliable, differential effects of variables without heavy parameterisation, the use of cross-level interactions between time-variant and invariant variables, and the modelling of complex variance functions. We are arguing not simply for technical solutions to endogeneity, but for the substantive importance of modelling context, and RE models’ ability to do so. Two empirical examples show that failing to do this can lead to misleading results. This paper is distinctive in stressing the substantive interpretations of within- and between-effects. This has implications beyond political science, to all datasets with multilevel structures.
There is ongoing debate regarding the shape of life-course trajectories in mental health. Many a... more There is ongoing debate regarding the shape of life-course trajectories in mental health. Many argue the relationship is U-shaped, with mental health declining with age to mid-life, then improving. However, I argue that these models are beset by the age-period-cohort (APC) identification problem, whereby age, cohort and year of measurement are exactly collinear and their effects cannot be meaningfully separated. This means an apparent life-course effect could be explained by cohorts. This paper critiques two sets of literature: the substantive literature regarding life-course trajectories in mental health, and the methodological literature that claims erroneously to have ‘solved’ the APC identification problem statistically (e.g. using Yang and Land’s Hierarchical APC – HAPC – model). I then use a variant of the HAPC model, making strong but justified assumptions that allow the modelling of life-course trajectories in mental health (measured by the General Health Questionnaire) net of any cohort effects, using data from the British Household Panel Survey, 1991-2008. The model additionally employs a complex multilevel structure that allows the relative importance of spatial (households, local authority districts) and temporal (periods, cohorts) levels to be assessed. Mental health is found to increase throughout the life-course; this slows at mid-life before worsening again into old age, but there is no evidence of a U-shape – I argue that such findings result from confounding with cohort processes (whereby more recent cohorts have generally worse mental health). Other covariates were also evaluated; income, smoking, education, social class, urbanity, ethnicity, gender and marriage were all related to mental health, with the latter two in particular affecting life-course and cohort trajectories. The paper shows the importance of understanding APC in life-course research generally, and mental health research in particular.
This paper reanalyses data used by Reinhart and Rogoff (2010c - RR), and later Herndon et al. (20... more This paper reanalyses data used by Reinhart and Rogoff (2010c - RR), and later Herndon et al. (2013) to consider the relationship between growth and debt in developed countries. The consistency over countries and the causal direction of RR’s so called ‘stylised fact’ is considered. Using multilevel models, we find that when the effect of debt on growth is allowed to vary, and linear time trends are fully controlled, the average effect of debt on growth disappears, whilst country-specific debt effects vary significantly. Additionally, countries with high debt levels debt appear more volatile in their growth rates. Regarding causality, we develop a new method extending distributed lag models to multilevel situations. These models suggest the causal direction is predominantly growth-to-debt, and is consistent (with some exceptions) across countries. We argue RR’s findings are too simplistic, with limited policy relevance, whilst demonstrating how multilevel models can explicate realistically complex scenarios.
This commentary clarifies our original commentary (Bell & Jones, 2014c) and illustrates some conc... more This commentary clarifies our original commentary (Bell & Jones, 2014c) and illustrates some concerns we have regarding the response article in this issue (Reither et al., 2015). In particular, we argue that (a) linear effects do not have to be produced by exact linear mathematical functions to behave as if they were linear, (b) linear effects by this wider definition are extremely common in real life social processes, and (c) in the presence of these effects, the Hierarchical Age Period Cohort (HAPC) model will often not work. Although Reither et al. do not define what a ‘non-linear monotonic trend’ is (instead, only stating that it isn’t a linear effect) we show that the model often doesn’t work in the presence of such effects, by using data generated as a ‘non-linear monotonic trend’ by Reither et al. themselves. We then question their discussion of fixed and random effects before finishing with a discussion of how we argue that theory should be used, in the context of the obesity epidemic.
Reither et al. (2009) use a Hierarchical Age-Period-Cohort model (HAPC - Yang & Land, 2006) to as... more Reither et al. (2009) use a Hierarchical Age-Period-Cohort model (HAPC - Yang & Land, 2006) to assess changes in obesity in the USA population. Their results suggest that there is only a minimal effect of cohorts, and that it is periods which have driven the increase in obesity over time. We use simulations to show that this result may be incorrect. Using simulated data in which it is cohorts, rather than periods, that are responsible for the rise in obesity, we are able to replicate the period-trending results of Reither et al. In this instance, the HAPC model misses the true cohort trend entirely, erroneously finds a period trend, and underestimates the age trend. Reither et al.’s results may be correct, but because age, period and cohort are confounded there is no way to tell. This is typical of age-period-cohort models, and shows the importance of caution when any APC model is used. We finish with a discussion of ways forward for researchers wishing to model age, period and cohort in a robust and non-arbitrary manner.
This paper: (a) finds rankings of who are the best formula 1 (F1) drivers of all time, conditiona... more This paper: (a) finds rankings of who are the best formula 1 (F1) drivers of all time, conditional on team performance; (b) quantifies how much teams and drivers matter; and (c) quantifies how team and driver effects vary over time and under different racing conditions. The finishing position of drivers is used as the response variable in a cross-classified multilevel model that partitions variance into team, team-year and driver levels. These effects are then allowed to vary by year, track type and weather conditions. Juan Manuel Fangio is found to be the greatest driver of all time. Team effects are shown to be more important than driver effects (and increasingly so over time), although their importance may be reduced in wet weather and on street tracks.
BACKGROUND
Whilst some argue that a solution to the age-period-cohort (APC) ‘identification prob... more BACKGROUND
Whilst some argue that a solution to the age-period-cohort (APC) ‘identification problem’ is impossible, numerous methodological solutions have been proposed, including Yang and Land’s Hierarchical-APC (HAPC) model: a multilevel model considering periods and cohorts as cross-classified contexts in which individuals exist.
OBJECTIVE
To assess the assumptions made by the HAPC model, and the situations in which it does and does not work.
METHODS
Simulation study. Simulation scenarios assess the effect of (a) cohort trends in the Data Generating Process (DGP) (compared to only random variation), and (b) grouping cohorts (in both DGP and fitted model).
RESULTS
The model only works if either (a) we can assume that there are no linear (or non-linear) trends in period or cohorts, (b) we control any cohort trend in the model’s fixed part and assume there is no period trend, or (c) we group cohorts in such a way that they exactly match the groupings in the (unknown) DGP. Otherwise, the model can arbitrarily reapportion APC effects, radically impacting interpretation.
CONCLUSIONS
Since the purpose of APC analysis is often to ascertain the presence of period and/or cohort trends, and since we rarely have solid (if any) theory regarding cohort groupings, there are
few circumstances in which this model achieves what Yang and Land claim it can. The results bring into question findings of several published studies using the HAPC model. However, the structure of the model remains a conceptual advance that is useful when we can assume the DGP has no period trends.
This commentary discusses the age-period-cohort identification problem. It shows that, despite a ... more This commentary discusses the age-period-cohort identification problem. It shows that, despite a plethora of proposed solutions in the literature, no model is able to solve the identification problem because the identification problem is inherent to the real-world processes being modelled. As such, we cast doubt on the conclusions of a number of papers, including one presented here (Page et al., this issue). We conclude with some recommendations for those wanting to model age, period and cohort in a compelling way.
Previous work (Bell and Jones 2013a, c; Luo and Hodges 2013) has shown that, when there are trend... more Previous work (Bell and Jones 2013a, c; Luo and Hodges 2013) has shown that, when there are trends in either the period or cohort residuals of Yang and Land’s Hierarchical Age-Period-Cohort (APC) model (Yang and Land 2006; Yang and Land 2013), the model can incorrectly estimate those trends, because of the well-known APC identification problem. Here we consider modelling possibilities when the age effect is known, allowing any period or cohort trends to be estimated. In particular, we suggest the application of informative priors, in a Bayesian framework, to the age trend, and we use a variety of simulated but realistic datasets to explicate this. Similarly, an informative prior could be applied to an estimated period or cohort trend, allowing the other two APC trends to be estimated. We show that a very strong informative prior is required for this purpose. As such, models of this kind can be fitted but are only useful when very strong evidence of the age trend (for example physiological evidence regarding health). Alternatively, a variety of strong priors can be tested and the most plausible solution argued for on the basis of theory.
This comment assesses how age, period and cohort (APC) effects are modelled with panel data in th... more This comment assesses how age, period and cohort (APC) effects are modelled with panel data in the social sciences. It considers variations on a 2-level multilevel model which has been used to show apparent evidence for simultaneous APC effects. We show that such an interpretation is often misleading, and that the formulation and interpretation of these models requires a better understanding of age, period and cohort effects and the exact collinearity present between them. This interpretation must draw on theory to justify the claims that are made. By comparing two papers which over-interpret such a model, and another that in our view interprets it appropriately, we outline best practice for researchers aiming to use panel datasets to find APC effects, with an understanding that it is impossible for any statistical model to find and separate all three effects.
Multilevel models have recently been used to uncover the effects of socio-demographic intersectio... more Multilevel models have recently been used to uncover the effects of socio-demographic intersections-that is, interactions between variables such as age, sex, ethnicity, socioeconomic position etc. Some have argued that this approach solves the problem of multiple testing found in standard dummy variable (fixed effects) regression techniques, because the intersections are automatically shrunk towards the mean. This means that effects that appear statistically significant by chance alone in a fixed effects regression will not appear so in a multilevel model. However, this requires certain assumptions to be true, and we know those assumptions are likely to be broken in these sorts of scenarios. We use simulation studies to show the effect of breaking these assumptions, specifically when there are true main effects and interactions that are unmodeled in the fixed part of the multilevel model. We show that, in these scenarios, shrinkage is less than is desired, and so intersectional effects are likely to appear statistically significant just by chance. We conclude by giving advice for the best approach to make this promising method work robustly.
This paper considers the modelling choices available to researchers using multilevel data, includ... more This paper considers the modelling choices available to researchers using multilevel data, including longitudinal data of various types. Specifically, we consider fixed effects (FE) and random effects (RE) models, including the within-between RE model, often misleadingly termed the ‘hybrid’ model. We argue that the latter is unambiguously a RE model, and that it is the most general of the three models, and as such a sensible starting point, given its flexibility to incorporate the positive aspects of both FE and RE models, and its ability to allow extensions (such as random slopes) that are often important. We present simulations that reveal the extent to which these models cope with mis-specification, finding that failing to include random slopes (e.g. in a FE or standard RE model) can lead to anti-conservative standard errors, and that mis-specifying non-Normal random effects as Normally distributed can introduce some small biases to variance and random effect estimates, but not fixed-part parameter estimates. We conclude with advice for applied researchers, and present a glossary, that gives clear definitions to terms that are confusing or have more than one meaning. Overall, we hope the paper gives practical advice to researchers in many different social science disciplines and beyond, looking to understand and use multilevel and longitudinal data.
The screening effectiveness of a chemical similarity search depends on a range of factors, includ... more The screening effectiveness of a chemical similarity search depends on a range of factors, including the bioactivity of interest, the types of similarity coefficient and fingerprint that comprise the similarity measure, and the nature of the reference structure that is being searched against a database. This study introduces the use of cross-classified multilevel modelling as a way to investigate the relative importance of these four factors when carrying out similarity searches on the ChEMBL database. Two principal conclusions can be drawn from the analyses: that the fingerprint plays a more important role than the similarity coefficient in determining the effectiveness of a similarity search, and that comparative studies of similarity measures should involve many more reference structures than has been the case in much previous work.
Kelley at al. argue that group-mean-centering covariates in multilevel models is dangerous, since... more Kelley at al. argue that group-mean-centering covariates in multilevel models is dangerous, since— they claim—it generates results that are biased and misleading. We argue instead that what is dangerous is Kelley et al.'s unjustified assault on a simple statistical procedure that is enormously helpful, if not vital, in analyses of multilevel data. Kelley et al.'s arguments appear to be based on a faulty algebraic operation, and on a simplistic argument that parameter estimates from models with mean-centered covariates must be wrong merely because they are different than those from models with uncentered covariates. They also fail to explain why researchers should dispense with mean-centering when it is central to the estimation of fixed effects models—a common alternative approach to the analysis of clustered data, albeit one increasingly incorporated within a random effects framework. Group-mean-centering is, in short, no more dangerous than any other statistical procedure, and should remain a normal part of multilevel data analyses where it can be judiciously employed to good effect.
It is claimed the hierarchical-age–period–cohort (HAPC) model solves the age–period–cohort (APC) ... more It is claimed the hierarchical-age–period–cohort (HAPC) model solves the age–period–cohort (APC) identification problem. However, this is debateable; simulations show situations where the model produces incorrect results, countered by proponents of the model arguing those simulations are not relevant to real-life scenarios. This paper moves beyond questioning whether the HAPC model works, to why it produces the results it does. We argue HAPC estimates are the result not of the distinctive substantive APC processes occurring in the dataset, but are primarily an artefact of the data structure—that is, the way the data has been collected. Were the data collected differently, the results produced would be different. This is illustrated both with simulations and real data, the latter by taking a variety of samples from the National Health Interview Survey (NHIS) data used by Reither et al. (Soc Sci Med 69(10):1439–1448, 2009) in their HAPC study of obesity. When a sample based on a small range of cohorts is taken, such that the period range is much greater than the cohort range, the results produced are very different to those produced when cohort groups span a much wider range than periods, as is structurally the case with repeated cross-sectional data. The paper also addresses the latest defence of the HAPC model by its proponents (Reither et al. in Soc Sci Med 145:125–128, 2015a). The results lend further support to the view that the HAPC model is not able to accurately discern APC effects, and should be used with caution when there appear to be period or cohort near-linear trends.
Urbanization has long been seen by scholars and policymakers as a disruptive process that can con... more Urbanization has long been seen by scholars and policymakers as a disruptive process that can contribute to social and political unrest, yet there is little cross-national quantitative empirical research on the topic. In this paper we provide a comprehensive analysis of the links between urban geography and the incidence of protests (i.e. demonstrations, riots and strikes) in African countries since 1990. In contrast to previous studies, we are careful to distinguish between urban population scale effects, urban population ratio effects, population rate-of-change effects and urban population distribution effects. We also provide an explicit test of the long-standing hypothesis that ‘over-urbanization’ increases the risk of civil unrest. Employing multilevel negative binomial models that control for key political and economic variables we find that urban population size and the number of large cities in a country are both positively and significantly associated protest incidence. By contrast, we find that a country's level of urbanization is negatively associated with protest incidence and reject the over-urbanization hypothesis: higher levels of urbanization are associated with less frequent protests at all income levels. We find no evidence that the pace of urban population growth or urban primacy significantly influence protest mobilization. In sum, our results provide a nuanced picture of the relationship between urban geography and protest incidence that challenges conventional wisdom and contemporary hyperbole about the dangers of ‘rapid urbanization’ in Africa in particular, and developing countries more generally.
This article challenges Fixed Effects (FE) modelling’s status as the ‘default option’ when using ... more This article challenges Fixed Effects (FE) modelling’s status as the ‘default option’ when using time-series-cross-sectional and panel data. We argue that understanding the difference between within- and between-effects of predictor variables is important when considering what modelling strategy to use. The downside of Random Effects (RE) compared to FE modelling – correlation between lower-level covariates and higher-level residuals - is a case of omitted variable bias, readily solvable using a variant of Mundlak’s (1978a) formulation. Consequently, RE modelling provides everything that FE modelling promises, and more. It allows time-invariant variables to be modelled, more parsimoniously than Plümper and Troeger’s (2007) suggested method. It is also readily extendable to Random Coefficients Models, allowing reliable, differential effects of variables without heavy parameterisation, the use of cross-level interactions between time-variant and invariant variables, and the modelling of complex variance functions. We are arguing not simply for technical solutions to endogeneity, but for the substantive importance of modelling context, and RE models’ ability to do so. Two empirical examples show that failing to do this can lead to misleading results. This paper is distinctive in stressing the substantive interpretations of within- and between-effects. This has implications beyond political science, to all datasets with multilevel structures.
There is ongoing debate regarding the shape of life-course trajectories in mental health. Many a... more There is ongoing debate regarding the shape of life-course trajectories in mental health. Many argue the relationship is U-shaped, with mental health declining with age to mid-life, then improving. However, I argue that these models are beset by the age-period-cohort (APC) identification problem, whereby age, cohort and year of measurement are exactly collinear and their effects cannot be meaningfully separated. This means an apparent life-course effect could be explained by cohorts. This paper critiques two sets of literature: the substantive literature regarding life-course trajectories in mental health, and the methodological literature that claims erroneously to have ‘solved’ the APC identification problem statistically (e.g. using Yang and Land’s Hierarchical APC – HAPC – model). I then use a variant of the HAPC model, making strong but justified assumptions that allow the modelling of life-course trajectories in mental health (measured by the General Health Questionnaire) net of any cohort effects, using data from the British Household Panel Survey, 1991-2008. The model additionally employs a complex multilevel structure that allows the relative importance of spatial (households, local authority districts) and temporal (periods, cohorts) levels to be assessed. Mental health is found to increase throughout the life-course; this slows at mid-life before worsening again into old age, but there is no evidence of a U-shape – I argue that such findings result from confounding with cohort processes (whereby more recent cohorts have generally worse mental health). Other covariates were also evaluated; income, smoking, education, social class, urbanity, ethnicity, gender and marriage were all related to mental health, with the latter two in particular affecting life-course and cohort trajectories. The paper shows the importance of understanding APC in life-course research generally, and mental health research in particular.
This paper reanalyses data used by Reinhart and Rogoff (2010c - RR), and later Herndon et al. (20... more This paper reanalyses data used by Reinhart and Rogoff (2010c - RR), and later Herndon et al. (2013) to consider the relationship between growth and debt in developed countries. The consistency over countries and the causal direction of RR’s so called ‘stylised fact’ is considered. Using multilevel models, we find that when the effect of debt on growth is allowed to vary, and linear time trends are fully controlled, the average effect of debt on growth disappears, whilst country-specific debt effects vary significantly. Additionally, countries with high debt levels debt appear more volatile in their growth rates. Regarding causality, we develop a new method extending distributed lag models to multilevel situations. These models suggest the causal direction is predominantly growth-to-debt, and is consistent (with some exceptions) across countries. We argue RR’s findings are too simplistic, with limited policy relevance, whilst demonstrating how multilevel models can explicate realistically complex scenarios.
This commentary clarifies our original commentary (Bell & Jones, 2014c) and illustrates some conc... more This commentary clarifies our original commentary (Bell & Jones, 2014c) and illustrates some concerns we have regarding the response article in this issue (Reither et al., 2015). In particular, we argue that (a) linear effects do not have to be produced by exact linear mathematical functions to behave as if they were linear, (b) linear effects by this wider definition are extremely common in real life social processes, and (c) in the presence of these effects, the Hierarchical Age Period Cohort (HAPC) model will often not work. Although Reither et al. do not define what a ‘non-linear monotonic trend’ is (instead, only stating that it isn’t a linear effect) we show that the model often doesn’t work in the presence of such effects, by using data generated as a ‘non-linear monotonic trend’ by Reither et al. themselves. We then question their discussion of fixed and random effects before finishing with a discussion of how we argue that theory should be used, in the context of the obesity epidemic.
Reither et al. (2009) use a Hierarchical Age-Period-Cohort model (HAPC - Yang & Land, 2006) to as... more Reither et al. (2009) use a Hierarchical Age-Period-Cohort model (HAPC - Yang & Land, 2006) to assess changes in obesity in the USA population. Their results suggest that there is only a minimal effect of cohorts, and that it is periods which have driven the increase in obesity over time. We use simulations to show that this result may be incorrect. Using simulated data in which it is cohorts, rather than periods, that are responsible for the rise in obesity, we are able to replicate the period-trending results of Reither et al. In this instance, the HAPC model misses the true cohort trend entirely, erroneously finds a period trend, and underestimates the age trend. Reither et al.’s results may be correct, but because age, period and cohort are confounded there is no way to tell. This is typical of age-period-cohort models, and shows the importance of caution when any APC model is used. We finish with a discussion of ways forward for researchers wishing to model age, period and cohort in a robust and non-arbitrary manner.
This paper: (a) finds rankings of who are the best formula 1 (F1) drivers of all time, conditiona... more This paper: (a) finds rankings of who are the best formula 1 (F1) drivers of all time, conditional on team performance; (b) quantifies how much teams and drivers matter; and (c) quantifies how team and driver effects vary over time and under different racing conditions. The finishing position of drivers is used as the response variable in a cross-classified multilevel model that partitions variance into team, team-year and driver levels. These effects are then allowed to vary by year, track type and weather conditions. Juan Manuel Fangio is found to be the greatest driver of all time. Team effects are shown to be more important than driver effects (and increasingly so over time), although their importance may be reduced in wet weather and on street tracks.
BACKGROUND
Whilst some argue that a solution to the age-period-cohort (APC) ‘identification prob... more BACKGROUND
Whilst some argue that a solution to the age-period-cohort (APC) ‘identification problem’ is impossible, numerous methodological solutions have been proposed, including Yang and Land’s Hierarchical-APC (HAPC) model: a multilevel model considering periods and cohorts as cross-classified contexts in which individuals exist.
OBJECTIVE
To assess the assumptions made by the HAPC model, and the situations in which it does and does not work.
METHODS
Simulation study. Simulation scenarios assess the effect of (a) cohort trends in the Data Generating Process (DGP) (compared to only random variation), and (b) grouping cohorts (in both DGP and fitted model).
RESULTS
The model only works if either (a) we can assume that there are no linear (or non-linear) trends in period or cohorts, (b) we control any cohort trend in the model’s fixed part and assume there is no period trend, or (c) we group cohorts in such a way that they exactly match the groupings in the (unknown) DGP. Otherwise, the model can arbitrarily reapportion APC effects, radically impacting interpretation.
CONCLUSIONS
Since the purpose of APC analysis is often to ascertain the presence of period and/or cohort trends, and since we rarely have solid (if any) theory regarding cohort groupings, there are
few circumstances in which this model achieves what Yang and Land claim it can. The results bring into question findings of several published studies using the HAPC model. However, the structure of the model remains a conceptual advance that is useful when we can assume the DGP has no period trends.
This commentary discusses the age-period-cohort identification problem. It shows that, despite a ... more This commentary discusses the age-period-cohort identification problem. It shows that, despite a plethora of proposed solutions in the literature, no model is able to solve the identification problem because the identification problem is inherent to the real-world processes being modelled. As such, we cast doubt on the conclusions of a number of papers, including one presented here (Page et al., this issue). We conclude with some recommendations for those wanting to model age, period and cohort in a compelling way.
Previous work (Bell and Jones 2013a, c; Luo and Hodges 2013) has shown that, when there are trend... more Previous work (Bell and Jones 2013a, c; Luo and Hodges 2013) has shown that, when there are trends in either the period or cohort residuals of Yang and Land’s Hierarchical Age-Period-Cohort (APC) model (Yang and Land 2006; Yang and Land 2013), the model can incorrectly estimate those trends, because of the well-known APC identification problem. Here we consider modelling possibilities when the age effect is known, allowing any period or cohort trends to be estimated. In particular, we suggest the application of informative priors, in a Bayesian framework, to the age trend, and we use a variety of simulated but realistic datasets to explicate this. Similarly, an informative prior could be applied to an estimated period or cohort trend, allowing the other two APC trends to be estimated. We show that a very strong informative prior is required for this purpose. As such, models of this kind can be fitted but are only useful when very strong evidence of the age trend (for example physiological evidence regarding health). Alternatively, a variety of strong priors can be tested and the most plausible solution argued for on the basis of theory.
This comment assesses how age, period and cohort (APC) effects are modelled with panel data in th... more This comment assesses how age, period and cohort (APC) effects are modelled with panel data in the social sciences. It considers variations on a 2-level multilevel model which has been used to show apparent evidence for simultaneous APC effects. We show that such an interpretation is often misleading, and that the formulation and interpretation of these models requires a better understanding of age, period and cohort effects and the exact collinearity present between them. This interpretation must draw on theory to justify the claims that are made. By comparing two papers which over-interpret such a model, and another that in our view interprets it appropriately, we outline best practice for researchers aiming to use panel datasets to find APC effects, with an understanding that it is impossible for any statistical model to find and separate all three effects.
Multilevel models have recently been used to uncover the effects of socio-demographic intersectio... more Multilevel models have recently been used to uncover the effects of socio-demographic intersections-that is, interactions between variables such as age, sex, ethnicity, socioeconomic position etc. Some have argued that this approach solves the problem of multiple testing found in standard dummy variable (fixed effects) regression techniques, because the intersections are automatically shrunk towards the mean. This means that effects that appear statistically significant by chance alone in a fixed effects regression will not appear so in a multilevel model. However, this requires certain assumptions to be true, and we know those assumptions are likely to be broken in these sorts of scenarios. We use simulation studies to show the effect of breaking these assumptions, specifically when there are true main effects and interactions that are unmodeled in the fixed part of the multilevel model. We show that, in these scenarios, shrinkage is less than is desired, and so intersectional effects are likely to appear statistically significant just by chance. We conclude by giving advice for the best approach to make this promising method work robustly.
This paper considers the modelling choices available to researchers using multilevel data, includ... more This paper considers the modelling choices available to researchers using multilevel data, including longitudinal data of various types. Specifically, we consider fixed effects (FE) and random effects (RE) models, including the within-between RE model, often misleadingly termed the ‘hybrid’ model. We argue that the latter is unambiguously a RE model, and that it is the most general of the three models, and as such a sensible starting point, given its flexibility to incorporate the positive aspects of both FE and RE models, and its ability to allow extensions (such as random slopes) that are often important. We present simulations that reveal the extent to which these models cope with mis-specification, finding that failing to include random slopes (e.g. in a FE or standard RE model) can lead to anti-conservative standard errors, and that mis-specifying non-Normal random effects as Normally distributed can introduce some small biases to variance and random effect estimates, but not fixed-part parameter estimates. We conclude with advice for applied researchers, and present a glossary, that gives clear definitions to terms that are confusing or have more than one meaning. Overall, we hope the paper gives practical advice to researchers in many different social science disciplines and beyond, looking to understand and use multilevel and longitudinal data.
Uploads
Whilst some argue that a solution to the age-period-cohort (APC) ‘identification problem’ is impossible, numerous methodological solutions have been proposed, including Yang and Land’s Hierarchical-APC (HAPC) model: a multilevel model considering periods and cohorts as cross-classified contexts in which individuals exist.
OBJECTIVE
To assess the assumptions made by the HAPC model, and the situations in which it does and does not work.
METHODS
Simulation study. Simulation scenarios assess the effect of (a) cohort trends in the Data Generating Process (DGP) (compared to only random variation), and (b) grouping cohorts (in both DGP and fitted model).
RESULTS
The model only works if either (a) we can assume that there are no linear (or non-linear) trends in period or cohorts, (b) we control any cohort trend in the model’s fixed part and assume there is no period trend, or (c) we group cohorts in such a way that they exactly match the groupings in the (unknown) DGP. Otherwise, the model can arbitrarily reapportion APC effects, radically impacting interpretation.
CONCLUSIONS
Since the purpose of APC analysis is often to ascertain the presence of period and/or cohort trends, and since we rarely have solid (if any) theory regarding cohort groupings, there are
few circumstances in which this model achieves what Yang and Land claim it can. The results bring into question findings of several published studies using the HAPC model. However, the structure of the model remains a conceptual advance that is useful when we can assume the DGP has no period trends.
Whilst some argue that a solution to the age-period-cohort (APC) ‘identification problem’ is impossible, numerous methodological solutions have been proposed, including Yang and Land’s Hierarchical-APC (HAPC) model: a multilevel model considering periods and cohorts as cross-classified contexts in which individuals exist.
OBJECTIVE
To assess the assumptions made by the HAPC model, and the situations in which it does and does not work.
METHODS
Simulation study. Simulation scenarios assess the effect of (a) cohort trends in the Data Generating Process (DGP) (compared to only random variation), and (b) grouping cohorts (in both DGP and fitted model).
RESULTS
The model only works if either (a) we can assume that there are no linear (or non-linear) trends in period or cohorts, (b) we control any cohort trend in the model’s fixed part and assume there is no period trend, or (c) we group cohorts in such a way that they exactly match the groupings in the (unknown) DGP. Otherwise, the model can arbitrarily reapportion APC effects, radically impacting interpretation.
CONCLUSIONS
Since the purpose of APC analysis is often to ascertain the presence of period and/or cohort trends, and since we rarely have solid (if any) theory regarding cohort groupings, there are
few circumstances in which this model achieves what Yang and Land claim it can. The results bring into question findings of several published studies using the HAPC model. However, the structure of the model remains a conceptual advance that is useful when we can assume the DGP has no period trends.