From: "Julius Chang" <[p 00302] at [psilink.com]> Newsgroups: talk.politics.guns Subject: NEJM article analysis Date: Thu, 21 Oct 93 19:52:36 -0500 I also read it carefully and feel that it is garbage. Some weaknesses: 1. The authors combine homicide (case) data and control data from three separate counties in WA State, TN, and OH but do not mention anything about analyzing these subgroups to see if it is valid to combine them into a single population. 2. The data was collected from Aug. 23, 1987 to Aug. 23, 1992 for King County, WA and Shelby County, TN. For Cuyahoga County, OH, the collection period was from Jan. 1, 1990 to Aug. 23, 1992. Time-series analysis might be more appropriate than binned, multivariate regression. Over a five-year time span, many variables can change. Do the authors really know what they are measuring? Are data from 1987 comparable to data from 1992? 3. Binning of the data results in loss of information and is subject to arbitrariness (e.g., arbitrary selection of bins deemed relevant by the researchers). In fact, the authors completely ignore issues such as firearms training, income level, educational level, and whether or not the firearm is legally owned as potential variables. 4. Selection of the control group is not random. First, control households were offered a $10 incentive to respond. This introduces an automatic bias into the data. Second, the authors use "quota sampling" since the authors considered only households which matched the case households in age, sex, race, and neighborhood. This is a non-probability sampling method and it is impossible to calculate error estimates. The authors use the common "method of comparison" to look for the effects of a "treatment" on a "response". The best way to do this is with a randomized, controlled experiment. The treatment (case) group and control group are identical in every way except for the single variable of interest (the treatment). A well-designed experiment is run "double-blind" to guard against bias. Does any of this resemble the work in this NEJM paper? 5. Selection of the case group is also not random. The authors also offered a $10 incentive payment to participate in the survey. 6. Non-response effects were not analyzed. Census data was obtained from only 70% of the control households approached. 7. It is unclear if the same interviewer conducted all of the data collection from the case and control groups. Using different interviewers can bias the results. 8. It is not stated when the interviewer(s) canvassed the neighborhood to find controls. If they went around during the daytime, then that biases the types of people who might be home to participate (perhaps unemployed, lazy scum instead of responsible, employed adults). 9. Possible collinearity (dependent variable) problems. The authors claim that their final six variables are independent (home rented, case or control subject lived alone, any household member hit or hurt in a fight in the house, any household member arrested, any household member used illicit drugs, gun or guns kept in the home). I find it hard to believe that "any household member arrested" is independent from "illicit drug use" and/or "household member hit or hurt in a fight". 10. The results show that the odds of getting killed for renters is 4.4 times greater (2.3-8.2 95% confidence level) vs. homeowners. This clearly shows the difference between correlation and causation. Yet the authors made no such distinction on CNN (they mention it in the paper) when discussing any link between firearms ownership and homicide. 11. The authors claim that they diminished any recall bias by the case group by interviewing them after a three week mourning period. They make no mention of any data or study supporting the validity of this method. In fact, issues such as tachypsychia (distortion of perceived time) and tunnel vision can significantly alter one's perception of the true facts. The authors also used a "forced-choice questionnaire to ascertain information in a comparable manner from case proxies and controls." But this does not mean that the data is accurate since the actual situation may be very different from the fixed choices. But the respondent must force his answer into one of the pre-selected choices. 12. The authors admit that they have no way to verify the accuracy of the respondent's answers independently. To conduct a regression analysis including confidence levels (as the authors have done), one must know something about the statistical distribution of the data. For example, consider a linear regression of the form Y = a + bX + E where E is the error. The errors Ei for each independent variable value Xi must be normally distributed, have the same variance for every Xi, and be independent. 13. The authors dismiss potential under-reporting problems by the controls. They reason that previous studies of owners of registered handguns showed that the respondents' answers to questions about their gun ownership was valid. The problem is that none of the counties studied requires gun registration. It is quite possible that guns were obtained illegally, which reduces the chances of reporting possession. If the gun is required to be registered, then there is less of a chance of under-reporting (as was claimed). 14. The protective use of firearms was rated solely based on killings. The authors did not considered woundings, scaring off of attackers, or any other non-lethal result. Yet the authors claim that "our methodology was capable of demonstrating significant protective effects of gun ownership as readily as any evidence of increased risk." 15. Here is a cross-section of the case subjects: 24.8% had drinking problems 20.3% used illicit drugs 31.8% were hit or hurt in a fight in the home 17.3% required medical attention becuase of a fight in the home 29.9% were involved in a fight outside the home 36.0% were arrested at some time Does this in any way sound like a stable household? Adding any weapon into this mix is like gasoline on a fire. The authors state that their research was conducted "in three urban counties that lack a substantial percentage of Hispanic citizens. Our results may therefore not be generalizable to more rural communities or to Hispanic households." Do you think that their results apply to normal households with stable people who just happen to own firearms? 16. The authors use discredited research to support their claim that there is a "strong link between the availability of guns and community rates of homicide." These references include the well-known Sloan et al. study of Seattle and Vancouver and Loftin et al.'s study of the effects of restrictive licensing in Washington, DC. 17. The authors do not seem to make a distinction in the types of homicide when determining the protective effect of firearms. Table 1 of their paper shows that there were 15 excusable homicide cases. The authors state that "any death ruled a homicide was included." 18. The authors do not consider the ownership of the murder weapon. If the case victim were killed by a gun owned by the attacker, that is the same as a case victim killed with his own gun by some attacker. 19. The authors depend upon medical examiner reports, police reports, and newspaper stories for the homicide data. Thus, killings by an "intimate acquaintance" might include the victim's best buddy and drug connection. Such a potential bias makes it hard to generalize the findings to normal people. References: Arthur L. Kellerman et al., Gun Ownership as a Risk Factor for Homicide in the Home, The New England Journal of Medicine, vol. 329, no. 15, Oct. 7, 1993, pp. 1084-1091. Gary Kleck, Point Blank: Guns and Violence in America, Aldine de Gruyter. William H. Press et al., Numerical Recipes in Fortran, 2nd Edition, Cambridge University Press. R. Kapadia and G. Andersson, Statistics Explained: Basic Concepts and Methods, Wiley. Norman L. Johnson and Fred C. Leone, Statistics and Experimental Design in Engineering and the Physical Sciences, Vol. 1, Wiley. K.A. Brownlee, Statistical Theory and Methodology in Science and Engineering, Wiley. David Freedman, Robert Pisani, and Roger Purves, Statistics, W.W. Norton and Company. Harrison M. Wadsworth, Handbook of Statistical Methods for Engineers and Scientists, McGraw-Hill. If you think that any of this analysis is incorrect, let me know. -Julius