Understanding The Redesigned National Crime Victimization Survey0
God help me. This article was supposed to be about school crime. It turned into something considerably more complicated. It took multiple drafts and many hours to (hopefully) get it right.
The U.S. Department of Justice’s Bureau of Justice Statistics recently released a report examining criminal victimization among students ages 12 to 18, both at and away from school. I thought it would make for an interesting article that would take about thirty minutes to write.
Easy-peasy.
Nope.
I read the narrative, and then I examined the charts. What the narrative said about school crime didn’t match what was presented in the charts. Something was off. Maybe my morning coffee intake was excessive? Maybe I forgot how to read and interpret BJS documents?
Yes, accurate data on school crime is in the article below.
Understanding What Happened To The National Crime Victimization Survey
If you need to understand crime in the US or the country’s criminal justice system, please at least skim this article. The report turned into a lesson in statistical interpretation, survey methodology, standard errors, and statistical significance.
Yes, I address social science research methodology. My deepest apologies. But if you use the National Crime Victimization Survey, you may be asked about the changes.
I try to make the research process layperson-friendly.
Understanding what’s happening to the NCVS may be critical to your understanding of crime and justice research in the US.
Crime Surveys Compared To Reported Crimes
Crime surveys are used by many countries along with incidents reported to law enforcement. All crime researchers understand that the great majority of criminal events are not reported to the police; thus, the need for surveys to understand the totality of criminal activity (minus homicides [you can’t interview dead people], people under the age of 12, and business crimes).
The data is also used to understand justice systems, crime victims, crime characteristics, and criminal offenders.
I use the recently released school crime data from the Bureau of Justice Statistics as an example.
The differences between the two measurements (FBI-NCVS) of crime in terms of the numbers counted are huge, and generally speaking, the larger the numbers, the better the results.
If you want to understand crime in America, understanding the National Crime Victimization Survey is vital. If you want to understand crime in America, you can’t (in my opinion) rely solely on FBI data because the good majority of crime is not reported in the US and other countries.
The Reason For This Article—The Redesign
The National Crime Victimization Survey is undergoing its first major redesign since 1992. The 2024 survey was a transition year, and the way BJS handled that transition has consequences for how readers should interpret some of the numbers.
The latest school-victimization report provides an excellent demonstration of the problem.
Will This Apply to Future National Crime Survey Documents?
I assume that it will, which is why I created this article. The new Criminal Victimization document from the Bureau of Justice Statistics for 2025 will be out soon (it should have been out by now).
Previous Bureau of Justice Statistics national crime reports indicated record highs for rates of violent crime for 2022 to 2024. FBI numbers based on reported crimes (most are not) show stark decreases.
Will the redesigned National Crime Victimization Survey change this finding? We shall see.
What Is the National Crime Victimization Survey?
The National Crime Victimization Survey, or NCVS, is the federal government’s principal survey for measuring criminal victimization in the US.
Unlike FBI crime statistics, which are based primarily on a much smaller number of crimes reported to law enforcement, the NCVS asks people about crimes they experienced, including events that were never reported to police.
Every survey (including Gallup and other well-known instruments) uses samples of the population and then makes judgments based on what the samples say. To my knowledge, the National Crime Victimization Survey samples and methodology haven’t been criticized by the research community, and the reason is that the USDOJ updates its process in the same way that all survey companies update theirs.
BJS says the NCVS was last comprehensively redesigned in 1992. The agency began a multiyear effort to modernize the survey, improve the quality and efficiency of data collection, and improve the measurement and classification of crime. The redesign affects the questions used to identify victimization and the detailed questions asked about individual incidents.
This is not simply a change in the appearance of a questionnaire. It changes how the federal government asks Americans about their experiences with crime.
The 2024 NCVS Was A Transition Year
BJS did not simply abandon the old questionnaire and replace it with the new one in 2024. Instead, BJS used a split-sample design.
Approximately half of the 2024 sample received the traditional, or legacy, NCVS questionnaire. The other half received the redesigned questionnaire.
The purpose was to continue producing national estimates while allowing BJS to evaluate the new instrument against the old one.
The school-victimization report discussed here uses the legacy (older) portion of the 2024 survey. BJS says it will separately release estimates from the redesigned 2024 questionnaire.
That distinction is important.
There are two sets of 2024 NCVS estimates because two different questionnaires were being used during the split-sample transition.
BJS transitioned to the redesigned instrument in 2025. The agency is now processing and reviewing the redesigned data and has said that release timelines for the redesigned 2024 estimates will be developed following that work.
Splitting the Sample Reduced Precision
There was a price for this transition. Normally, a larger survey sample allows researchers to produce more precise estimates. In 2024, however, the NCVS sample was divided between two questionnaires.
BJS explicitly acknowledges that splitting the sample resulted in decreased precision during the transition period. The agency also says the 2024 weighting procedure produced more variation in estimates (findings) and higher standard errors.
Weighting adjusts survey responses so the people in the sample more accurately represent the larger population being studied.
In plain English, some 2024 numbers are less precise than readers might expect from a large national federal survey. That does not mean the numbers are useless.
It means that readers need to be particularly careful when interpreting relatively small differences between groups or years.
There is another methodological change worth knowing about. BJS says the standard errors for the 2024 victimization estimates were calculated using a different variance-estimation method than in previous Criminal Victimization reports.
If you fully understand the following, you are better at research methodology than me. The 2024 report used balanced repeated replication, or BRR, rather than the generalized variance function parameters used in earlier reports.
Balanced repeated replication is a statistical method that repeatedly tests different versions of the survey sample to estimate how much the results might vary because of sampling.
BJS notes that this can result in differences in reported standard errors (explained below) and in which year-to-year differences are judged statistically significant (also explained below). For the victimization estimates discussed here, BJS used balanced repeated replication, or BRR, to calculate standard errors rather than the generalized variance function parameters used in earlier reports.
For most readers, that is a lot of statistical information. But it matters.
What Is a Standard Error?
This is where crime statistics become difficult for most readers. Suppose a BJS chart says that the victimization rate is 27.0 per 1,000 students.
Most people will see 27.0.
But that number is an estimate based on a sample of the population. It is not a count of every victimization experienced by every student in the United States.
BJS therefore calculates a standard error.
A standard error provides information about the uncertainty surrounding a survey estimate. Generally, a smaller standard error indicates greater precision, while a larger standard error indicates greater uncertainty.
Imagine that you wanted to know the average number of hours Americans sleep each night. You cannot ask every person in America, so you select a representative sample of people and calculate their average.
If you selected a different representative sample tomorrow, you would probably get a somewhat different answer. If you repeated the process many times, you would get a distribution of slightly different estimates.
The standard error provides information about how much variation would be expected from the sampling process.
This Creates a Problem With Charts
Bureau of Justice Statistics charts are extremely useful. They are also capable of misleading readers who do not understand what the numbers represent.
Imagine a chart showing this:
Group A: 30
Group B: 20
The difference is obvious.
A reader naturally thinks that Group A experienced more crime than Group B.
But that conclusion is not necessarily justified simply because the point estimates are different. A point estimate is the single number a survey produces as its best estimate of the actual value. Researchers also have to consider the uncertainty surrounding those estimates.
Statistical Significance
Statistical significance is a way of assessing whether an observed difference is sufficiently large, given the uncertainty in the estimates, to conclude that it is unlikely to be explained by sampling variation alone.
You could have an overrepresentation or underrepresentation of people in your sample by race, income, or location, which could significantly skew results.
That is different from saying that a finding is “true,” that it is caused by a particular factor, or that there are no competing explanations. The sample has to be representative of the US population.
The charts in the school crime document show the point estimates.
The statistical analysis helps determine whether the difference between those estimates is large enough to support a conclusion about the underlying population.
This distinction is easy to miss when someone is looking at a colorful bar chart.
The Latest School Report Demonstrates the Problem
Consider the new BJS report, Victimization At and Away From School Among Students Ages 12 to 18, 2014–2024.
The 2024 total victimization rate was 27.0 per 1,000 students at school and 17.7 per 1,000 away from school. A national document used this as its headline.
That is a substantial-looking difference.
BJS states that the at-school crime rate was greater than the away-from-school rate.
The report also states that comparisons made in the text are statistically significant unless otherwise noted. Therefore, this particular comparison is being treated by BJS as statistically significant.
The standard errors were 4.51 for the at-school estimate and 3.40 for the away-from-school estimate. The at-school estimate of 27.0 is less precise than the away-from-school estimate of 17.7 because it has a larger standard error (4.51 versus 3.40). Less precise does not mean the 27.0 estimate is wrong or less accurate. It means there is more statistical uncertainty surrounding that estimate.
The important lesson is not that the school crime chart was wrong. It is that the chart alone does not tell the entire statistical story.
A 35 Percent Increase That Was Not Statistically Significant
BJS estimated total victimization at school at 20.0 per 1,000 students in 2023 and 27.0 per 1,000 in 2024.
That is an increase of approximately 35 percent.
A headline writer could easily write: School criminal victimization increased 35 percent in one year.
But that would leave out an important qualification. BJS says the difference between 2023 and 2024 was not statistically significant.
Therefore, the more accurate description is:
The estimated rate of total victimization at school increased from 20.0 to 27.0 per 1,000 students between 2023 and 2024, but BJS found that the difference was not statistically significant.
That is a very different statement. The number went up. The survey estimate went up.
But the evidence was not strong enough for BJS to conclude that the underlying rate had actually changed between the two years.
This is precisely the sort of distinction that can disappear when a reader sees only a chart.
There Was A Statistically Significant Increase in Violent Victimization
The story becomes more complicated when we look at violent victimization.
BJS reports that nonfatal violent victimization at school increased from 14.2 per 1,000 students in 2023 to 21.8 per 1,000 in 2024.
Unlike the total-victimization comparison, BJS found this increase statistically significant at the 90 percent confidence level.
But even this finding requires another qualification.
When simple assault was excluded, the estimated rate increased from 2.9 per 1,000 in 2023 to 5.8 per 1,000 in 2024.
That difference was not statistically significant.
So two seemingly similar statements produce different statistical conclusions.
Violent victimization, including simple assault, increased significantly.
Violent victimization excluding simple assault (most assaults are simple assaults) did not increase significantly.
That is much more complicated than a headline simply saying violent victimization increased.
Demographic Comparisons Create the Same Problem
The same issue appears when the report examines sex, race, age, residence, and household income.
Charts display differences.
The statistical analysis determines whether those differences are significant.
For example, total victimization at school was estimated at 24.3 per 1,000 for male students and 29.9 per 1,000 for female students.
The female point estimate is higher.
But BJS says there were no statistically significant differences between males and females at school across the types of crime measured.
Away from school, however, the estimates were 12.2 per 1,000 for males and 23.5 per 1,000 for females, and BJS found a statistically significant difference.
This is an excellent illustration of why simply comparing the heights of bars on a chart can lead readers to conclusions that the statistical analysis does not support.
Race Provides Another Example
Some racial and Hispanic-origin estimates in the report appear dramatically different.
For total victimization at school, the estimates included 18.7 per 1,000 for White students, 35.8 for Black students, 22.5 for Hispanic students, and 65.7 for students classified in the “Other” category.
Those numbers look dramatically different.
But BJS concluded that total victimization at school did not differ significantly by race or Hispanic origin.
The same was true for total victimization away from school.
Several of these estimates also carry BJS’s warning symbol (in the footnotes) indicating that they should be interpreted with caution.
Again, the chart is not necessarily wrong. It is displaying the estimates.
The statistical analysis is telling us that the apparent differences are not sufficiently reliable to support a conclusion that the underlying population rates are different.
BJS Warns Readers About Small Samples
The report contains another important warning. BJS places an exclamation point next to certain estimates (my first indication that understanding this document would be difficult).
The symbol means the estimate should be interpreted with caution because it is based on 10 or fewer sample cases or has a coefficient of variation greater than 50 percent. This is an important distinction.
The “10 or fewer” refers to actual survey sample cases, not the estimated number of victims in the entire United States.
A survey can produce a large weighted national estimate even when the underlying number of survey cases is relatively small. That is because survey responses are weighted to represent the national population.
The Coefficient Of Variation
The coefficient of variation is a measure of relative variability that helps researchers assess the reliability of an estimate. A very high coefficient of variation is a warning that an estimate is relatively unstable and should be interpreted cautiously.
This is one of the more difficult concepts for non-statisticians to understand.
A number such as 35.8 victimizations per 1,000 students does not mean that researchers literally observed 35.8 victimizations for every 1,000 students.
It is a statistical estimate based on survey responses and weighted to represent the U.S. student population.
The 2024 NCVS Transition Is The Bigger Story
This is where I believe the NCVS redesign becomes more important than the school report itself, or other documents from the Bureau of Justice Statistics.
The school report is only one of many reports produced using NCVS data. The 2024 transition therefore affects the broader interpretation of 2024 NCVS findings.
BJS has taken steps to preserve continuity during the transition. The agency retained the legacy questionnaire for part of the sample so that it could continue producing estimates based on the established instrument while testing the redesigned instrument.
BJS also has been transparent about the reduction in precision associated with splitting the sample.
The agency concluded that maintaining the existing sample size and continuity of reporting outweighed the loss of precision during the transition.
That is a methodological judgment. It is not evidence that the NCVS is unreliable. In fact, the entire purpose of the redesign is to improve the survey.
The Problem Is Communication
The problem is that most readers are unlikely to study the methodology section before reading the findings. I don’t unless there is a difference between the narrative and the charts. That’s when I go to the footnotes and methodology section to understand what’s happening.
Readers are much more likely to see a headline saying that crime increased. Or a chart showing that one group has a higher rate than another. Or a graph showing a large change between two years.
The standard error will usually not be the first thing they notice. Neither will the footnote. Neither will the statistical-significance notation.
Yet those details can determine whether an apparent difference should actually be interpreted as a meaningful difference.
This Is Not Unique To BJS
It is a general problem with communicating social science (beyond the fact that many (most?) researchers cannot communicate in plain English.
Crime statistics are often reduced to a single number because a single number is easy to understand (why readers gravitate to crimes reported to law enforcement through FBI reports). Standard errors are not. Statistical significance is not. Survey weights are not. Sampling error is not.
And a questionnaire redesign certainly is not. Yet all of those things can affect what a crime statistic actually tells us.
There Is Another NCVS Question Coming
The transition does not end with the 2024 legacy (older) estimates.
BJS says the redesigned NCVS instrument was fully implemented in 2025. The agency is now processing and reviewing the redesigned data. That raises an important question for future crime reporting.
How comparable will the new estimates be with the historical NCVS series? That question does not mean the redesigned survey is bad.
Quite the opposite. BJS redesigned the survey because it believes the new instrument can improve the measurement of criminal victimization.
But whenever a major survey instrument changes, researchers, criminal justice personnel, and journalists need to determine whether a change in the resulting numbers represents an actual change in victimization, a change in how respondents answer questions, a change in how crimes are identified or classified, a change caused by the survey instrument itself, or some combination of these factors.
That is why methodological documentation matters.
School Crime Results
The latest BJS school-victimization report contains important findings.
In 2024, the estimated total victimization rate was 27.0 per 1,000 students at school compared with 17.7 per 1,000 away from school, a difference that BJS treats as statistically significant.
At the same time, the estimated total at-school rate increased from 20.0 to 27.0 between 2023 and 2024, but that change was not statistically significant.
Violent victimization at school increased from 14.2 to 21.8 per 1,000 and was statistically significant at the 90 percent confidence level.
But when simple assault was excluded, the increase from 2.9 to 5.8 was not statistically significant.
Those distinctions are important.
And they demonstrate why the larger story may not be school crime. BJS has explained these limitations, but unfortunately, readers have to move between the narrative, charts, footnotes, appendices, and methodology to understand what the numbers actually mean.
Non-researchers should not have to do this. BJS needs to do better at communicating.
But there is no contradiction between a chart showing two different numbers and BJS concluding that the difference is not statistically significant.
The chart shows the estimates.
The statistical analysis tells us whether the difference is sufficiently supported to draw a conclusion about the underlying population.
That distinction is easy to miss.
And it is precisely why journalists, policymakers, criminal justice professionals, and most readers should be cautious about drawing conclusions from the chart numbers alone.
The NCVS is one of America’s most important measures of criminal victimization. A major redesign is changing how that information is collected.
Before we debate whether a particular crime rate went up or down, we should understand how the number was produced and how certain we can be about the difference.
As with so much crime data, the number is only the beginning of the story.
Suggestions And A Personal Note
I have suggestions regarding the use of redesigned Bureau of Justice Statistics NCVS documents; stick to the narratives.
BJS has already done the heavy lifting as to methodology and the correct use of statistics. The narrative and the charts may cause confusion. Please pay attention to the footnotes or methodology, or stick to the narrative.
Although I have been reading and interpreting crime statistics for decades, I am not a methodologist. I am not a trained researcher beyond required university research courses. Occasionally (actually once) my interpretations of BJS documents have been wrong, which I corrected. But my analyses have been used by countless reporters and publications throughout many years without issue.
Much of what’s above is my writing, and much is from ChatGPT. This is my third attempt at this article, and the first two were entirely my writing before asking ChatGPT to fact-check what I wrote. I use ChatGPT to see if my conclusions are correct. ChatGPT does not hesitate to tell me if my writing needs corrections or improvement. Sometimes, I find that ChatGPT can be wrong, and I tell it as much. I do not use everything it suggests. AI can create some interesting conversations.
I’m not sure why BJS did not create a layperson-friendly document describing the redesign of the National Crime Victimization Survey. Yes, the redesign is documented on their website, but the vast majority of users and those creating conclusions (including reporters) based on the NCVS have little exposure to the intricacies of survey research.
In my humble opinion, that’s a mistake that the USDOJ and the Bureau of Justice Statistics need to correct.
Comments