Predicting Response Rates Once Again

1. Questions

Counting response burden scores (Schmid and Axhausen 2019b) is not a very popular task at the Institute for Transport Planning and Systems (IVT, ETH Zurich). One or the other former PhD student got away without reporting them. However, the collective effort over the years yielded a unique dataset allowing us to understand response rates as a function of recruitment efforts and incentives. Other research institutes might therefore be encouraged to also contribute. This paper briefly elaborates the methodology once again and introduces the responseRateAnalysis R-package (https://github.com/dheimgartner/responseRateAnalysis) with its helper functions to easily replicate the results. Since the last update (Schmid and Axhausen 2019a) a handful of students left not until the final response burden score of their surveys was counted. Thanks to them, we feel that the time is right to update response rates results once again.

The main purpose of this paper is to replicate the analysis, update the estimates based on the enlarged sample and to make the analysis easily replicable: The R-package can be installed as explained below but we hope that other research groups rather clone the repository and contribute with their scored survey instruments according to the guidelines outlined on GitHub so that we or they can predict response rates from time to time again.

R> devtools::install_github("dheimgartner/responseRateAnalyis")

2. Methods

The main data collection effort consists of scoring the survey instruments according to Table 1.

In comparison to the previous publication (Schmid and Axhausen 2019b), 14 additional surveys have been scored and categorized by IVT members. Recruitment means that the respondents were contacted prior to the survey and they agreed to participate. Incentives mean any form of stimulus to motivate respondents to participate (usually a monetary payment upon completion of the survey). Unfortunately, the form and level of the incentive have not been recorded. A new category (no recruitment but with incentive payments) can now be distinguished. However, only five studies fall in this category. The current state of the database is attached to the package response_rates and its variables documented (?response_rates). The full sample underlying this report can be found in Appendix A.

The distribution of the response burden scores (RBS) for the four recruitment (R) and incentive (I) categories are visualized in Fig. 1. The following abbreviations are used throughout the text: For example, RxI stands for the category where respondents were recruited as well as incentives paid. An N negates; hence as an example, NRxNI reflects the category without recruitment and without incentives (i.e., participants were not contacted prior to the survey administration and were not promised any form of incentive upon completion). The median RBS is 399 and surveys with a score higher than 1’500 are rare (twelve points roughly correspond to a one-minute response time).

Fig. 1.Distribution of the response burden scores for recruitment $x$ incentive categories. Rax stands for recruited, with incentives, $N$ negates.

Building on Schmid and Axhausen (2019a), we estimate a logistic regression model relating response burden scores to log-transformed response rates:

$$\log \left( \frac{y_i}{100 - y_i} \right) = \beta_{0} + \beta_{1} \frac{x_i}{1000} + \varepsilon_{i} \tag{1}$$

where $y_i$ denotes the response rate (in percent), $x_i$ represents the ex-ante response burden score, and $\varepsilon_i$ is a normally distributed clustered error term (similar errors for different survey waves of the same study). Observations were weighted by the square root of the sample size. The model was estimated for the entire sample (excluding surveys with a sample size less than ten) as well as separately for recruitment by incentive category. The exponential $\exp(\beta_1)$ represents a marginal change in the odds ratio (i.e., participation vs. non-participation).

If a survey instrument’s response burden score increases by 100 points, the odds of participating decrease according to:

$$\left( \exp \left( \frac{\beta_1}{1000} \right) -1 \right) \cdot 100 \tag{2}$$

The basic workflow is as follows:

1. default_data() loads and prepares the response_rates data for estimation. It selects the input variables, computes the weights (sqrt (sample_size)), rescales the response burden score (response_burden_score / 1000) and $\log$ transforms the response rate ((log(response_rate / (100 - response_rate))).

2. Fit a linear regression model with lm().

3. Add clustered standard errors with add_clustered() which uses the clubSandwich (Pustejovsky 2024) and lmtest package (Zeileis and Hothorn 2002) under the hood to correct standard errors and related statistics.

4. The functions from the texreg (Leifeld 2013) package (e.g., screenreg() or texreg()) work together with the class "clustered" "lm" (as returned by add_clustered()) and produce regression tables (such as the ones reported here).

The following code conducts the analysis for the full sample (i.e., same slope but different intercepts for the recruitment x incentive categories):

R> dat <- default_data() %>%
+ filter(sample_size >= 10) # to be consistent with previous publications
R> fit <- lm(y - 0 + x + RxI + RxNI + NRxI + NRxNI,
+ data = dat, weights = veight)
R> m1 <- add_clustered(fit, cluster = dat$survey_id, type = "CR2")
R> class(m1)
[1] "clustered" "lm"

We repeat the above estimation for different samples, comparing the estimates of the last publication to the updated ones as well as estimate separate models for the four different recruitment and incentive categories. Table 2 summarises the results.

3. Findings

For the newly added group (no recruitment but with incentive payments, NRxI) the effect of the response burden is not significant, as expected because of limited sample size. Due to the logistic transformation, parameters reflect changes in log-odds when the RBS marginally increases. The effect of response burden is generally more negative than previously expected. The strongest effect can be found for the recruited subsample without incentive payment (RxNI), where the odds of participating decrease by -0.297 (i.e., roughly 30%) according to Eq. (2) if the RBS increases by 100 points. The other comparisons are listed in Table 3.

The back-transformed relationship between response burden and response rates (response rate curve) is visualized in Fig. 2, along their confidence intervals (i.e., the shaded area reflects the uncertainty of the curve estimates and is not a prediction interval). Recruitment shifts the curve, while incentives flatten it. Notably, the domain above a response burden score of 2’000 is sparsely populated, and the few observations potentially strongly influence the curve’s shape (however, according to Cook’s distance no influential outliers are present in our data). The results indicate that surveys beyond 2’000 points appear overly burdensome for respondents, sustaining high response rates only through recruitment efforts combined with incentive payments, intensive care of the respondents and general interest of the respondents in the topic of these intense studies.

Fig. 2.Response rate curves (response rates as a function of the response burden). The left-hand panel compares the curves for each recruitment $x$ incentive category based on the four separately estimated models (RxI stands for recruited, with incentives, $N$ negates). The right-hand (smaller) panels compare the response rate curves to the ones based on the data of the previous publication (pink lines). New data points (since the last publication) are enlarged.

Generally, the 14 new data points do not dramatically change the overall shape of the curves (Fig. 2, RHS), but the curves are slightly steeper as explained based on the parameter estimates. RxI is almost identical (only two new data points were added). For NRxNI the confidence bounds increased because two of the five added surveys have unprecedented high response rates. For the category RxNI the function has gained support for higher response burdens which substantially steepened the curve and reduced its uncertainty. In particular, we now have higher confidence that the curve quickly joins the other response curves on the domain above 1’500 response burden points. I.e., recruitment without incentive payments only matters for surveys with low response burden (but can make a big difference there).

Similar to Schmid and Axhausen (2019b), we can add a linear time-trend with the year 2004 (when the survey scoring effort started) serving as the base. The following code shows the trivial addition of the time-trend to the pooled model.

R> dat$year <- dat$year - 2004
R> fit_t <- fit_t <- lm(y ~ 0 + x + RxI + RxNI + NRxI + NRxNI + year,
+                       data = dat, weights = weight)
R> mt <- add_clustered(fit_t, cluster = dat$survey_id, type = "CR2")

We repeat the steps for the individual categories and synthesise the results in Table 4. In contrast to Schmid and Axhausen (2019b) we do not find a negative time-trend and therefore do not support the hypothesis of a general fatigue and less willingness to participate in our surveys.

Acknowledgements

We would like to thank all the students scoring the survey instruments and contributing to the data underlying this analysis.