Did you know that when you spend time on an online platform, you could be experiencing between six to eight different experimental treatments that stem from several hundred A/B tests that run concurrently? That's how common digital experimentation is today. And while this may be acceptable in industry, large-scale digital experimentation poses some substantial challenges for researchers wanting to evaluate theories and disconfirm hypotheses through randomized controlled trials done on digital platforms. Thankfully, the brilliant Ahmed Abbasi has a new paper forthcoming that illuminates the orthogonal testing plane problem and offers some guidelines for sidestepping the issue. So if experiments are your thing, you really need to listen to what is really going on out there.

References

Abbasi, A., Somanchi, S., & Kelley, K. (2024). The Critical Challenge of using Large-scale Digital Experiment Platforms for Scientific Discovery. MIS Quarterly, https://doi.org/10.25300/MISQ/2024/18201.

Miranda, S. M., Berente, N., Seidel, S., Safadi, H., & Burton-Jones, A. (2022). Computationally Intensive Theory Construction: A Primer for Authors and Reviewers. MIS Quarterly, 46(2), i-xvi.

Karahanna, E., Benbasat, I., Bapna, R., & Rai, A. (2018). Editor's Comments: Opportunities and Challenges for Different Types of Online Experiments. MIS Quarterly, 42(4), iii-x.

Kohavi, R., & Thomke, S. (2017). The Surprising Power of Online Experiments. Harvard Business Review, 95(5), 74-82.

Fisher, R. A. (1935). The Design of Experiments. Oliver and Boyd.

Pienta, D., Vishwamitra, N., Somanchi, S., Berente, N., & Thatcher, J. B. (2024). Do Crowds Validate False Data? Systematic Distortion and Affective Polarization. MIS Quarterly, https://doi.org/10.25300/MISQ/2024/17482.

Bapna, R., Goes, P. B., Gupta, A., & Jin, Y. (2004). User Heterogeneity and Its Impact on Electronic Auction Market Design: An Empirical Exploration. MIS Quarterly, 28(1), 21-43.

Somanchi, S., Abbasi, A., Kelley, K., Dobolyi, D., & Yuan, T. T. (2023). Examining User Heterogeneity in Digital Experiments. ACM Transactions on Information Systems, 41(4), 1-34.

Mertens, W., & Recker, J. (2020). New Guidelines for Null Hypothesis Significance Testing in Hypothetico-Deductive IS Research. Journal of the Association for Information Systems, 21(4), 1072-1102.

GRADE Working Group. (2004). Grading Quality of Evidence and Strength of Recommendations. British Medical Journal, 328(7454), 1490-1494.

Abbasi, A., Parsons, J., Pant, G., Liu Sheng, O. R., & Sarker, S. (2024). Pathways for Design Research on Artificial Intelligence. Information Systems Research, 35(2), 441-459.

Abbasi, A., Chiang, R. H. L., & Xu, J. (2023). Data Science for Social Good. Journal of the Association for Information Systems, 24(6), 1439-1458.

Babar, Y., Mahdavi Adeli, A., & Burtch, G. (2023). The Effects of Online Social Identity Signals on Retailer Demand. Management Science, 69(12), 7335-7346.

Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design Science in Information Systems Research. MIS Quarterly, 28(1), 75-105.

Kahneman, D., & Tversky, A. (1979). Prospect Theory: An Analysis of Decision under Risk. Econometrica, 47(2), 263-291.

Benbasat, I., & Zmud, R. W. (2003). The Identity Crisis Within The IS Discipline: Defining and Communicating The Discipline's Core Properties. MIS Quarterly, 27(2), 183-194.

Gregor, S., & Hevner, A. R. (2013). Positioning and Presenting Design Science Research for Maximum Impact. MIS Quarterly, 37(2), 337-355.

Rai, A. (2017). Editor's Comments: Avoiding Type III Errors: Formulating IS Research Problems that Matter. MIS Quarterly, 41(2), iii-vii.

Burton-Jones, A. (2023). Editor's Comments: Producing Significant Research. MIS Quarterly, 47(1), i-xv. 

Abbasi, A., Dillon, R., Rao, H. R., & Liu Sheng, O. R. (2024). Preparedness and Response in the Century of Disasters: Overview of Information Systems Research Frontiers. Information Systems Research, 35(2), 460-468.