What Does ChatGPT Know About Information Systems?
Abstract
Large language models such as ChatGPT provide efficient access to a wealth of information. However, there are significant questions regarding the depth and quality of knowledge in any one domain. This paper focuses specifically on the information systems (IS) field and assesses ChatGPT's knowledge. To analyze the extent and quality of information systems knowledge derived from queries to ChatGPT, we used over 3,000 queries from a broad range of exam and quiz questions. These queries were obtained from university courses and professional information system certification exams. The query topics are based on a framework for information systems education, with queries gathered from multiple faculty members at different universities. The results of querying ChatGPT with these questions find that ChatGPT can answer 65% to 85% of information systems queries successfully, across each of the areas of the framework. ChatGPT performed well on essay, true/false and multiple-choice questions, with no statistical difference between its success rate on multiple-choice and true/false questions. We conclude that ChatGPT tends to perform approximately at the level of an average student, but may not perform at a level sufficient to pass certain professional exams. ChatGPT's knowledge is very broad, covering virtual ly all areas of the information systems knowledge we identified, but not at an expert level.
Recommended Citation
O'leary, D. E., Storey, V. C., Buckman, J. R., Green, A. W., Niederman, F., Templeton, G., French, A. M., Chua, C. E., Gu, G., Pereira, F., & Wallace, L. (2025). What Does ChatGPT Know About Information Systems?. Communications of the Association for Information Systems, 56 Association for Information Systems.
Department(s)
Business and Information Technology
Keywords and Phrases
ChatGPT; Generative AI (GenAI); Information Systems Competency; Knowledge; Large Language Models (LLM)
International Standard Serial Number (ISSN)
1529-3181
Document Type
Article - Journal
Document Version
Final Version
File Type
text
Language(s)
English
Rights
© 2025 Association for Information Systems, All rights reserved.
Publication Date
01 Jan 2025
