Thesis Title
Augmented Insight: Leveraging Large Language Models for Analysis of Polarised Discourse on Social Issues in News Content

Research Description
While Large Language Models (LLMs) are adept at analysing natural language in response to user-generated prompts, users lack insight into their internal mechanisms (Bender et al., 2021; Ziems et al., 2023). Their use as research tools depends on understanding the quality, accuracy, and ethical implications of their outputs (Liang et al., 2023). Large Language Models (LLMs) offer accessibility and cost-effectiveness in their usage; however, their ‘black box’ nature makes evaluating outputs a challenge for research (Liang et al., 2023; Ouyang et al., 2023; Zhao et al., 2021). The application of LLMs in research prompts consideration of how they can aid in tasks such as detecting polarised discourse within online news media and enhance our understanding of the evolution of societal narratives.I will develop a software pipeline and evaluation framework to apply LLMs to social science tasks, such as news content analysis, stance detection and frame analysis. This project will contribute to broader discussions on polarisation by demonstrating how LLMs can be leveraged to detect polarised perspectives in online media.

References:
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? . Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–623. https://doi.org/10.1145/3442188.3445922

Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., Newman, B., Yuan, B., Yan, B., Zhang, C., Cosgrove, C., Manning, C. D., Ré, C., Acosta-Navas, D., Hudson, D. A., … Koreeda, Y. (2023). Holistic Evaluation of Language Models (arXiv:2211.09110). arXiv. https://doi.org/10.48550/arXiv.2211.09110

Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2023). LLM is Like a Box of Chocolates: The Non-determinism of ChatGPT in Code Generation (arXiv:2308.02828). arXiv. http://arxiv.org/abs/2308.02828

Zhao, Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021). Calibrate Before Use: Improving Few-shot Performance of Language Models. Proceedings of the 38th International Conference on Machine Learning, 12697–12706. https://proceedings.mlr.press/v139/zhao21c.html

Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2023). Can Large Language Models Transform Computational Social Science? (arXiv:2305.03514). arXiv. http://arxiv.org/abs/2305.03514

Supervisors
Dr Katharina Esau
Professor Axel Bruns
Professor Nicolas Suzor
Professor Daniel Angus