Publications
Publications by categories in reversed chronological order
2024
- Exploring Parent’s Needs for Children-Centered AI to Support Preschoolers’ Storytelling and Reading ActivitiesarXiv preprint arXiv:2401.13804, 2024
Interactive storytelling is vital for preschooler development. While children’s interactive partners have traditionally been their parents and teachers, recent advances in artificial intelligence (AI) have sparked a surge of AI-based storytelling technologies. As these technologies become increasingly ubiquitous in preschoolers’ lives, questions arise regarding how they function in practical storytelling scenarios and, in particular, how parents, the most critical stakeholders, experience and perceive these technologies. This paper investigates these questions through a qualitative study with 17 parents of children aged 3-6. Our findings suggest that even though AI-based storytelling technologies provide more immersive and engaging interaction, they still cannot meet parents’ expectations due to a series of interactive, functional, and algorithmic challenges. We elaborate on these challenges and discuss the possible implications of future AI-based storytelling technologies for preschoolers. We conclude by highlighting the design implications for future AI-based storytelling technologies.
@article{sun2024exploring, title = {Exploring Parent's Needs for Children-Centered AI to Support Preschoolers' Storytelling and Reading Activities}, author = {Sun, Yuling and Liu, Jiali and Yao, Bingsheng and Chen, Jiaju and Wang, Dakuo and Ma, Xiaojuan and Lu, Yuxuan and Xu, Ying and He, Liang}, journal = {arXiv preprint arXiv:2401.13804}, year = {2024}, } - Who Changed the Destiny of Rural Students, and How?: Unpacking ICT-Mediated Remote Education in Rural ChinaYuling Sun, Xiuqi Zhu, Xiaomu Zhou, Bingsheng Yao , Kai Zhang, and 3 more authorsarXiv preprint arXiv:2401.13799, 2024
The proliferation of Information and Communication Technologies (ICTs) has shown great promise in addressing educational challenges facing rural areas. However, the complex rural context poses significant challenges to the effective utilization of these technologies. This paper examines the empirical integration of live-streaming-based remote classrooms (LSRC) through a qualitative study in rural China. Our findings suggest that while LSRC enables rural students equal access to high-quality educational resources, its practical integration faces numerous challenges. In particular, we emphasize the crucial role of local teachers in addressing these challenges, ultimately achieving the desired improvement of students’ learning outcomes. We also examine the impact of LSRC on the original rural education ecosystem. Building upon our findings, we call for a reconsideration of interaction paradigms and evaluation systems of ICT-mediated rural education, emphasizing the significance of rural teachers. We conclude by discussing the implications for future ICT-mediated technology interventions in rural settings.
@article{sun2024changed, title = {Who Changed the Destiny of Rural Students, and How?: Unpacking ICT-Mediated Remote Education in Rural China}, author = {Sun, Yuling and Zhu, Xiuqi and Zhou, Xiaomu and Yao, Bingsheng and Zhang, Kai and Wang, Dakuo and Chen, Jiaju and He, Liang}, journal = {arXiv preprint arXiv:2401.13799}, year = {2024}, }
2023
- More Samples or More Prompt Inputs? Exploring Effective In-Context Sampling for LLM Few-Shot Prompt EngineeringBingsheng Yao , Guiming Chen, Ruishi Zou, Yuxuan Lu, Jiachen Li, and 4 more authorsarXiv preprint arXiv:2311.09782, 2023
While most existing works on LLM prompt-engineering focus only on how to select a better set of data samples inside one single prompt input (In-Context Learning or ICL), why can’t we design and leverage multiple prompt inputs together to further improve the LLM performance? In this work, we propose In-Context Sampling (ICS), a low-resource LLM prompt-engineering technique to produce the most confident prediction results by optimizing the construction of multiple ICL prompt inputs. Extensive experiments with two SOTA LLMs (FlanT5-XL and Mistral-7B) on three NLI datasets (e-SNLI, Multi-NLI, and ANLI) illustrate that ICS can consistently enhance LLM’s prediction performance and confidence. An ablation study suggests that a diversity-based ICS strategy may further improve LLM’s performance, which sheds light on a new yet promising future research direction.
@article{yao2023samples, title = {More Samples or More Prompt Inputs? Exploring Effective In-Context Sampling for LLM Few-Shot Prompt Engineering}, author = {Yao, Bingsheng and Chen, Guiming and Zou, Ruishi and Lu, Yuxuan and Li, Jiachen and Zhang, Shao and Liu, Sijia and Hendler, James and Wang, Dakuo}, year = {2023}, journal = {arXiv preprint arXiv:2311.09782}, } - Human Still Wins over LLM: An Empirical Study of Active Learning on Domain-Specific Annotation TasksYuxuan Lu, Bingsheng Yao , Shao Zhang, Yun Wang, Peng Zhang, and 3 more authorsarXiv preprint arXiv:2311.09825, 2023
Large Language Models (LLMs) have demonstrated considerable advances, and several claims have been made about their exceeding human performance. However, in real-world tasks, domain knowledge is often required. Low-resource learning methods like Active Learning (AL) have been proposed to tackle the cost of domain expert annotation, raising this question: Can LLMs surpass compact models trained with expert annotations in domain-specific tasks? In this work, we conduct an empirical experiment on four datasets from three different domains comparing SOTA LLMs with small models trained on expert annotations with AL. We found that small models can outperform GPT-3.5 with a few hundreds of labeled data, and they achieve higher or similar performance with GPT-4 despite that they are hundreds time smaller. Based on these findings, we posit that LLM predictions can be used as a warmup method in real-world applications and human experts remain indispensable in tasks involving data annotation driven by domain-specific knowledge.
@article{lu2023human, title = {Human Still Wins over LLM: An Empirical Study of Active Learning on Domain-Specific Annotation Tasks}, author = {Lu, Yuxuan and Yao, Bingsheng and Zhang, Shao and Wang, Yun and Zhang, Peng and Lu, Tun and Li, Toby Jia-Jun and Wang, Dakuo}, year = {2023}, journal = {arXiv preprint arXiv:2311.09825}, } - FairytaleCQA: Integrating a Commonsense Knowledge Graph into Children’s Storybook NarrativesJiaju Chen, Yuxuan Lu, Shao Zhang, Bingsheng Yao , Yuanzhe Dong, and 5 more authorsarXiv preprint arXiv:2311.09756, 2023
AI models (including LLM) often rely on narrative question-answering (QA) datasets to provide customized QA functionalities to support downstream children education applications; however, existing datasets only include QA pairs that are grounded within the given storybook content, but children can learn more when teachers refer the storybook content to real-world knowledge (e.g., commonsense knowledge). We introduce the FairytaleCQA dataset, which is annotated by children education experts, to supplement 278 storybook narratives with educationally appropriate commonsense knowledge. The dataset has 5,868 QA pairs that not only originate from the storybook narrative but also contain the commonsense knowledge grounded by an external knowledge graph (i.e., ConceptNet). A follow-up experiment shows that a smaller model (T5-large) fine-tuned with FairytaleCQA reliably outperforms much larger prompt-engineered LLM (e.g., GPT-4) in this new QA-pair generation task (QAG). This result suggests that: 1) our dataset brings novel challenges to existing LLMs, and 2) human experts’ data annotation are still critical as they have much nuanced knowledge that LLMs do not know in the children educational domain.
@article{chen2023fairytalecqa, title = {FairytaleCQA: Integrating a Commonsense Knowledge Graph into Children's Storybook Narratives}, author = {Chen, Jiaju and Lu, Yuxuan and Zhang, Shao and Yao, Bingsheng and Dong, Yuanzhe and Xu, Ying and Li, Yunyao and Wang, Qianwen and Wang, Dakuo and Sun, Yuling}, year = {2023}, journal = {arXiv preprint arXiv:2311.09756}, } - " Mango Mango, How to Let The Lettuce Dry Without A Spinner?”: Exploring User Perceptions of Using An LLM-Based Conversational Assistant Toward Cooking PartnerSzeyi Chan, Jiachen Li, Bingsheng Yao , Amama Mahmood, Chien-Ming Huang, and 3 more authorsarXiv preprint arXiv:2310.05853, 2023
The rapid advancement of the Large Language Model (LLM) has created numerous potentials for integration with conversational assistants (CAs) assisting people in their daily tasks, particularly due to their extensive flexibility. However, users’ real-world experiences interacting with these assistants remain unexplored. In this research, we chose cooking, a complex daily task, as a scenario to investigate people’s successful and unsatisfactory experiences while receiving assistance from an LLM-based CA, Mango Mango. We discovered that participants value the system’s ability to provide extensive information beyond the recipe, offer customized instructions based on context, and assist them in dynamically planning the task. However, they expect the system to be more adaptive to oral conversation and provide more suggestive responses to keep users actively involved. Recognizing that users began treating our LLM-CA as a personal assistant or even a partner rather than just a recipe-reading tool, we propose several design considerations for future development.
@article{chan2023mango, title = {" Mango Mango, How to Let The Lettuce Dry Without A Spinner?'': Exploring User Perceptions of Using An LLM-Based Conversational Assistant Toward Cooking Partner}, author = {Chan, Szeyi and Li, Jiachen and Yao, Bingsheng and Mahmood, Amama and Huang, Chien-Ming and Jimison, Holly and Mynatt, Elizabeth D and Wang, Dakuo}, journal = {arXiv preprint arXiv:2310.05853}, year = {2023}, } - LLM-Powered Conversational Voice Assistants: Interaction Patterns, Opportunities, Challenges, and Design GuidelinesAmama Mahmood, Junxiang Wang, Bingsheng Yao , Dakuo Wang, and Chien-Ming HuangarXiv preprint arXiv:2309.13879, 2023
Conventional Voice Assistants (VAs) rely on traditional language models to discern user intent and respond to their queries, leading to interactions that often lack a broader contextual understanding, an area in which Large Language Models (LLMs) excel. However, current LLMs are largely designed for text-based interactions, thus making it unclear how user interactions will evolve if their modality is changed to voice. In this work, we investigate whether LLMs can enrich VA interactions via an exploratory study with participants (N=20) using a ChatGPT-powered VA for three scenarios (medical self-diagnosis, creative planning, and debate) with varied constraints, stakes, and objectivity. We observe that LLM-powered VA elicits richer interaction patterns that vary across tasks, showing its versatility. Notably, LLMs absorb the majority of VA intent recognition failures. We additionally discuss the potential of harnessing LLMs for more resilient and fluid user-VA interactions and provide design guidelines for tailoring LLMs for voice assistance.
@article{mahmood2023llm, title = {LLM-Powered Conversational Voice Assistants: Interaction Patterns, Opportunities, Challenges, and Design Guidelines}, author = {Mahmood, Amama and Wang, Junxiang and Yao, Bingsheng and Wang, Dakuo and Huang, Chien-Ming}, journal = {arXiv preprint arXiv:2309.13879}, year = {2023}, } - Talk2Care: Facilitating Asynchronous Patient-Provider Communication with Large-Language-ModelZiqi Yang, Xuhai Xu, Bingsheng Yao , Shao Zhang, Ethan Rogers, and 4 more authorsarXiv preprint arXiv:2309.09357, 2023
Despite the plethora of telehealth applications to assist home-based older adults and healthcare providers, basic messaging and phone calls are still the most common communication methods, which suffer from limited availability, information loss, and process inefficiencies. One promising solution to facilitate patient-provider communication is to leverage large language models (LLMs) with their powerful natural conversation and summarization capability. However, there is a limited understanding of LLMs’ role during the communication. We first conducted two interview studies with both older adults (N=10) and healthcare providers (N=9) to understand their needs and opportunities for LLMs in patient-provider asynchronous communication. Based on the insights, we built an LLM-powered communication system, Talk2Care, and designed interactive components for both groups: (1) For older adults, we leveraged the convenience and accessibility of voice assistants (VAs) and built an LLM-powered VA interface for effective information collection. (2) For health providers, we built an LLM-based dashboard to summarize and present important health information based on older adults’ conversations with the VA. We further conducted two user studies with older adults and providers to evaluate the usability of the system. The results showed that Talk2Care could facilitate the communication process, enrich the health information collected from older adults, and considerably save providers’ efforts and time. We envision our work as an initial exploration of LLMs’ capability in the intersection of healthcare and interpersonal communication.
@article{yang2023talk2care, title = {Talk2Care: Facilitating Asynchronous Patient-Provider Communication with Large-Language-Model}, author = {Yang, Ziqi and Xu, Xuhai and Yao, Bingsheng and Zhang, Shao and Rogers, Ethan and Intille, Stephen and Shara, Nawar and (Gordon) Gao, Guodong and Wang, Dakuo}, journal = {arXiv preprint arXiv:2309.09357}, year = {2023}, } - " It’s a Fair Game”, or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational AgentsZhiping Zhang, Michelle Jia, Bingsheng Yao , Sauvik Das, Ada Lerner, and 3 more authorsarXiv preprint arXiv:2309.11653, 2023
The widespread use of Large Language Model (LLM)-based conversational agents (CAs), especially in high-stakes domains, raises many privacy concerns. Building ethical LLM-based CAs that respect user privacy requires an in-depth understanding of the privacy risks that concern users the most. However, existing research, primarily model-centered, does not provide insight into users’ perspectives. To bridge this gap, we analyzed sensitive disclosures in real-world ChatGPT conversations and conducted semi-structured interviews with 19 LLM-based CA users. We found that users are constantly faced with trade-offs between privacy, utility, and convenience when using LLM-based CAs. However, users’ erroneous mental models and the dark patterns in system design limited their awareness and comprehension of the privacy risks. Additionally, the human-like interactions encouraged more sensitive disclosures, which complicated users’ ability to navigate the trade-offs. We discuss practical design guidelines and the needs for paradigmatic shifts to protect the privacy of LLM-based CA users.
@article{zhang2023s, title = {" It's a Fair Game'', or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents}, author = {Zhang, Zhiping and Jia, Michelle and Yao, Bingsheng and Das, Sauvik and Lerner, Ada and Wang, Dakuo and Li, Tianshi and others}, journal = {arXiv preprint arXiv:2309.11653}, year = {2023}, } - Rethinking Human-AI Collaboration in Complex Medical Decision Making: A Case Study in Sepsis DiagnosisShao Zhang, Jianing Yu, Xuhai Xu, Changchang Yin, Yuxuan Lu, and 6 more authorsarXiv preprint arXiv:2309.12368, 2023
Today’s AI systems for medical decision support often succeed on benchmark datasets in research papers but fail in real-world deployment. This work focuses on the decision making of sepsis, an acute life-threatening systematic infection that requires an early diagnosis with high uncertainty from the clinician. Our aim is to explore the design requirements for AI systems that can support clinical experts in making better decisions for the early diagnosis of sepsis. The study begins with a formative study investigating why clinical experts abandon an existing AI-powered Sepsis predictive module in their electrical health record (EHR) system. We argue that a human-centered AI system needs to support human experts in the intermediate stages of a medical decision-making process (e.g., generating hypotheses or gathering data), instead of focusing only on the final decision. Therefore, we build SepsisLab based on a state-of-the-art AI algorithm and extend it to predict the future projection of sepsis development, visualize the prediction uncertainty, and propose actionable suggestions (i.e., which additional laboratory tests can be collected) to reduce such uncertainty. Through heuristic evaluation with six clinicians using our prototype system, we demonstrate that SepsisLab enables a promising human-AI collaboration paradigm for the future of AI-assisted sepsis diagnosis and other high-stakes medical decision making.
@article{zhang2023rethinking, title = {Rethinking Human-AI Collaboration in Complex Medical Decision Making: A Case Study in Sepsis Diagnosis}, author = {Zhang, Shao and Yu, Jianing and Xu, Xuhai and Yin, Changchang and Lu, Yuxuan and Yao, Bingsheng and Tory, Melanie and Padilla, Lace M and Caterino, Jeffrey and Zhang, Ping and others}, journal = {arXiv preprint arXiv:2309.12368}, year = {2023}, } - Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text DataXuhai Xu, Bingsheng Yao , Yuanzhe Dong, Saadia Gabriel, Hong Yu, and 4 more authorsarXiv preprint arXiv:2307.14385, 2023
Advances in large language models (LLMs) have empowered a variety of applications. However, there is still a significant gap in research when it comes to understanding and enhancing the capabilities of LLMs in the field of mental health. In this work, we present the first comprehensive evaluation of multiple LLMs, including Alpaca, Alpaca-LoRA, FLAN-T5, GPT-3.5, and GPT-4, on various mental health prediction tasks via online text data. We conduct a broad range of experiments, covering zero-shot prompting, few-shot prompting, and instruction fine-tuning. The results indicate a promising yet limited performance of LLMs with zero-shot and few-shot prompt designs for the mental health tasks. More importantly, our experiments show that instruction finetuning can significantly boost the performance of LLMs for all tasks simultaneously. Our best-finetuned models, Mental-Alpaca and Mental-FLAN-T5, outperform the best prompt design of GPT-3.5 (25 and 15 times bigger) by 10.9% on balanced accuracy and the best of GPT-4 (250 and 150 times bigger) by 4.8%. They further perform on par with the state-of-the-art task-specific language model. We also conduct an exploratory case study on LLMs’ capability on the mental health reasoning tasks, illustrating the promising capability of certain models such as GPT-4. We summarize our findings into a set of action guidelines for potential methods to enhance LLMs’ capability for mental health tasks. Meanwhile, we also emphasize the important limitations before achieving deployability in real-world mental health settings, such as known racial and gender bias. We highlight the important ethical risks accompanying this line of research.
@article{xu2023mental, title = {Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data}, author = {Xu, Xuhai and Yao, Bingsheng and Dong, Yuanzhe and Gabriel, Saadia and Yu, Hong and Hendler, James and Ghassemi, Marzyeh and Dey, Anind K and Wang, Dakuo}, journal = {arXiv preprint arXiv:2307.14385}, year = {2023}, } - Beyond Labels: Empowering Human Annotators with Natural Language Explanations through a Novel Active-Learning ArchitectureIn Findings of the Association for Computational Linguistics: EMNLP 2023, Dec 2023
Real-world domain experts (e.g., doctors) rarely annotate only a decision label in their day-to-day workflow without providing explanations. Yet, existing low-resource learning techniques, such as Active Learning (AL), that aim to support human annotators mostly focus on the label while neglecting the natural language explanation of a data point. This work proposes a novel AL architecture to support experts’ real-world need for label and explanation annotations in low-resource scenarios. Our AL architecture leverages an explanation-generation model to produce explanations guided by human explanations, a prediction model that utilizes generated explanations toward prediction faithfully, and a novel data diversity-based AL sampling strategy that benefits from the explanation annotations. Automated and human evaluations demonstrate the effectiveness of incorporating explanations into AL sampling and the improved human annotation efficiency and trustworthiness with our AL architecture. Additional ablation studies illustrate the potential of our AL architecture for transfer learning, generalizability, and integration with large language models (LLMs). While LLMs exhibit exceptional explanation-generation capabilities for relatively simple tasks, their effectiveness in complex real-world tasks warrants further in-depth study.
@inproceedings{yao-etal-2023-beyond, title = {Beyond Labels: Empowering Human Annotators with Natural Language Explanations through a Novel Active-Learning Architecture}, author = {Yao, Bingsheng and Jindal, Ishan and Popa, Lucian and Katsis, Yannis and Ghosh, Sayan and He, Lihong and Lu, Yuxuan and Srivastava, Shashank and Li, Yunyao and Hendler, James and Wang, Dakuo}, editor = {Bouamor, Houda and Pino, Juan and Bali, Kalika}, booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023}, month = dec, year = {2023}, address = {Singapore}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2023.findings-emnlp.778}, doi = {10.18653/v1/2023.findings-emnlp.778}, pages = {11629--11643}, } - Are Human Explanations Always Helpful? Towards Objective Evaluation of Human Natural Language ExplanationsIn Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Jul 2023
Human-annotated labels and explanations are critical for training explainable NLP models. However, unlike human-annotated labels whose quality is easier to calibrate (e.g., with a majority vote), human-crafted free-form explanations can be quite subjective. Before blindly using them as ground truth to train ML models, a vital question needs to be asked: How do we evaluate a human-annotated explanation’s quality? In this paper, we build on the view that the quality of a human-annotated explanation can be measured based on its helpfulness (or impairment) to the ML models’ performance for the desired NLP tasks for which the annotations were collected. In comparison to the commonly used Simulatability score, we define a new metric that can take into consideration the helpfulness of an explanation for model performance at both fine-tuning and inference. With the help of a unified dataset format, we evaluated the proposed metric on five datasets (e.g., e-SNLI) against two model architectures (T5 and BART), and the results show that our proposed metric can objectively evaluate the quality of human-annotated explanations, while Simulatability falls short.
@inproceedings{yao-etal-2023-human, title = {Are Human Explanations Always Helpful? Towards Objective Evaluation of Human Natural Language Explanations}, author = {Yao, Bingsheng and Sen, Prithviraj and Popa, Lucian and Hendler, James and Wang, Dakuo}, editor = {Rogers, Anna and Boyd-Graber, Jordan and Okazaki, Naoaki}, booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, month = jul, year = {2023}, address = {Toronto, Canada}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2023.acl-long.821}, doi = {10.18653/v1/2023.acl-long.821}, pages = {14698--14713}, } - ‘Don’t Get Too Technical with Me’: A Discourse Structure-Based Framework for Automatic Science JournalismRonald Cardenas, Bingsheng Yao , Dakuo Wang, and Yufang HouIn Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Dec 2023
Science journalism refers to the task of reporting technical findings of a scientific paper as a less technical news article to the general public audience. We aim to design an automated system to support this real-world task (i.e., automatic science journalism ) by 1) introducing a newly-constructed and real-world dataset (SciTechNews), with tuples of a publicly-available scientific paper, its corresponding news article, and an expert-written short summary snippet; 2) proposing a novel technical framework that integrates a paper’s discourse structure with its metadata to guide generation; and, 3) demonstrating with extensive automatic and human experiments that our model outperforms other baseline methods (e.g. Alpaca and ChatGPT) in elaborating a content plan meaningful for the target audience, simplify the information selected, and produce a coherent final report in a layman’s style.
@inproceedings{cardenas-etal-2023-dont, title = {{`}Don{'}t Get Too Technical with Me{'}: A Discourse Structure-Based Framework for Automatic Science Journalism}, author = {Cardenas, Ronald and Yao, Bingsheng and Wang, Dakuo and Hou, Yufang}, editor = {Bouamor, Houda and Pino, Juan and Bali, Kalika}, booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing}, month = dec, year = {2023}, address = {Singapore}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2023.emnlp-main.76}, doi = {10.18653/v1/2023.emnlp-main.76}, pages = {1186--1202}, }
2022
- It is AI’s Turn to Ask Humans a Question: Question-Answer Pair Generation for Children’s Story BooksIn Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), May 2022
Existing question answering (QA) techniques are created mainly to answer questions asked by humans. But in educational applications, teachers often need to decide what questions they should ask, in order to help students to improve their narrative understanding capabilities. We design an automated question-answer generation (QAG) system for this education scenario: given a story book at the kindergarten to eighth-grade level as input, our system can automatically generate QA pairs that are capable of testing a variety of dimensions of a student’s comprehension skills. Our proposed QAG model architecture is demonstrated using a new expert-annotated FairytaleQA dataset, which has 278 child-friendly storybooks with 10,580 QA pairs. Automatic and human evaluations show that our model outperforms state-of-the-art QAG baseline systems. On top of our QAG system, we also start to build an interactive story-telling application for the future real-world deployment in this educational scenario.
@inproceedings{yao-etal-2022-ais, title = {It is {AI}{'}s Turn to Ask Humans a Question: Question-Answer Pair Generation for Children{'}s Story Books}, author = {Yao*, Bingsheng and Wang*, Dakuo and Wu, Tongshuang and Zhang, Zheng and Li, Toby and Yu, Mo and Xu, Ying}, booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, month = may, year = {2022}, address = {Dublin, Ireland}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2022.acl-long.54}, doi = {10.18653/v1/2022.acl-long.54}, pages = {731--744}, } - Fantastic Questions and Where to Find Them: FairytaleQA – An Authentic Dataset for Narrative ComprehensionIn Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), May 2022
Question answering (QA) is a fundamental means to facilitate assessment and training of narrative comprehension skills for both machines and young children, yet there is scarcity of high-quality QA datasets carefully designed to serve this purpose. In particular, existing datasets rarely distinguish fine-grained reading skills, such as the understanding of varying narrative elements. Drawing on the reading education research, we introduce FairytaleQA, a dataset focusing on narrative comprehension of kindergarten to eighth-grade students. Generated by educational experts based on an evidence-based theoretical framework, FairytaleQA consists of 10,580 explicit and implicit questions derived from 278 children-friendly stories, covering seven types of narrative elements or relations. Our dataset is valuable in two folds: First, we ran existing QA models on our dataset and confirmed that this annotation helps assess models’ fine-grained learning skills. Second, the dataset supports question generation (QG) task in the education domain. Through benchmarking with QG models, we show that the QG model trained on FairytaleQA is capable of asking high-quality and more diverse questions.
@inproceedings{xu-etal-2022-fantastic, title = {Fantastic Questions and Where to Find Them: {F}airytale{QA} {--} An Authentic Dataset for Narrative Comprehension}, author = {Xu*, Ying and Wang*, Dakuo and Yu*, Mo and Ritchie*, Daniel and Yao*, Bingsheng and Wu, Tongshuang and Zhang, Zheng and Li, Toby and Bradford, Nora and Sun, Branda and Hoang, Tran and Sang, Yisi and Hou, Yufang and Ma, Xiaojuan and Yang, Diyi and Peng, Nanyun and Yu, Zhou and Warschauer, Mark}, booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, month = may, year = {2022}, address = {Dublin, Ireland}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2022.acl-long.34}, doi = {10.18653/v1/2022.acl-long.34}, pages = {447--460}, } - StoryBuddy: A Human-AI Collaborative Chatbot for Parent-Child Interactive Storytelling with Flexible Parental InvolvementIn Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, May 2022
Despite its benefits for children’s skill development and parent-child bonding, many parents do not often engage in interactive storytelling by having story-related dialogues with their child due to limited availability or challenges in coming up with appropriate questions. While recent advances made AI generation of questions from stories possible, the fully-automated approach excludes parent involvement, disregards educational goals, and underoptimizes for child engagement. Informed by need-finding interviews and participatory design (PD) results, we developed StoryBuddy, an AI-enabled system for parents to create interactive storytelling experiences. StoryBuddy’s design highlighted the need for accommodating dynamic user needs between the desire for parent involvement and parent-child bonding and the goal of minimizing parent intervention when busy. The PD revealed varied assessment and educational goals of parents, which StoryBuddy addressed by supporting configuring question types and tracking child progress. A user study validated StoryBuddy’s usability and suggested design insights for future parent-AI collaboration systems.
@inproceedings{10.1145/3491102.3517479, author = {Zhang*, Zheng and Xu*, Ying and Wang, Yanhao and Yao, Bingsheng and Ritchie, Daniel and Wu, Tongshuang and Yu, Mo and Wang, Dakuo and Li, Toby Jia-Jun}, title = {StoryBuddy: A Human-AI Collaborative Chatbot for Parent-Child Interactive Storytelling with Flexible Parental Involvement}, year = {2022}, isbn = {9781450391573}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3491102.3517479}, doi = {10.1145/3491102.3517479}, booktitle = {Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems}, articleno = {218}, numpages = {21}, keywords = {child-agent interactions, interactive storytelling, human-AI collaboration, co-reading, voice user interfaces, dialogic reading}, location = {New Orleans, LA, USA}, series = {CHI '22}, } - Nece: Narrative Event Chain Extraction ToolkitGuangxuan Xu*, Paulina Toro Isaza*, Moshi Li*, Akintoye Oloko, Bingsheng Yao , and 5 more authorsarXiv preprint arXiv:2208.08063, May 2022
To understand a narrative, it is essential to comprehend the temporal event flows, especially those associated with main characters; however, this can be challenging with lengthy and unstructured narrative texts. To address this, we introduce NECE, an open-access, document-level toolkit that automatically extracts and aligns narrative events in the temporal order of their occurrence. Through extensive evaluations, we show the high quality of the NECE toolkit and demonstrates its downstream application in analyzing narrative bias regarding gender. We also openly discuss the shortcomings of the current approach, and potential of leveraging generative models in future works. Lastly the NECE toolkit includes both a Python library and a user-friendly web interface, which offer equal access to professionals and layman audience alike, to visualize event chain, obtain narrative flows, or study narrative bias.
@article{xu2022nece, title = {Nece: Narrative Event Chain Extraction Toolkit}, author = {Xu*, Guangxuan and Isaza*, Paulina Toro and Li*, Moshi and Oloko, Akintoye and Yao, Bingsheng and Sanctos, Cassia and Adebiyi, Aminat and Hou, Yufang and Peng, Nanyun and Wang, Dakuo}, journal = {arXiv preprint arXiv:2208.08063}, year = {2022}, } - GEMv2: Multilingual NLG Benchmarking in a Single Line of CodeSebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, and 72 more authorsIn Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Dec 2022
Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work. The compatibility, often facilitated through leaderboards, thus leads to outdated but standardized evaluation practices. We pose that the standardization is taking place in the wrong spot. Evaluation infrastructure should enable researchers to use the latest methods and what should be standardized instead is how to incorporate these new evaluation advances.We introduce GEMv2, the new version of the Generation, Evaluation, and Metrics Benchmark which uses a modular infrastructure for dataset, model, and metric developers to benefit from each other’s work. GEMv2 supports 40 documented datasets in 51 languages, ongoing online evaluation for all datasets, and our interactive tools make it easier to add new datasets to the living benchmark.
@inproceedings{gehrmann-etal-2022-gemv2, title = {{GEM}v2: Multilingual {NLG} Benchmarking in a Single Line of Code}, author = {Gehrmann, Sebastian and Bhattacharjee, Abhik and Mahendiran, Abinaya and Wang, Alex and Papangelis, Alexandros and Madaan, Aman and Mcmillan-major, Angelina and Shvets, Anna and Upadhyay, Ashish and Bohnet, Bernd and Yao, Bingsheng and Wilie, Bryan and Bhagavatula, Chandra and You, Chaobin and Thomson, Craig and Garbacea, Cristina and Wang, Dakuo and Deutsch, Daniel and Xiong, Deyi and Jin, Di and Gkatzia, Dimitra and Radev, Dragomir and Clark, Elizabeth and Durmus, Esin and Ladhak, Faisal and Ginter, Filip and Winata, Genta Indra and Strobelt, Hendrik and Hayashi, Hiroaki and Novikova, Jekaterina and Kanerva, Jenna and Chim, Jenny and Zhou, Jiawei and Clive, Jordan and Maynez, Joshua and Sedoc, Jo{\~a}o and Juraska, Juraj and Dhole, Kaustubh and Chandu, Khyathi Raghavi and Beltrachini, Laura Perez and Ribeiro, Leonardo F . R. and Tunstall, Lewis and Zhang, Li and Pushkarna, Mahim and Creutz, Mathias and White, Michael and Kale, Mihir Sanjay and Eddine, Moussa Kamal and Daheim, Nico and Subramani, Nishant and Dusek, Ondrej and Liang, Paul Pu and Ammanamanchi, Pawan Sasanka and Zhu, Qi and Puduppully, Ratish and Kriz, Reno and Shahriyar, Rifat and Cardenas, Ronald and Mahamood, Saad and Osei, Salomey and Cahyawijaya, Samuel and {\v{S}}tajner, Sanja and Montella, Sebastien and Jolly, Shailza and Mille, Simon and Hasan, Tahmid and Shen, Tianhao and Adewumi, Tosin and Raunak, Vikas and Raheja, Vipul and Nikolaev, Vitaly and Tsai, Vivian and Jernite, Yacine and Xu, Ying and Sang, Yisi and Liu, Yixin and Hou, Yufang}, booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations}, month = dec, year = {2022}, address = {Abu Dhabi, UAE}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2022.emnlp-demos.27}, doi = {10.18653/v1/2022.emnlp-demos.27}, pages = {266--281}, } - A Corpus for Commonsense Inference in Story Cloze TestBingsheng Yao , Ethan Joseph, Julian Lioanag, and Mei SiIn Proceedings of the Thirteenth Language Resources and Evaluation Conference, Jun 2022
The Story Cloze Test (SCT) is designed for training and evaluating machine learning algorithms for narrative understanding and inferences. The SOTA models can achieve over 90% accuracy on predicting the last sentence. However, it has been shown that high accuracy can be achieved by merely using surface-level features. We suspect these models may not \textittruly understand the story. Based on the SCT dataset, we constructed a human-labeled and human-verified commonsense knowledge inference dataset. Given the first four sentences of a story, we asked crowd-source workers to choose from four types of narrative inference for deciding the ending sentence and which sentence contributes most to the inference. We accumulated data on 1871 stories, and three human workers labeled each story. Analysis of the intra-category and inter-category agreements show a high level of consensus. We present two new tasks for predicting the narrative inference categories and contributing sentences. Our results show that transformer-based models can reach SOTA performance on the original SCT task using transfer learning but don’t perform well on these new and more challenging tasks.
@inproceedings{yao-etal-2022-corpus, title = {A Corpus for Commonsense Inference in Story Cloze Test}, author = {Yao, Bingsheng and Joseph, Ethan and Lioanag, Julian and Si, Mei}, booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference}, month = jun, year = {2022}, address = {Marseille, France}, publisher = {European Language Resources Association}, url = {https://aclanthology.org/2022.lrec-1.375}, pages = {3500--3508}, } - Efficient Long Sequence Encoding via SynchronizationarXiv preprint arXiv:2203.07644, Jun 2022
Pre-trained Transformer models have achieved successes in a wide range of NLP tasks, but are inefficient when dealing with long input sequences. Existing studies try to overcome this challenge via segmenting the long sequence followed by hierarchical encoding or post-hoc aggregation. We propose a synchronization mechanism for hierarchical encoding. Our approach first identifies anchor tokens across segments and groups them by their roles in the original input sequence. Then inside Transformer layer, anchor embeddings are synchronized within their group via a self-attention module. Our approach is a general framework with sufficient flexibility – when adapted to a new task, it is easy to be enhanced with the task-specific anchor definitions. Experiments on two representative tasks with different types of long input texts, NarrativeQA summary setting and wild multi-hop reasoning from HotpotQA, demonstrate that our approach is able to improve the global information exchange among segments while maintaining efficiency.
@article{mou2022efficient, title = {Efficient Long Sequence Encoding via Synchronization}, author = {Mou, Xiangyang and Yu, Mo and Yao, Bingsheng and Huang, Lifu}, journal = {arXiv preprint arXiv:2203.07644}, year = {2022}, }
2021
- Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive StudyTransactions of the Association for Computational Linguistics, Jun 2021
Recent advancements in open-domain question answering (ODQA), that is, finding answers from large open-domain corpus like Wikipedia, have led to human-level performance on many datasets. However, progress in QA over book stories (Book QA) lags despite its similar task formulation to ODQA. This work provides a comprehensive and quantitative analysis about the difficulty of Book QA: (1) We benchmark the research on the NarrativeQA dataset with extensive experiments with cutting-edge ODQA techniques. This quantifies the challenges Book QA poses, as well as advances the published state-of-the-art with a ∼7% absolute improvement on ROUGE-L. (2) We further analyze the detailed challenges in Book QA through human studies.1 Our findings indicate that the event-centric questions dominate this task, which exemplifies the inability of existing QA models to handle event-oriented scenarios.
@article{mou-etal-2021-narrative, title = {Narrative Question Answering with Cutting-Edge Open-Domain {QA} Techniques: A Comprehensive Study}, author = {Mou*, Xiangyang and Yang*, Chenghao and Yu*, Mo and Yao, Bingsheng and Guo, Xiaoxiao and Potdar, Saloni and Su, Hui}, journal = {Transactions of the Association for Computational Linguistics}, volume = {9}, year = {2021}, address = {Cambridge, MA}, publisher = {MIT Press}, url = {https://aclanthology.org/2021.tacl-1.61}, doi = {10.1162/tacl_a_00411}, pages = {1032--1046}, } - Building a Storytelling Conversational Agent Through Parent-AI CollaborationJun 2021
In this paper, we describe the design of StoryBuddy, a prototype system that allows parents to collaborate with AI in creating interactive storytelling experiences. To accommodate dynamic user needs, StoryBuddy supports two modes: a parent-child joint reading mode where parents involve in book reading process and a child independent reading mode where parents have minimal involvement. StoryBuddy also allows parents to configure question types and track child progress, and generates questions automatically. A preliminary user study suggests that parents and children found Story-Buddy useful, helpful, and likable.
@article{zhang2021building, title = {Building a Storytelling Conversational Agent Through Parent-AI Collaboration}, author = {Zhang*, Zheng and Xu*, Ying and Wang, Yanhao and Wu, Tongshuang and Yao, Bingsheng and Ritchie, Daniel and Yu, Mo and Wang, Dakuo and Li, Toby Jia-Jun}, year = {2021}, }
2020
- Frustratingly Hard Evidence Retrieval For QA Over BooksarXiv preprint arXiv:2007.09878, Jun 2020
A lot of progress has been made to improve question answering (QA) in recent years, but the special problem of QA over narrative book stories has not been explored in-depth. We formulate BookQA as an open-domain QA task given its similar dependency on evidence retrieval. We further investigate how state-of-the-art open-domain QA approaches can help BookQA. Besides achieving state-of-the-art on the NarrativeQA benchmark, our study also reveals the difficulty of evidence retrieval in books with a wealth of experiments and analysis - which necessitates future effort on novel solutions for evidence retrieval in BookQA.
@article{mou2020frustratingly, title = {Frustratingly Hard Evidence Retrieval For QA Over Books}, author = {Mou, Xiangyang and Yu, Mo and Yao, Bingsheng and Yang, Chenghao and Guo, Xiaoxiao and Potdar, Saloni and Su, Hui}, journal = {arXiv preprint arXiv:2007.09878}, year = {2020}, } - Trust in AutoML: Exploring Information Needs for Establishing Trust in Automated Machine Learning SystemsIn Proceedings of the 25th International Conference on Intelligent User Interfaces, Jun 2020
We explore trust in a relatively new area of data science: Automated Machine Learning (AutoML). In AutoML, AI methods are used to generate and optimize machine learning models by automatically engineering features, selecting models, and optimizing hyperparameters. In this paper, we seek to understand what kinds of information influence data scientists’ trust in the models produced by AutoML? We operationalize trust as a willingness to deploy a model produced using automated methods. We report results from three studies - qualitative interviews, a controlled experiment, and a card-sorting task - to understand the information needs of data scientists for establishing trust in AutoML systems. We find that including transparency features in an AutoML tool increased user trust and understandability in the tool; and out of all proposed features, model performance metrics and visualizations are the most important information to data scientists when establishing their trust with an AutoML tool.
@inproceedings{10.1145/3377325.3377501, author = {Drozdal, Jaimie and Weisz, Justin and Wang, Dakuo and Dass, Gaurav and Yao, Bingsheng and Zhao, Changruo and Muller, Michael and Ju, Lin and Su, Hui}, title = {Trust in AutoML: Exploring Information Needs for Establishing Trust in Automated Machine Learning Systems}, year = {2020}, isbn = {9781450371186}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3377325.3377501}, doi = {10.1145/3377325.3377501}, booktitle = {Proceedings of the 25th International Conference on Intelligent User Interfaces}, pages = {297–307}, numpages = {11}, keywords = {AutoML, automated data science, automated machine learning, AutoDS, automated artificial intelligence, AutoAI, trust}, location = {Cagliari, Italy}, series = {IUI '20}, }