Tools

LEADER 03688cam a2200493 i 4500

001 on1506074729;

003 OCoLC;

005 20260529024817.0;

006 m o d ;

007 cr unu||||||||;

008 250310s2024 ncua obm 000 0 eng d;

035 a| (OCoLC)1506074729;

090 a| QA76.9.N38;

049 a| EREE;

100 1 a| Giramata, Suavis,e| author.?| UNAUTHORIZED;

245 10 a| Metamorphic testing prioritization for fairness evaluation in large language models /c| by Suavis Giramata.;

264 1 a| [Greenville, N.C.] :b| [East Carolina University],c| 2024.;

300 a| 1 online resource (77 pages) :b| illustrations (some color);

336 a| textb| txt2| rdacontent;

337 a| computerb| c2| rdamedia;

338 a| online resourceb| cr2| rdacarrier;

347 a| text fileb| application/pdfc| 528.06 KB2| rda;

538 a| System requirements: Adobe Reader.;

538 a| Mode of access: World Wide Web.;

502 b| M.S.c| East Carolina Universityd| 2024.;

500 a| Presented to the Faculty of the Department of Computer Science;

500 a| Advisor: Madhusudan Srinivasan;

500 a| Title from PDF t.p. (viewed March 6, 2026).;

520 3 a| Large language models (LLMs) face challenges in detecting fairness related faults due to the oracle problem, where it is difficult to define correct outputs for all scenarios. This research applies metamorphic testing (MT) as a solution, focusing on the prioritization of metamorphic relations (MRs) based on their diversity scores to maximize fault detection efficiency. The study hypothesizes that MRs with high diversity scores, indicating significant differences between source and follow-up test cases, are more likely to reveal faults related to fairness and bias in LLMs. To test this, several diversity metrics, including cosine similarity, sentiment analysis, and named entity recognition, are used to quantify differences between test cases. The proposed approach is evaluated on two popular LLMs, GPT and LLaMA, comparing it against random, fault-based, and distance-based MR ordering strategies. The results indicate that prioritizing high-diversity MRs significantly improves fault detection speed and effectiveness, particularly for identifying biases across sensitive attributes. Specifically, our proposed Total Diversity Score-based approach shows a 91.6% improvement in fault detection over the Random-Based approach at the first MR, gradually reducing to 21.05% by the fifth MR. Additionally, compared to the Distance-Based method, our approach achieves an initial 130% improvement in fault detection rate, decreasing to 1.61% by the ninth MR before performance levels stabilize. Notably, our approach also performs closely to the Fault-Based prioritization, offering a balanced and effective method for uncovering faults efficiently.;

504 a| Includes bibliographical references.;

650 0 a| Natural language processing (Computer science)=| ^A235570;

650 0 a| Machine learningx| Evaluation.=| ^A145211;

653 a| Metamorphic Testing;

653 a| Prioritization;

653 a| Large Language Models;

700 1 a| Srinivasan, Madhusudan,e| degree supervisor.?| UNAUTHORIZED;

710 2 a| East Carolina University.b| Department of Computer Science.?| UNAUTHORIZED;

856 40 z| Access via ScholarShipu| http://hdl.handle.net/10342/13872;

994 a| C0b| ERE;

596 a| 1 4;

998 a| 6560822;

999 a| CLICK ON WEB ADDRESSw| ASISc| 1i| 6560822-1001l| JNETm| JOYNERr| Ys| Yt| JNE3ETDu| 3/10/2025x| ETDz| JERESOURCE;

999 a| CLICK ON WEB ADDRESSw| ASISc| 1i| 6560822-2001l| HSLELECm| HSLr| Ys| Yt| HEETDu| 3/10/2025x| ETDz| HERESOURCE;

Metamorphic testing prioritization for fairness evaluation in large language models / by Suavis Giramata.

Author/creator	Giramata, Suavis author.
Other author	Srinivasan, Madhusudan, degree supervisor.
Other author	East Carolina University. Department of Computer Science.
Format	Theses and dissertations
Publication	[Greenville, N.C.] : [East Carolina University], 2024.
Description	1 online resource (77 pages) : illustrations (some color)
Supplemental Content	Access via ScholarShip
Subjects	Natural language processing (Computer science) Machine learning--Evaluation.

Summary	Large language models (LLMs) face challenges in detecting fairness related faults due to the oracle problem, where it is difficult to define correct outputs for all scenarios. This research applies metamorphic testing (MT) as a solution, focusing on the prioritization of metamorphic relations (MRs) based on their diversity scores to maximize fault detection efficiency. The study hypothesizes that MRs with high diversity scores, indicating significant differences between source and follow-up test cases, are more likely to reveal faults related to fairness and bias in LLMs. To test this, several diversity metrics, including cosine similarity, sentiment analysis, and named entity recognition, are used to quantify differences between test cases. The proposed approach is evaluated on two popular LLMs, GPT and LLaMA, comparing it against random, fault-based, and distance-based MR ordering strategies. The results indicate that prioritizing high-diversity MRs significantly improves fault detection speed and effectiveness, particularly for identifying biases across sensitive attributes. Specifically, our proposed Total Diversity Score-based approach shows a 91.6% improvement in fault detection over the Random-Based approach at the first MR, gradually reducing to 21.05% by the fifth MR. Additionally, compared to the Distance-Based method, our approach achieves an initial 130% improvement in fault detection rate, decreasing to 1.61% by the ninth MR before performance levels stabilize. Notably, our approach also performs closely to the Fault-Based prioritization, offering a balanced and effective method for uncovering faults efficiently.
General note	Presented to the Faculty of the Department of Computer Science
General note	Advisor: Madhusudan Srinivasan
General note	Title from PDF t.p. (viewed March 6, 2026).
Dissertation note	M.S. East Carolina University 2024.
Bibliography note	Includes bibliographical references.
Technical details	System requirements: Adobe Reader.
Technical details	Mode of access: World Wide Web.

Availability

Library	Location	Call Number	Status	Item Actions
	Electronic Resources	Access Content Online	✔ Available

Metamorphic testing prioritization for fairness evaluation in large language models / by Suavis Giramata.

Click here for more information about this title

Availability