New Approach in Self Checking Software Patches
Researchers at IMDEA Software introduce FIXCHECK, enhancing patch fix analysis with static analysis, randomised testing, and large language models.
IMDEA Software (Madrid Institute for Advanced Studies in Software Development Technologies) researchers Facundo Molina, Juan Manuel Copia, and Alessandra Gorla have introduced FIXCHECK, a pioneering method to enhance patch fix analysis. This innovative approach, detailed in their paper “Improving Patch Correctness Analysis via Random Testing and Large Language Models,” was recently presented at the International Conference on Software Testing, Verification, and Validation (ICST 2024).
Generating effective patches for software defects is a critical aspect of software maintenance. Typically, defects are reported through test cases that highlight undesirable software behaviours. Developers create patches to address these defects, which must be validated to ensure that the defect is resolved without introducing new issues. However, this process often results in “bad fixes” or incorrect patches that fail to adequately address the underlying problems or create additional bugs.
The recent unprecedented IT outage caused by a CrowdStrike patch has underscored the critical reliance on software systems and the potential repercussions of software failures. This global event disrupted multiple sectors, grounding flights, halting production lines, and affecting healthcare services worldwide .
Automatic program repair (APR) tools, which automatically generate patches for buggy programs, have exposed numerous incorrect patches that do not effectively resolve the reported bugs. The IMDEA researchers developed FIXCHECK to tackle this issue by combining static analysis, random testing, and large language models (LLMs) to automatically generate tests for detecting bugs in potentially incorrect patches.
FIXCHECK’s two-step process involves generating random tests to create a large set of test cases, and then using LLMs to derive meaningful assertions for each test case. The tool includes a selection and prioritization mechanism that executes these test cases on the patched program, discarding or ranking them based on their likelihood of revealing bugs.
Facundo Molina, a postdoctoral researcher at IMDEA Software, states, “The effectiveness of FIXCHECK in generating test cases that reveal bugs in incorrect patches was evaluated on 160 patches, including both developer-created patches and patches generated by RPA tools.” The results show that FIXCHECK can effectively generate bug detection tests for 62% of incorrect developer-written patches, providing a robust solution for automating test generation and detecting faults during software maintenance.
Had tools like FIXCHECK been widely adopted, some of the cascading failures caused by the recent CrowdStrike patche problem might have been mitigated. FIXCHECK’s ability to detect bugs in patches with high confidence could reduce the occurrence of bad fixes, potentially preventing such widespread disruptions.
For more detailed information, you can access the full paper “Improving Patch Correctness Analysis via Random Testing and Large Language Models” presented at ICST 2024.
