VSCode extension to help developers write complete and unambiguous docstrings for their Python functions. Install it now from the Visual Studio Code Marketplace.

You need Python 3 installed on your system to use GuardRails.

Note: GuardRails is fully tested only on Windows and currently provides experimental support for Linux.

$ python -m pip install hypothesis hypothesis[cli] hypothesis[ghostwriter] black
$ python -m pip install git+https://github.com/mrigankpawagi/mutpy.git#egg=mutpy
$ python -m pip install func-timeout

def first_nonzero(nums: list[float]) -> float:
    """ Return the first non-zero value in nums.
    
    >>> first_nonzero([0.0 , 3.7 , 0.0])
    3.7
    """

This is an example of an incomplete function definition in Python. It has a purpose statement (called the docstring) which contains a functional example (called a doctest). The purpose statement is ambiguous because it does not specify the intended behavior on lists with no non-zero elements.

We propose a novel heuristic that suggests such inputs using Large Language Models (LLMs). Using these suggestions, the programmer may choose to clarify the purpose statement (e.g., by providing a functional example that specifies the intended behavior on such an input). Check out our paper for more details on our heuristic.

To assess the quality of inputs suggested by our heuristic, and to facilitate future research, we create an open dataset of purpose statements with known ambiguities. For each function, we provide four variants of prompts.

• S (Signature)
• SP (S + Purpose statement)
• SP1 (SP + 1 functional example)
• SPx (SP + More functional examples)

Our dataset is publicly available here. We use this dataset to compare GuardRails with test cases generated by GitHub Copilot Chat. More details on this are available in our paper.