Building on the version control practices described in Chapter 4 and Chapter 5, each iteration of testing and refining should be captured in your repository. Commit or branch your changes as you address issues, add new features, or revert problematic updates. This ensures that when you iterate until satisfied, you also preserve a complete record of every revision and can quickly return to any previous state of the code.
The process of refining code with an LLM is rarely a one-time event. More often, it involves a cycle of review and refinement that continues until the generated script meets all the necessary requirements and performs as expected. This chapter focuses on the Iterate Until Satisfied step, emphasizing the importance of continuous testing and assessment throughout this journey 56.
After each round of edits or feature additions in the previous step, it is crucial to test and assess the code 61. This might involve actually running the script on a sample of your data to see if it produces the expected outputs and if any errors occur. Alternatively, for smaller changes or if running the full script is time-consuming, you might mentally walk through the modified code with some sample data to anticipate its behavior. The key is to verify whether the implemented changes work as intended and if any new, unintended issues have been introduced.
Based on the results of your testing, you will likely need to repeat the review and refine cycle 66. This means going back to Chapter 4 to review the current state of the code and then to Chapter 5 to make further refinements based on any issues or missing features you identified during testing. This iterative process of reviewing and refining with the LLM is a normal and expected part of working with these tools. Think of it as a back-and-forth collaboration where you provide feedback, and the AI assistant makes adjustments accordingly.
As you get closer to a satisfactory solution, you will often find yourself fine-tuning smaller details 71. This could involve addressing edge cases that you hadn't initially considered, making small performance tweaks to improve the efficiency of the code, or enhancing the formatting of the output to make it more presentable. For example, in our BMI harmonization project, you might realize that you need to handle cases where two patient encounters have the same measurement date and decide on a rule for selecting the representative BMI in such scenarios. Or, you might identify a slow loop in the code and prompt the LLM to suggest a more efficient, vectorized approach.
Periodic red teaming during the iterative process can also be valuable. As your code evolves and becomes more complex, applying the red teaming techniques from Chapter 4 at key iterations can help identify new edge cases or methodological concerns that emerged during development. Consider red teaming your code after major feature additions or when preparing for final validation.
Throughout this iterative process, it is important to know when to stop 76. Once you are satisfied that the code is correct, meaning it runs without errors and produces the expected outputs, and that it is reasonably efficient and readable, you can conclude the code generation phase. At this point, you can move on to the subsequent steps of refactoring and documenting your final solution. Recognizing when the code is "good enough" is a key skill that helps to prevent over-engineering and ensures that you are using your time effectively.
In essence, this iterative journey of testing, reviewing, and refining is central to achieving a successful outcome when working with LLMs for coding tasks 81. It allows for continuous improvement and ensures that the final code not only addresses your research question but also meets the desired standards of quality and functionality. This process often mirrors the experience of collaborating with a human assistant, where ongoing feedback and adjustments are essential for reaching the best possible results.