Support additional contexts for ProgramSynthesis

[datvo06]

In some component-based synthesis settings, it is useful to provide LLMs with functions that the resulting program can use.
While currently the tools are provided via tools parameters in the Template, we lack support for the resulting programs to import these modules automatically.
Example usage:

Template.define(ctxs=[functionA, functionB])
def generate_new_function() -> Callable[[a,b],c]:
       """generate this function that do x,y,z"""

Can reuse the type importing logic in #422

[datvo06]

Maybe another way to do this would be putting functionA and functionB as parameters and have a Handler that automatically converts them to strings and puts them into context?

@Template.define
def generate_new_function(functionA: Callable, functionB: Callable) -> Callable[[a,b], c]:
      """generate a function that do x, y, z with {functionA}, {functionB}"""

But then we would have to have extra mechanisms to infer that the modules containing functionA and functionB need to be imported when this is a Callable.

[datvo06]

Discussed with @jfeser , we're going with the 2nd approach.

[eb8680]

Isn't this behavior implicit in the lexical scope semantics of Template that we haven't yet implemented? I don't think there's anything specific to synthesis here.

[datvo06]

Hi, I see. I think the difference is how independent Callable needs to be:

• Whether it would be expected to operate within some handler's scope: with handler ({op: implementation}) or
• As a normal function.
  In the latter case, we would need to do module importing/symbol bindings to provide the synthesized function context to work with. Is that right?

[kiranandcode]

Sorry, just to confirm I'm understanding the discussion correctly, the question is when the context for imports should be decided for the generated callables?

# file: library.py
import requests

@Template.define
def build_movie_generate_function(database: str) -> Callable[[str], MovieSummary]:
    ...

# file: client.py
from library import build_movie_generate_function
import httpx

with handler(LiteLLMProvider()), handler(ProgramSynthesis()):
   imdb_reviews = build_movie_generate_function('imdb')

print(imdb_reviews('die hard'))

and the question is whether the imdb_reviews function should have access to the requests library, or the httpx library?

if I'm understanding that correctly, then I think it would make sense for the former, not the latter. One approach could be at the time of invocation @Template.define collects all symbols in its lexical context and stores that in the template, so when it is called to generate a function it can use that stored context. I think that semantics would be the most consistent with what a user would expect if the Template were just another function.

[eb8680]

Yes, that's right. I agree that having Template.define collect symbols in its lexical context would lead to behavior consistent with ordinary functions, which is why I am advocating for that design.

[eb8680]

A lot of the work needed to support this was done in #434, which added a __context__ field to Template that contains a view of the lexical context of its definition. Resolving this issue is now just a matter of providing Template.__context__ as the environment to compile()/exec() during code generation, and perhaps injecting function or module source code into the LLM context as appropriate.

[kiranandcode]

This is done via #519