Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 13 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
[![PyPI version](https://badge.fury.io/py/transfunctions.svg)](https://badge.fury.io/py/transfunctions)
[![Checked with mypy](http://www.mypy-lang.org/static/mypy_badge.svg)](http://mypy-lang.org/)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/mutating/transfunctions)

</details>

Expand Down Expand Up @@ -79,7 +80,7 @@ You can also quickly try out this and other packages without having to install u

## The problem

Since the `asyncio` module appeared in Python more than 10 years ago, many well-known libraries have gained asynchronous counterparts. A lot of the code in the Python ecosystem has been [duplicated](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself), and you probably know many such examples.
Since the `asyncio` module appeared in Python more than 10 years ago, many well-known libraries have developed asynchronous counterparts. A lot of the code in the Python ecosystem has been [duplicated](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself), and you probably know many such examples.

The reason for this problem is that the Python community has chosen a syntax-based approach to asynchrony. There are new keywords in the language, such as `async` and `await`. Their use makes the code so-called "[multicolored](https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/)": functions become “red” or “blue”, and depending on the color, the rules for calling them are different. You can only call blue functions from red ones, but not vice versa.

Expand All @@ -100,7 +101,7 @@ def template():
print('something')
```

Executing this code will actually return to us not a function, but a special object that can *produce* functions:
Executing this code will actually give us not a function, but a special object that can *produce* functions:

```python
print(template)
Expand All @@ -125,7 +126,7 @@ run(async_function())
#> something
```

That's more interesting. In fact, we transferred all the contents from the original function to the generated async function. The content itself has not changed in any way, that is, we got a function that would look something like this:
That's more interesting. In fact, all the contents from the original function are included in the generated async function. The content itself has not changed in any way, that is, we got a function that would look something like this:

```python
async def template():
Expand All @@ -147,7 +148,7 @@ def template():
yield
```

The `get_usual_function` method will return a function that will contain a common part (the first `print`) and a part highlighted using the context manager as related to ordinary functions. It will look something like this:
The `get_usual_function` method returns a function that contains a common part (the first `print`) and a part highlighted using the context manager as related to ordinary functions. It will look something like this:

```python
def template():
Expand Down Expand Up @@ -213,7 +214,7 @@ from transfunctions import (
)
```

Make sure that the generated functions do not include keywords that are not related to this type of function. For example, you cannot generate a regular function using the `get_usual_function` method from such a template:
Make sure that the generated functions do not include keywords that are not related to this type of function. For example, you cannot generate a regular function from such a template using the `get_usual_function` method:

```python
from asyncio import sleep
Expand All @@ -223,12 +224,12 @@ def template():
await_it(sleep(5))
```

Regular or generator functions cannot use the `await` keyword, so you will get an exception when you try to generate such a function. The same applies to the `yield` and `yield from` keywords. You cannot use them outside of code blocks that relate *only* to generator functions. Please note that not in all such cases, the `transfunctions` library will offer you an informative exception. Here you'd better rely on your own knowledge of `Python` syntax. However, even if such an exception is provided, it will only be raised when trying to generate a function of the type in which this syntax is inappropriate. At the template definition stage, you won't get an exception telling you that something went wrong, because the code generation here is lazy and the code is not analyzed for correctness in any way before you request it.
Regular or generator functions cannot use the `await` keyword, so you will get an exception when you try to generate such a function. The same applies to the `yield` and `yield from` keywords. You cannot use them outside of code blocks that relate *only* to generator functions. Please note that not in all such cases, the `transfunctions` library will offer you an informative exception. You'll need to rely on your knowledge of Python syntax in these cases. However, even if such an exception is provided, it will only be raised when trying to generate a function of the type in which this syntax is inappropriate. At the template definition stage, you won't get an exception telling you that something went wrong, because the code generation here is lazy and the code is not analyzed for correctness in any way before you request it.


## Superfunctions

Superfunctions are the most powerful feature of the library. They allow you to completely "put under the hood" all the machinery for selecting the desired type of function based on the template function. The selection is completely automatic.
Superfunctions are the most powerful feature of the library. They completely hide the machinery for selecting the desired type of function based on the template function. The selection is completely automatic.

Let's take a look at the sample code:

Expand All @@ -253,7 +254,7 @@ def my_superfunction():
yield
```

With the `@superfunction` decorator, you no longer need to call special methods for code generation. You can use the resulting function right away, and it will behave differently depending on how you use it.
With the `@superfunction` decorator, the code generation happens automatically. You can use the resulting function right away, and it will behave differently depending on how you use it.

If you use it as a regular function, a regular function will be created "under the hood" based on the template and then called:

Expand Down Expand Up @@ -282,7 +283,7 @@ list(my_superfunction())
#> so, it's a generator function!
```

How does it work? In fact, `my_superfunction` returns some kind of intermediate object that can behave as a coroutine, a generator, or a regular callable. Depending on how it is handled, it lazily code-generates the desired version of the function from a given template and uses it.
How does it work? In fact, calling `my_superfunction` gives you an object that can behave as a coroutine, a generator, or a regular callable. Depending on how it is handled, it lazily code-generates the desired version of the function from a given template and uses it.

By default, a superfunction is called as a regular function using tilde syntax, but there is another mode. To enable it, use the appropriate flag in the decorator:

Expand All @@ -300,7 +301,7 @@ my_superfunction()
However, it comes with trade-offs. The fact is that this mode uses a special trick with a reference counter, a special mechanism inside the interpreter that cleans up memory. When there is no reference to an object, the interpreter deletes it, and you can link your callback to this process. It is inside such a callback that the contents of your function are actually executed. This imposes some restrictions on you:

- You cannot use the return values from this function in any way. If you try to save the result of a function call to a variable, the reference counter to the returned object will not drop to zero while this variable exists, and accordingly the function will not actually be called.
- Exceptions will not work normally inside this function. Rather, they can be picked up and intercepted in [`sys.unraisablehook`](https://docs.python.org/3/library/sys.html#sys.unraisablehook), but they will not go up the stack above this function. This is due to a feature of CPython: exceptions that occur inside callbacks for finalizing objects are completely escaped.
- Exceptions will not work normally inside this function. Rather, they can be picked up and intercepted in [`sys.unraisablehook`](https://docs.python.org/3/library/sys.html#sys.unraisablehook), but they will not go up the stack above this function. This is a CPython limitation: exceptions raised inside object finalizer callbacks can't propagate normally.

This mode is well suited for functions such as logging or sending statistics from your code: simple functions from which no exceptions or return values are expected. In all other cases, I recommend using the tilde syntax.

Expand All @@ -312,9 +313,9 @@ Typing is the most difficult problem we faced when developing this library. In m
There are 2 main difficulties in developing typing here:

- Code generation creates code at runtime that is not in the source files of your project. Whereas most type analyzers look at your code statically, at what is actually present in your files.
- We mix several types of syntax in a single template function, but the static analyzer does not know that this is a template and part of the code will be deleted from here. In its opinion, this is the final function that will continue to be used in your project.
- We put different syntax blocks in a single template function, but the static analyzer does not know that this is a template and part of the code will be deleted from here. In its opinion, this is the final function that will continue to be used in your project.

As you can see, typing in Python is not well suited for metaprogramming. However, in this project, almost all the problems with typing turned out to be solved in one way or another. The main reason why this is so is that we mostly *remove* code from functions, but hardly *add* it there during code generation. In other words, we almost never encounter the problem of how to type the *added* code. This makes the solution to most typing problems accessible. However, we were not able to completely hide all the typing problems under the hood, but you should still be aware of some of them if you use `mypy` or another analyzer.
As you can see, typing in Python is not well suited for metaprogramming. However, in this project, almost all the problems with typing turned out to be solved in one way or another. The main reason is that we mostly *remove* code from functions, but hardly *add* it there during code generation. In other words, we almost never encounter the problem of how to type the *added* code. This makes the solution to most typing problems accessible. However, we were not able to completely hide all the typing problems under the hood, but you should still be aware of some of them if you use `mypy` or another analyzer.

If you use the keyword `yield from`, you need to call the function `yield_from_it` instead:

Expand Down
7 changes: 4 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,14 @@ build-backend = "setuptools.build_meta"

[project]
name = "transfunctions"
version = "0.0.12"
version = "0.0.13"
authors = [{ name = "Evgeniy Blinov", email = "zheni-b@yandex.ru" }]
description = 'Say NO to Python fragmentation on sync and async'
readme = "README.md"
requires-python = ">=3.8"
dependencies = [
'displayhooks>=0.0.6',
'getsources>=0.0.3',
'displayhooks>=0.0.7',
'getsources>=0.0.4',
'typing_extensions ; python_version <= "3.10"',
]
classifiers = [
Expand All @@ -36,6 +36,7 @@ classifiers = [
'Topic :: Software Development :: Libraries',
'Topic :: Software Development :: Code Generators',
'Framework :: AsyncIO',
'Typing :: Typed',
]
keywords = [
'async',
Expand Down
5 changes: 5 additions & 0 deletions tests/documentation/test_readme.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,11 @@


def test_quick_start():
"""
Ensure a decorated template generates usual, async, and generator callables that share common code and retain only their matching marker block.

Each generated callable is executed in the way required for its type, including awaiting the async function and iterating the generator, and stdout is checked for the common prefix plus the type-specific message.
"""
@transfunction
def template():
print('so, ', end='') # noqa: T201
Expand Down
Loading
Loading