skip to main content

@tonyfast s notebooks

site navigation
notebook summary
title
asynchronous imports in python
description
lets say we have a python module with a top level await statement. this is valid in javascript and ipython, but not python due to the ambiguity of the top level await. we'll start digging into what it takes to have async imports in python
cells
15 total
7 code
state
executed in order
kernel
Python [conda env:root] *
language
python
name
conda-root-py
lines of code
38
outputs
2
table of contents
{"kernelspec": {"display_name": "Python [conda env:root] *", "language": "python", "name": "conda-root-py"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.10"}, "title": "asynchronous imports in python", "description": "lets say we have a python module with a top level await statement.\nthis is valid in javascript and ipython, but not python due to the ambiguity of the top level await.\nwe'll start digging into what it takes to have async imports in python"}
notebook toolbar
Activate
cell ordering

asynchronous imports in python

lets say we have a python module with a top level await statement. this is valid in javascript and ipython, but not python due to the ambiguity of the top level await. we'll start digging into what it takes to have async imports in python

https://gist.github.com/Rich-Harris/0b6f317657f5167663b493c722647221

dream is a our async function and catcher is a value we have access to 

    async def dream():
        await __import__("asyncio").sleep(1)
        return "dream on"
    print(catcher := await dream())

dream on

by default, importing modules with top level await is a fail because await can't be outside a function. 

    try: 
        with __import__("importnb").Notebook(): import __
    except SyntaxError as error: assert "'await' outside function" == error.args[0], error

    # boiler plate
    import importnb, ast; from IPython import get_ipython
    shell = get_ipython();

we're about to do some nasty nested async business when we are working interactively. below we ask IPython to prefer to the trio thread when we execute code cells this way we can own the asyncio event loop. 

    if "__file__" not in locals(): 
        __import__('nest_asyncio').apply()
        shell.loop_runner  =__import__("IPython").core.async_helpers._trio_runner

luckily there is a flag for top level awaits thanks to matthias and his hard work on IPython .

https://docs.python.org/3/library/ast.html#ast.PyCF_ALLOW_TOP_LEVEL_AWAIT

we'll make an importer that includes the PyCF_ALLOW_TOP_LEVEL_AWAIT flag and executes the module through an asynchronous version of eval 

    class ANotebook(importnb.Notebook):
        def exec_module(self, module):
            code = self.get_code(self.name)
            return __import__("asyncio").run(eval(code, vars(module)))
        
        def source_to_code(self, nodes, path, *, _optimize=-1):
            if not isinstance(nodes, ast.Module):
                nodes = self.parse(nodes)
            return compile(self.visit(nodes), path, "exec", optimize=_optimize, flags=ast.PyCF_ALLOW_TOP_LEVEL_AWAIT)

    with ANotebook(): import __11_12_async_import as anb
    assert anb.catcher == "dream on"

dream on

we haven't done anything fancy with asyncing the reading the and decoding of the source. we'll get there though. this is just opening the can of worms. the jokes should follow.