customizing the IPython contextual help

an original feature of pidgy is the use jupyters contextual help to provide wysiwyg experience.

as author's write pidgy they reveal a preview of the output and explore variables in the workspace.

• when source is found show help
• when inspection is found show the found results
• when is not found show the markdown preview

the inspector uses jupyters rich display system. we can send html and markdown to the contextual help.

we create an ipython extension that modifies the IPython contextual help using rich displays. we are going to create a wsyiwyg for

for any of this to work need to enable the html pager shell.enable_html_pager

shell.enable_html_pager = True

do_inspect will be our new contextual help completer. it provides completion when:

• there is no content to inspect
• there was no result found
• you found the bangs

def do_inspect(self, cell, cursor_pos, detail_level=1, omit_sections=(), *, cache={}):
    post = post_bangs(cell, get_bangs(cell, cursor_pos))
    if post:
        data = _do_inspect("", 0, detail_level=0, omit_sections=())
        data["data"]= post
        data["found"] = True
    else:
        data = _do_inspect(cell, cursor_pos, detail_level=0, omit_sections=())
        if not data["found"]:
            data.update(inspect_weave(cell, cursor_pos))
    return data

def inspect_weave(code, cursor_pos, cache={}):
    data = dict(found=True)
    if not code.strip(): 
        data["data"] = {"text/markdown": help}
    else:
        tokens = cache.get(code)
        line, offset= lineno_at_cursor(code, cursor_pos)
        if tokens is None:               
            cache.clear()
            tokens = cache[code] = shell.tangle.parse(code)
        where = get_md_pointer(tokens, line, offset)
        data["data"] = {"text/markdown": F"""`{"/".join(where)}`\n\n{code}"""}
    return data

this is a hack! we'll hold the original inspect function and monkey patch it.

<bound method IPythonKernel.do_inspect of <ipykernel.ipkernel.IPythonKernel object at 0x7fd8eb2b0fa0>>

locals().setdefault("_do_inspect", shell.kernel.do_inspect)

if I := ("__file__" not in locals()): load_ipython_extension(shell)

line positions and tokenization

it seemed important to have line and column numbers that align exactly with the code mirror line numbers and the line/column number in the bottom inspector ribbon. for extra confidence when writing pidgy we include the markdown block token the cursor is within.

def lineno_at_cursor(cell, cursor_pos=0):
    offset = 0
    for i, line in enumerate(cell.splitlines(True)):
        next_offset = offset + len(line)
        if not line.endswith('\n'): next_offset += 1
        if next_offset > cursor_pos:  break
        offset = next_offset
    col = cursor_pos-offset
    return i, col

def get_md_pointer(tokens, line, offset):
    where = [F"L{line+1}:{offset+1}"]
    for node in tokens:
        if node.type == "root": continue
        if node.map:
            if line < node.map[0]: break
            elif node.map[0] <= line < node.map[1]: where.append(node.type)
    return where

the bangs

the bangs give extra interactivity. when inspecting code a group of exclamation !!! points with template the woven source, or with enough emphasism !!!!!! we'll actually run code. consider these easter eggs cause i want them.

def get_bangs(cell, cursor_pos):
    if cell[cursor_pos-1] == "!":
        l, r = cell[:cursor_pos], cell[cursor_pos:]
        return len(l) - len(l.rstrip("!")) + len(r) - len(r.strip("!"))
    return 0

def post_bangs(cell, bangs):
    if bangs>=6:
        result = shell.run_cell(cell, store_history=False, silent=True)
        error = result.error_before_exec or result.error_in_exec
        if error: return {"text/markdown": F"""```pycon\n{"".join(
                traceback.format_exception(type(error), error, error.__traceback__)
            )}\n```"""}
        else: shell.weave.update()
    if bangs >=3:
        result = shell.environment.from_string(cell, None, pidgy.environment.IPythonTemplate)
        return {"text/markdown": result.render()}

the opportunity in the misses

the contextual help renders on keypress when a result is found.

when we use the normal inspection. we find that there is a lot of time where the contextual help is not found. for example, when we process all the code inputs in this document the inspector finds information 36.0% of the time, meaning 64.0% opportunity!!

def measure(x): yield from (shell.kernel.do_inspect(x, i) for i in range(len(x)))
shell.kernel.do_inspect = _do_inspect
inspection = pandas.Series(In).apply(measure).apply(list).explode().dropna().apply(pandas.Series)
load_ipython_extension(shell)

a use is the blank screen is to provide a help interface for people to learn our new tool. with links to good docs.

help =\

pidgy is markdown/python literate programming language.

• indented code and fenced code are executed

documentation

• jinja2 documentation
• markdown documentation
• python documentation

including the current markdown token

sometimes when writing pidgy its possible to lose your you position. we include the line number and token the cursor is in

an accidental outcome is this hack adds contextual help to markdown cells. its good.