Skip to content

when markdown and python collide¤

a story about literacy written in the literate programming style of pidgy

notebooks commonly communicate with two languages: markdown & python. what happen when we dissolve the boundaries between markdown & python or language & code entirely.

    class STATE:
        NB = "__file__" not in globals() and __name__ == "__main__"
        SCRIPT = "__file__" in globals() and __name__ != "__main__"
        MAIN = "__file__" in globals() and __name__ == "__main__"
        LITE = __import__("sys").platform == "emscripten"
# execute this code in lite.
if STATE.LITE:
    try:
        %reload_ext pidgy
    except ModuleNotFoundError:
        import micropip
        await micropip.install("pidgy importnb qrcode midgy ipywidgets".split(), pre=True)
    import pidgy, pathlib, IPython
    shell = IPython.get_ipython()
    if STATE.NB:
        %reload_ext pidgy
        shell.displays_manager.template_cls = pidgy.weave.IPythonHtml
import pidgy, pathlib, IPython
shell = IPython.get_ipython()
if STATE.NB:
    %reload_ext pidgy
    shell.displays_manager.template_cls = pidgy.weave.IPythonHtml
<style>
img {
    height: 500px !important;
}
.jupyter-wrapper .jp-Cell-inputWrapper {
    display: none;
}
</style>
<script src="https://cdn.jsdelivr.net/npm/mermaid/dist/mermaid.min.js"></script>
<script>mermaid.initialize({startOnLoad:true, securityLevel: "loose"}, ".mermaid");</script>
## literate computing with literary machines using literate programming for mass computational literacy

literacy rings through the history of computer science.
in Annette Vee's [Understanding Computer Programming as a Literacy] we get
a high level view of literacy through history and in the our little microcosm of computer programming.

&gt; We compare mass ability to read and write software with mass literacy, and predict equally pervasive changes to society.

    history =\
```mermaid
gantt 
title Literacy in computer programming
dateFormat YYYY
axisFormat %Y

section moderninity
understanding media :mcluhan, 1964, 2022
mother of all demos :englebert, 1968, 2022
section post moderninity
literate programming :knuth, 1984, 2022
literary machines :nelson, 1987, 2022
www :tbl, 1989, 2022

section post web
computer programming for everybody :cp4e, 1999, 2022
ipython :fperez, 2001, 2022
markdown :gruber, 2004, 2022
ipython notebooks :ipynb, 2011, 2022
anaconda :ana, 2012, 2022
literate computing :lc, 2013, 2022
understanding computer programming as a literacy :vee, 2013, 2022
the birth &amp; death of javascript: js, 2014, 2022
big split :split, 2015, 2022
jupyterlab :lab, 2018, 2022
jupyterlite :lab, 2021, 2022

click vee href "https://licsjournal.org/index.php/LiCS/article/view/794/608"
click knuth href "http://www.literateprogramming.com/knuthweb.pdf"
click nelson href "https://monoskop.org/images/b/be/Nelson_Ted_Literary_Machines_c1987_chs_0-1.pdf"
click lc href "https://web.archive.org/web/20220510083647/http://blog.fperez.org/2013/04/literate-computing-and-computational.html"
click cp4e href "https://www.python.org/doc/essays/cp4e/"


```

    shell.displays_manager.template_cls = pidgy.weave.IPythonMarkdown

literate computing with literary machines using literate programming for mass computational literacy¤

literacy rings through the history of computer science. in Annette Vee's [Understanding Computer Programming as a Literacy] we get a high level view of literacy through history and in the our little microcosm of computer programming.

> We compare mass ability to read and write software with mass literacy, and predict equally pervasive changes to society.

history =\

mermaid gantt title Literacy in computer programming dateFormat YYYY axisFormat %Y section moderninity understanding media :mcluhan, 1964, 2022 mother of all demos :englebert, 1968, 2022 section post moderninity literate programming :knuth, 1984, 2022 literary machines :nelson, 1987, 2022 www :tbl, 1989, 2022 section post web computer programming for everybody :cp4e, 1999, 2022 ipython :fperez, 2001, 2022 markdown :gruber, 2004, 2022 ipython notebooks :ipynb, 2011, 2022 anaconda :ana, 2012, 2022 literate computing :lc, 2013, 2022 understanding computer programming as a literacy :vee, 2013, 2022 the birth &amp; death of javascript: js, 2014, 2022 big split :split, 2015, 2022 jupyterlab :lab, 2018, 2022 jupyterlite :lab, 2021, 2022 click vee href "https://licsjournal.org/index.php/LiCS/article/view/794/608" click knuth href "http://www.literateprogramming.com/knuthweb.pdf" click nelson href "https://monoskop.org/images/b/be/Nelson_Ted_Literary_Machines_c1987_chs_0-1.pdf" click lc href "https://web.archive.org/web/20220510083647/http://blog.fperez.org/2013/04/literate-computing-and-computational.html" click cp4e href "https://www.python.org/doc/essays/cp4e/"

shell.displays_manager.template_cls = pidgy.weave.IPythonMarkdown

donald knuth's diagram for literate programming¤

image.png

# tangle (code) and weave (display)

    knit =\
```mermaid
flowchart LR
web-- tangle ---pas
pas-- pascal ---rel
web-- weave ---tex
tex-- TEX ---dvi
```

tangle (code) and weave (display)¤

knit =\

mermaid flowchart LR web-- tangle ---pas pas-- pascal ---rel web-- weave ---tex tex-- TEX ---dvi

## `midgy/pidgy` family of input kernels in `IPython`

literate inputs with markdown and python as the document and programming languages

<div style="font-size: 2rem;">

    import midgy as tangle, pidgy as weave

</div>

we'll explore a little program in markdown inside a bigger notebook program. hold tight.

midgy/pidgy family of input kernels in IPython¤

literate inputs with markdown and python as the document and programming languages

import midgy as tangle, pidgy as weave

we'll explore a little program in markdown inside a bigger notebook program. hold tight.

### how does `IPython` python?


* `shell.kernel.do_execute`
    * `shell.run_cell`
        * tangle
            * `shell.transform_cell`
            * `shell.compile.ast_parse`
            * `shell.transform_ast`
            * `shell.run_ast_nodes`        
        * weave
            * if `shell.ast_node_interactivity` 
            * `shell.display_formatter.format`

`IPython` is very hackable! that is what makes it fun along with all the development happening in python.

how does IPython python?¤

  • shell.kernel.do_execute
    • shell.run_cell
      • tangle
        • shell.transform_cell
        • shell.compile.ast_parse
        • shell.transform_ast
        • shell.run_ast_nodes
      • weave
        • if shell.ast_node_interactivity
        • shell.display_formatter.format

IPython is very hackable! that is what makes it fun along with all the development happening in python.

    %%file a-little-markdown-program.md
    this markdown file is a program!

            δϵατκ=\
    death to pseudo code, long live pseudo code

            if __name__ == "__main__":
                print(δϵατκ.upper())
Overwriting a-little-markdown-program.md

we can run this like a script. WTF!

!midgy a-little-markdown-program.md
    import midgy.loader
    with midgy.loader.Markdown() as loader:
        import a_little_markdown_program
    a_little_markdown_program, a_little_markdown_program.δϵατκ
(<module 'a-little-markdown-program' from '/home/tbone/Documents/tonyfast/tonyfast/xxii/oct/a-little-markdown-program.md'>,
 'death to pseudo code, long live pseudo code')
import midgy.loader
with midgy.loader.Markdown() as loader:
    import a_little_markdown_program
a_little_markdown_program, a_little_markdown_program.δϵατκ
`a_little_markdown_program` is a module like any other

`{{a_little_markdown_program}}`

🐇and we even generated a compiled version of the markdown source🎩

    if not STATE.LITE:
        assert next(
            (pathlib.Path(a_little_markdown_program.__file__).parent / "__pycache__").glob("a-little-markdown-program.*.pyc")
        ),\
verifies there is a compiled file.

a_little_markdown_program is a module like any other

<module '="" 'a-little-markdown-program'="" a-little-markdown-program.md'="" documents="" from="" home="" oct="" tbone="" tonyfast="" xxii="">

🐇and we even generated a compiled version of the markdown source🎩

if not STATE.LITE:
    assert next(
        (pathlib.Path(a_little_markdown_program.__file__).parent / "__pycache__").glob("a-little-markdown-program.*.pyc")
    ),\

verifies there is a compiled file.

## [literate computing widgets and `IPython` interactive displays](2022-10-21-pidgy-displays.ipynb)

literate computing widgets and IPython interactive displays¤

## reusing and importing notebooks

we can import this presentation.

        import midgy.loader
        with midgy.loader.Markdown(extensions=[".ipynb", ".md"]):
            import _022_10_21_markdown_future
        "__file__" not in locals() and display(
            _022_10_21_markdown_future,
            _022_10_21_markdown_future.a_little_markdown_program,
            _022_10_21_markdown_future.a_little_markdown_program.δϵατκ
        )
Overwriting a-little-markdown-program.md
gantt 
title Literacy in computer programming
dateFormat YYYY
axisFormat %Y

section moderninity
understanding media :mcluhan, 1964, 2022
mother of all demos :englebert, 1968, 2022
section post moderninity
literate programming :knuth, 1984, 2022
literary machines :nelson, 1987, 2022
www :tbl, 1989, 2022

section post web
computer programming for everybody :cp4e, 1999, 2022
ipython :fperez, 2001, 2022
markdown :gruber, 2004, 2022
ipython notebooks :ipynb, 2011, 2022
anaconda :ana, 2012, 2022
literate computing :lc, 2013, 2022
understanding computer programming as a literacy :vee, 2013, 2022
the birth &amp; death of javascript: js, 2014, 2022
big split :split, 2015, 2022
jupyterlab :lab, 2018, 2022
jupyterlite :lab, 2021, 2022

click vee href "https://licsjournal.org/index.php/LiCS/article/view/794/608"
click knuth href "http://www.literateprogramming.com/knuthweb.pdf"
click nelson href "https://monoskop.org/images/b/be/Nelson_Ted_Literary_Machines_c1987_chs_0-1.pdf"
click lc href "https://web.archive.org/web/20220510083647/http://blog.fperez.org/2013/04/literate-computing-and-computational.html"
click cp4e href "https://www.python.org/doc/essays/cp4e/"

flowchart LR
web-- tangle ---pas
pas-- pascal ---rel
web-- weave ---tex
tex-- TEX ---dvi
<module '2022-10-21-markdown-future' from '/home/tbone/Documents/tonyfast/tonyfast/xxii/oct/2022-10-21-markdown-future.ipynb'>
<module 'a-little-markdown-program' from '/home/tbone/Documents/tonyfast/tonyfast/xxii/oct/a-little-markdown-program.md'>
'death to pseudo code, long live pseudo code'

reusing and importing notebooks¤

we can import this presentation.

    import midgy.loader
    with midgy.loader.Markdown(extensions=[".ipynb", ".md"]):
        import _022_10_21_markdown_future
    "__file__" not in locals() and display(
        _022_10_21_markdown_future,
        _022_10_21_markdown_future.a_little_markdown_program,
        _022_10_21_markdown_future.a_little_markdown_program.δϵατκ
    )
### what about reusing that diagram


    display(
        IPython.display.Markdown(_022_10_21_markdown_future.history),
        IPython.display.Markdown(_022_10_21_markdown_future.knit)
    )
gantt 
title Literacy in computer programming
dateFormat YYYY
axisFormat %Y

section moderninity
understanding media :mcluhan, 1964, 2022
mother of all demos :englebert, 1968, 2022
section post moderninity
literate programming :knuth, 1984, 2022
literary machines :nelson, 1987, 2022
www :tbl, 1989, 2022

section post web
computer programming for everybody :cp4e, 1999, 2022
ipython :fperez, 2001, 2022
markdown :gruber, 2004, 2022
ipython notebooks :ipynb, 2011, 2022
anaconda :ana, 2012, 2022
literate computing :lc, 2013, 2022
understanding computer programming as a literacy :vee, 2013, 2022
the birth &amp; death of javascript: js, 2014, 2022
big split :split, 2015, 2022
jupyterlab :lab, 2018, 2022
jupyterlite :lab, 2021, 2022

click vee href "https://licsjournal.org/index.php/LiCS/article/view/794/608"
click knuth href "http://www.literateprogramming.com/knuthweb.pdf"
click nelson href "https://monoskop.org/images/b/be/Nelson_Ted_Literary_Machines_c1987_chs_0-1.pdf"
click lc href "https://web.archive.org/web/20220510083647/http://blog.fperez.org/2013/04/literate-computing-and-computational.html"
click cp4e href "https://www.python.org/doc/essays/cp4e/"

flowchart LR
web-- tangle ---pas
pas-- pascal ---rel
web-- weave ---tex
tex-- TEX ---dvi

what about reusing that diagram¤

display(
    IPython.display.Markdown(_022_10_21_markdown_future.history),
    IPython.display.Markdown(_022_10_21_markdown_future.knit)
)
i'm not saying importing markdown or notebooks is a good idea. 
coffeescript died didn't it?
sometimes it is too soon.
it can be done and some folks might thrive working that way.

i'm not saying importing markdown or notebooks is a good idea. coffeescript died didn't it? sometimes it is too soon. it can be done and some folks might thrive working that way.

### an inclusive future for code

![](https://pbs.twimg.com/media/FfmxiqlaAAIzTvT?format=jpg&amp;name=large)

in this demo, we used markdown and notebooks for source.

literate programs explicitly define a document and program language.
notebooks are literate programs.
there are times when when code and narrative are inseparable.

```future
docker up my-paper.pdf # can a paper contain containers?
importpdf my-paper --input-file my-data.csv # could you run a pdf with your data?

with importdocx():
    import my_word_document # https://nbviewer.org/github/deathbeds/deathbeds.github.io/blob/master/deathbeds/2019-03-08-say-word.ipynb
```

an inclusive future for code¤

in this demo, we used markdown and notebooks for source.

literate programs explicitly define a document and program language. notebooks are literate programs. there are times when when code and narrative are inseparable.

docker up my-paper.pdf # can a paper contain containers?
importpdf my-paper --input-file my-data.csv # could you run a pdf with your data?

with importdocx():
    import my_word_document # https://nbviewer.org/github/deathbeds/deathbeds.github.io/blob/master/deathbeds/2019-03-08-say-word.ipynb
## the power of markdown

often notebooks or markdown are means not an ends.

* how many agendas have we made in `hackmd`?
* how many `README`s guided our way?
* how much good code is lost in notebooks?

the power of markdown¤

often notebooks or markdown are means not an ends.

  • how many agendas have we made in hackmd?
  • how many READMEs guided our way?
  • how much good code is lost in notebooks?

    def qr(url, size=8):
        q = __import__("qrcode").QRCode(version=1, error_correction=1,
                          box_size=size, border=1, mask_pattern=None,)
        q.add_data(url), q.make()
        return q.make_image()
## what if the code was just the start something?

y'all, like all y'all can modify this document in jupyterlite on github pages.

<style>
[data-mime-type="image/png"] img {
    max-width: 200px; max-height: 200px;
}
</style>

    qr(        
https://tonyfast.github.io/tonyfast/run/lab/index.html?path=xxii/oct/2022-10-21-markdown-future.ipynb&amp;room=deathbeds-more-like-deft-breads

    )
No description has been provided for this image

what if the code was just the start something?¤

y'all, like all y'all can modify this document in jupyterlite on github pages.

qr(

https://tonyfast.github.io/tonyfast/run/lab/index.html?path=xxii/oct/2022-10-21-markdown-future.ipynb&room=deathbeds-more-like-deft-breads

)