nbdev v2 overview: Git-friendly Jupyter Notebooks

There are numerous methods to go about programming. Probably the most productive paradigms is interactive: You employ a REPL (read-eval-print loop) to jot down and check your code as you code, after which copy the examined code right into a file.

The REPL technique, which originated in LISP growth environments, is well-suited to Python programming, as Python has all the time had good interactive growth instruments. The disadvantage of this fashion of programming is that after you’ve written the code you must individually pull out the assessments and write the documentation, save all that to a repository, do your packaging, and publish your package deal and documentation.

Donald Knuth’s literate programming paradigm prescribes writing the documentation and code in the identical doc, with the documentation geared toward people interspersed with the code meant for the pc. Literate programming has been used extensively for scientific programming and knowledge science, typically utilizing pocket book environments, resembling Jupyter Notebooks, Jupyter Lab, Visual Studio Code, and PyCharm. One problem with notebooks is that they generally don’t play effectively with repositories as a result of they save an excessive amount of data, together with metadata that doesn’t matter to anybody. That creates an issue when there are merge conflicts, as notebooks are cell-oriented and supply code repositories resembling Git are line-oriented.

Jeremy Howard and Hamel Husain of fast.ai, together with about two dozen minor contributors, have give you a set of command-line utilities that not solely enable Jupyter Notebooks to play effectively with Git, but in addition allow a extremely productive interactive literate programming fashion. Along with producing right Python code shortly, you may produce documentation and assessments on the identical time, reserve it all to Git with out worry of corruption from merge conflicts, and publish to PyPI and Conda with just a few instructions. Whereas there’s a studying curve for these utilities, that funding pays dividends, as you will be executed along with your growth undertaking in in regards to the time it will usually take to easily write the code.

As you may see within the diagram beneath, nbdev works with Jupyter Notebooks, GitHub, Quarto, Anaconda, and PyPI. To summarize what every bit of this technique does:

• You possibly can generate documentation utilizing Quarto and host it on GitHub Pages. The docs assist LaTeX, are searchable, and are mechanically hyperlinked.
• You possibly can publish packages to PyPI and Conda in addition to instruments to simplify package deal releases. Python greatest practices are mechanically adopted, for instance, solely exported objects are included in __all__.
• There’s two-way sync between notebooks and plaintext supply code, permitting you to make use of your IDE for code navigation or fast edits.
• Exams written as abnormal pocket book cells are run in parallel with a single command.
• There’s steady integration with GitHub Actions that run your assessments and rebuild your docs.
• Git-friendly notebooks with Jupyter/Git hooks that clear undesirable metadata and render merge conflicts in a human-readable format.

IDG

nbdev set up

nbdev works on macOS, Linux, and most Unix-style working programs. It requires a latest model of Python 3; I used Python 3.9.6 on macOS Ventura, working on an M1 MacBook Professional. nbdev works on Home windows below WSL (Home windows Subsystem for Linux), however not below cmd or PowerShell. You possibly can set up nbdev with pip or Conda. I used pip:

pip set up nbdev

That put in 29 command-line utilities, which you’ll record utilizing nbdev_help:

% nbdev_help
nbdev_bump_version              Increment model in settings.ini by one
nbdev_changelog                 Create a CHANGELOG.md file from closed and labeled GitHub points
nbdev_clean                     Clear all notebooks in `fname` to keep away from merge conflicts
nbdev_conda                     Create a `meta.yaml` file able to be constructed right into a package deal, and optionally construct and add it
nbdev_create_config             Create a config file.
nbdev_docs                      Create Quarto docs and README.md
nbdev_export                    Export notebooks in `path` to Python modules
nbdev_filter                    A pocket book filter for Quarto
nbdev_fix                       Create working pocket book from conflicted pocket book `nbname`
nbdev_help                      Present assist for all console scripts
nbdev_install                   Set up Quarto and the present library
nbdev_install_hooks             Set up Jupyter and git hooks to mechanically clear, belief, and repair merge conflicts in notebooks
nbdev_install_quarto            Set up newest Quarto on macOS or Linux, prints directions for Home windows
nbdev_merge                     Git merge driver for notebooks
nbdev_migrate                   Convert all markdown and pocket book recordsdata in `path` from v1 to v2
nbdev_new                       Create an nbdev undertaking.
nbdev_prepare                   Export, check, and clear notebooks, and render README if wanted
nbdev_preview                   Preview docs domestically
nbdev_proc_nbs                  Course of notebooks in `path` for docs rendering
nbdev_pypi                      Create and add Python package deal to PyPI
nbdev_readme                    None
nbdev_release_both              Launch each conda and PyPI packages
nbdev_release_gh                Calls `nbdev_changelog`, enables you to edit the end result, then pushes to git and calls `nbdev_release_git`
nbdev_release_git               Tag and create a launch in GitHub for the present model
nbdev_sidebar                   Create sidebar.yml
nbdev_test                      Take a look at in parallel notebooks matching `path`, passing alongside `flags`
nbdev_trust                     Belief notebooks matching `fname`
nbdev_update                    Propagate change in modules matching `fname` to notebooks that created them

The nbdev builders counsel both watching this 90-minute video or going by this roughly one-hour written walkthrough. I did each, and in addition learn by extra of the documentation and a few of the source code. I realized completely different materials from every, so I’d counsel watching the video first after which doing the walkthrough. For me, the video gave me a transparent sufficient concept of the package deal’s utility to inspire me to undergo the tutorial.

Start the nbdev walkthrough

The tutorial begins by having you put in Jupyter Pocket book:

pip set up pocket book

After which launching Jupyter:

jupyter pocket book

The set up continues within the pocket book, first by creating a brand new terminal after which utilizing the terminal to put in nbdev. You possibly can skip that set up if you happen to already did it in a shell, like I did.

Then you should utilize nbdev to put in Quarto:

nbdev_install_quarto

That requires root entry, so that you’ll must enter your password. You possibly can learn the Quarto source code or docs to confirm that it’s secure.

At this level you might want to browse to GitHub and create an empty repository (repo). I adopted the tutorial and referred to as mine nbdev_hello_world, and added a reasonably generic description. Create the repo. Seek the advice of the instructions if you happen to want them. Then clone the repo to your native machine. The directions counsel utilizing the Git command line in your machine, however I occur to love utilizing GitHub Desktop, which additionally labored advantageous.

In both case, cd into your repo in your terminal. It doesn’t matter whether or not you employ a terminal in your desktop or in your pocket book. Now run nbdev_new, which can create a bunch of recordsdata in your repo. Then commit and push your additions to GitHub:

git add .
git commit -m'Preliminary commit'
git push

Return to your repo on GitHub and open the Actions tab. You’ll see one thing like this:

IDG

Now allow GitHub Pages, following the optional instructions. It ought to appear to be this:

IDG

Open the Actions tab once more, and also you’ll see a 3rd workflow:

IDG

Now open your generated web site, at https://{person}.github.io/{repo}. Mine is at https://meheller.github.io/nbdev-hello-world/. You possibly can copy that and alter meheller to your personal GitHub deal with and see one thing much like the next:

IDG

Proceed the nbdev walkthrough

Now we’re lastly attending to the good stuff. You’ll set up net hooks to mechanically clear notebooks whenever you test them in,

nbdev_install_hooks

export your library,

nbdev_export

set up your package deal,

pip set up -e '.[dev]'

preview your docs,

nbdev_preview

(and click on the hyperlink) and in the end begin modifying your Python pocket book:

jupyter pocket book

(and click on on nbs, and click on on 00_core.ipynb).

Edit the pocket book as described, then put together your adjustments:

nbdev_prepare

Edit index.ipynb as described, then push your adjustments to GitHub:

git add .
git commit -m'Add `say_hello`; replace index'
git push

If you want, you may push on and add advanced functionality.

IDG

As you’ve seen, particularly if you happen to’ve labored by the tutorial your self, nbdev can allow a extremely productive Python growth workflow in notebooks, working easily with a GitHub repo and Quarto documentation displayed on GitHub Pages. For those who haven’t but labored by the tutorial, what are you ready for?

—

Contact: quick.ai, https://nbdev.fast.ai/

Price: Free open supply below Apache License 2.0.

Platforms: macOS, Linux, and most Unix-style working programs. It really works on Home windows below WSL, however not below cmd or PowerShell.